What caused the sudden spike in API errors for Salsify's Digital Shelf Analytics feature last week?

Introduction

The sudden spike in API errors for Salsify's Digital Shelf Analytics feature last week is a critical issue that demands immediate attention. As we dive into this product root cause analysis, I'll employ a systematic approach to identify, validate, and address the underlying causes while considering both short-term fixes and long-term strategic implications.

Step 1

Clarifying Questions (3 minutes)

• Considering the timing, I'm wondering if there were any recent updates or changes to the Digital Shelf Analytics feature. Have there been any deployments or configuration changes in the past week?

Why it matters: Recent changes often correlate with sudden performance issues. Expected answer: Yes, there was a minor update to improve data processing speed. Impact on approach: If confirmed, we'd focus on the recent changes as a primary area of investigation.

• Given the nature of API errors, I'm curious about the specific error types we're seeing. Can you provide more details on the most common error codes or messages being reported?

Why it matters: Different error types point to different root causes. Expected answer: Mostly 500 Internal Server Errors and some 503 Service Unavailable errors. Impact on approach: This would guide our technical investigation towards backend issues rather than client-side problems.

• Thinking about user impact, I'm wondering about the scale of this issue. What percentage of API calls are failing, and is this consistent across all customers or concentrated in specific segments?

Why it matters: Understanding the scope helps prioritize the severity and nature of the problem. Expected answer: About 15% of calls are failing, primarily affecting enterprise customers. Impact on approach: This would focus our efforts on enterprise-specific configurations or high-volume usage patterns.

• Considering potential external factors, have there been any significant changes in usage patterns or data volume from our customers recently?

Why it matters: Unusual spikes in usage or data volume could strain the system. Expected answer: There's been a 20% increase in API calls over the past month. Impact on approach: This would lead us to investigate scalability issues and capacity planning.