How can we explain the unexpected 20% decline in API calls to VTEX's Catalog service during peak holiday shopping hours?

Introduction

The unexpected 20% decline in API calls to VTEX's Catalog service during peak holiday shopping hours presents a critical issue that demands immediate attention and thorough analysis. This significant drop in API usage could have far-reaching implications for our e-commerce platform's performance, user experience, and ultimately, our bottom line during a crucial sales period.

To address this problem, I'll employ a systematic approach that covers issue identification, hypothesis generation, validation, and solution development. My analysis will focus on uncovering the root cause while considering both short-term fixes and long-term strategic implications.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Have there been any significant changes to the Catalog service or related systems in the past 48 hours?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a deployment yesterday. Impact on approach: If confirmed, we'd focus on rollback options and code review.

• Considering the scale of the decline, I'm wondering about the geographic distribution. Is this decline uniform across all regions, or are some areas more affected than others?

Why it matters: Helps identify if it's a global issue or localized problem. Expected answer: The decline is more pronounced in certain regions. Impact on approach: We'd investigate region-specific factors like CDN performance or local infrastructure.

• Given that it's during peak shopping hours, I'm curious about the load on our systems. Are we seeing any unusual patterns in server load or response times compared to previous holiday seasons?

Why it matters: Helps determine if it's a capacity issue or something else. Expected answer: Server load is within expected ranges, but response times are higher. Impact on approach: We'd focus on optimizing query performance and database interactions.

• Considering the nature of the Catalog service, I'm wondering about any recent changes in how our largest merchants are using the API. Have there been any significant shifts in API usage patterns from our top clients?

Why it matters: Large clients can significantly impact overall metrics. Expected answer: A few major clients have changed their integration recently. Impact on approach: We'd reach out to these clients and analyze their specific usage patterns.