What caused the sudden 50% decrease in API calls to TaxBit's enterprise reconciliation service last week?

Introduction

The sudden 50% decrease in API calls to TaxBit's enterprise reconciliation service last week is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term and long-term implications for our product and users.

I'll approach this problem by first clarifying key details, ruling out external factors, and then diving deep into our product ecosystem. We'll break down the metric, gather relevant data, form hypotheses, and conduct a thorough root cause analysis. Finally, we'll develop a comprehensive plan to validate our findings and implement solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent product update. Have there been any significant changes to the API or related systems in the past two weeks?

Why it matters: Recent changes often correlate with sudden metric shifts. Expected answer: Yes, there was a minor API version update. Impact on approach: If confirmed, we'd prioritize investigating the update's impact.

• Given the enterprise nature of the service, I'm wondering about our client base. Has there been any change in our top enterprise clients' usage patterns or contracts recently?

Why it matters: Enterprise clients often drive significant portions of API usage. Expected answer: No major changes reported in client contracts or usage. Impact on approach: If true, we'd focus more on technical issues rather than client-side changes.

• Considering the 50% drop, I'm curious about the distribution. Is this decrease uniform across all clients, or are some disproportionately affected?

Why it matters: Helps distinguish between global issues and client-specific problems. Expected answer: The decrease is not uniform; some clients are more affected than others. Impact on approach: We'd prioritize investigating common factors among the most affected clients.

• Given the critical nature of reconciliation services, I'm wondering about our monitoring systems. Have we observed any corresponding increase in error rates or latency?

Why it matters: Performance issues often precede usage drops. Expected answer: There's been a slight increase in API error rates. Impact on approach: We'd focus on the correlation between errors and the usage drop.