What caused the sudden spike in failed transactions on Zapper's token swap feature yesterday?

Introduction

The sudden spike in failed transactions on Zapper's token swap feature yesterday is a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term implications for our product ecosystem.

Our analysis will follow a structured framework, beginning with clarifying questions to establish context, ruling out external factors, understanding the product and user journey, breaking down the relevant metrics, gathering and prioritizing data, forming hypotheses, conducting root cause analysis, and finally proposing validation methods and next steps.

Step 1

Clarifying Questions (3 minutes)

• I'm noticing the issue is specific to the token swap feature. Would you say this problem is isolated to this feature, or are other parts of Zapper affected?

Why it matters: Helps determine if it's a localized issue or a broader system problem. Expected answer: Isolated to token swap feature. Impact on approach: If isolated, we focus on swap-specific components; if widespread, we investigate system-wide issues.

• Given the sudden nature of the spike, I'm thinking there might have been a recent deployment or change. Have there been any updates to the token swap feature or related systems in the past 48 hours?

Why it matters: Recent changes are often culprits in sudden performance shifts. Expected answer: Yes, a minor update was deployed yesterday morning. Impact on approach: If yes, we'd prioritize reviewing the recent changes; if no, we'd look at external factors or gradual degradation.

• Considering user impact, I'm curious about the scale. What percentage of token swap transactions are failing compared to the normal baseline?

Why it matters: Helps quantify the severity and urgency of the issue. Expected answer: Failure rate increased from 2% to 15%. Impact on approach: A significant increase would prioritize immediate action, while a smaller increase might allow for more thorough investigation.

• Looking at the timing, I'm thinking about potential external factors. Have there been any significant market events or volatility in the past 24 hours that could impact token swaps?

Why it matters: External market conditions can sometimes explain sudden changes in transaction behavior. Expected answer: No major market events, relatively stable conditions. Impact on approach: If yes, we'd consider market-related factors; if no, we'd focus more on internal systems and user behavior.