What caused the sudden spike in failed transactions on Indodax's Bitcoin trading pair yesterday?

Introduction

The sudden spike in failed transactions on Indodax's Bitcoin trading pair yesterday is a critical issue that demands immediate attention. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term implications for our platform.

I'll structure our analysis by first clarifying the situation, ruling out external factors, understanding the user journey, breaking down the relevant metrics, gathering data, forming hypotheses, conducting root cause analysis, and finally proposing validation methods and next steps.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent system update. Have there been any significant changes to our trading engine or infrastructure in the past 48 hours?

Why it matters: Recent changes often correlate with sudden performance issues. Expected answer: Yes, a minor update was pushed yesterday morning. Impact on approach: If confirmed, we'd focus on rollback considerations and code review.

• Considering user segments, I'm curious about the distribution of these failed transactions. Are we seeing this issue across all user types, or is it concentrated among specific groups like high-volume traders?

Why it matters: Helps narrow down potential causes and affected user base. Expected answer: The issue seems to affect all user types equally. Impact on approach: If it's widespread, we'd look at core system components rather than user-specific features.

• Given the nature of cryptocurrency markets, I'm wondering about any unusual market conditions. Was there any significant price volatility or trading volume spike for Bitcoin around the time of the incident?

Why it matters: External market factors can sometimes trigger internal system issues. Expected answer: Bitcoin price was relatively stable, but trading volume was higher than usual. Impact on approach: High volume could point us towards capacity issues or rate limiting problems.

• Thinking about our monitoring systems, I'm curious if we've seen any correlated alerts or anomalies in other parts of our system. Have there been any notable changes in API response times, database performance, or network metrics?

Why it matters: Helps identify if this is an isolated issue or part of a broader system problem. Expected answer: We noticed slightly elevated API response times shortly before the spike in failed transactions. Impact on approach: This could indicate a bottleneck in our API layer or downstream services.