What caused the sudden spike in API errors for Stytch's Phone Authentication service yesterday afternoon?

Introduction

The sudden spike in API errors for Stytch's Phone Authentication service yesterday afternoon is a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term implications for our product ecosystem.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any recent code push or system update to the Phone Authentication service?

Why it matters: Recent changes are often the culprit in sudden performance issues. Expected answer: Yes, there was a deployment yesterday morning. Impact on approach: If confirmed, we'd focus on changes in that deployment.

• Considering the nature of API errors, I'm wondering about the specific error types. Can you provide more details on the most common error codes or messages seen during this spike?

Why it matters: Different error types point to different root causes. Expected answer: A mix of timeout and authentication errors. Impact on approach: This would help narrow down whether it's a performance or logic issue.

• Given that it's a Phone Authentication service, I'm curious about the volume. Has there been any unusual increase in authentication requests coinciding with the error spike?

Why it matters: Unexpected load can cause service degradation. Expected answer: Traffic was within normal ranges. Impact on approach: If traffic was normal, we'd focus more on internal issues rather than capacity problems.

• Thinking about the service dependencies, I'm wondering if this could be related to our SMS provider. Have we seen any issues or changes with our SMS delivery service?

Why it matters: External dependencies can often be the source of cascading failures. Expected answer: No reported issues from the SMS provider. Impact on approach: If confirmed, we'd shift focus to internal systems and integration points.