What caused the sudden spike in failed API calls for NMI's tokenization service last week?

Introduction

The sudden spike in failed API calls for NMI's tokenization service last week is a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this might be related to a recent deployment. Has there been any significant update or change to the tokenization service in the past week?

Why it matters: Recent changes often correlate with sudden performance issues. Expected answer: Yes, there was a minor update to the API. Impact on approach: If confirmed, we'd focus on the recent changes and their potential side effects.

• Considering the nature of the issue, I'm wondering about the scale. What percentage increase in failed API calls are we seeing compared to the baseline?

Why it matters: The magnitude helps prioritize the issue and narrow down potential causes. Expected answer: A 30-50% increase in failed calls. Impact on approach: A larger spike might indicate a more systemic issue, while a smaller one could point to a specific component or user segment.

• Given that this is a tokenization service, I'm curious about the security implications. Have there been any changes in security protocols or encryption methods recently?

Why it matters: Security changes can sometimes lead to unexpected behavior in API calls. Expected answer: No recent security changes. Impact on approach: If there were changes, we'd focus on security-related issues; if not, we'd look elsewhere.

• Thinking about potential external factors, I'm wondering if there's been any unusual spike in API usage or new integrations. Have we onboarded any major new clients or seen any significant changes in usage patterns?

Why it matters: Sudden changes in load or usage patterns can strain systems and cause failures. Expected answer: No significant new clients or usage pattern changes. Impact on approach: If there were changes, we'd investigate capacity and scaling issues; if not, we'd focus more on internal system problems.