What caused the sudden spike in processing time for Datavant's TokenizR service last Tuesday?

Introduction

The sudden spike in processing time for Datavant's TokenizR service last Tuesday presents a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll employ a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term implications for the service.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any significant update or change to the TokenizR service in the past week?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a minor update deployed on Monday. Impact on approach: If confirmed, we'd focus on the recent changes as a primary area of investigation.

• Considering the nature of the service, I'm curious about the load patterns. Has there been any unusual spike in usage or change in user behavior around the time of the incident?

Why it matters: Unusual load patterns can strain the system and cause performance degradation. Expected answer: Usage has been relatively stable, with a slight increase in API calls. Impact on approach: If usage patterns are normal, we'd shift focus to internal system issues rather than external factors.

• Given that it's a tokenization service, I'm wondering about data complexity. Has there been any change in the type or volume of data being processed?

Why it matters: Changes in data characteristics can impact processing time. Expected answer: No significant changes in data types, but volume has increased by 15%. Impact on approach: A volume increase would lead us to investigate scaling and capacity issues.

• Thinking about the broader ecosystem, are there any dependent services or integrations that might have changed or experienced issues?

Why it matters: Issues in connected systems can ripple through and affect TokenizR's performance. Expected answer: One of our data providers reported some latency issues last week. Impact on approach: This would prompt us to investigate the interaction between TokenizR and its dependencies.