What caused the sudden spike in query latency for Amperity's Identity Resolution service yesterday afternoon?

Introduction

The sudden spike in query latency for Amperity's Identity Resolution service yesterday afternoon is a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll follow a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term implications for our service.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any significant code or infrastructure changes in the past 24-48 hours?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a deployment yesterday morning. Impact on approach: If confirmed, we'd focus on changes in that deployment.

• Considering the nature of Identity Resolution, I'm wondering about data volume. Have we seen any unusual spikes in data ingestion or processing volume recently?

Why it matters: Unexpected data volume can strain system resources. Expected answer: Data volume has been consistent. Impact on approach: If volumes are normal, we'd look more at system-level issues.

• Given the specificity of "yesterday afternoon," I'm curious about the exact timeline. Can you provide more details on when the latency spike started and if it's still ongoing?

Why it matters: The duration and pattern of the issue can indicate its nature. Expected answer: Started at 2 PM, peaked at 4 PM, still elevated but improving. Impact on approach: This timeline would help us correlate with other events or patterns.

• Thinking about the impact, I'm concerned about user experience. What percentage of queries are affected, and are all customers experiencing this or just a subset?

Why it matters: Helps prioritize the issue and identify potential segmentation factors. Expected answer: 30% of queries affected, spread across multiple customers. Impact on approach: If widespread, we'd look at core service components rather than customer-specific issues.