What caused the sudden spike in latency for Temporal's task queue operations yesterday afternoon?

Introduction

The sudden spike in latency for Temporal's task queue operations yesterday afternoon is a critical issue that demands immediate attention and thorough analysis. As we delve into this problem, we'll follow a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term implications for our product ecosystem.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this might be related to a recent deployment. Have there been any significant changes or updates to the task queue system in the past 48 hours?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, a minor update was deployed yesterday morning. Impact on approach: If confirmed, we'd focus on the recent changes as a primary area of investigation.

• Considering the nature of task queues, I'm wondering about sudden changes in workload. Has there been any unusual spike in the number of tasks or their complexity in the affected time frame?

Why it matters: Unexpected load can strain system resources and cause latency. Expected answer: No significant change in overall workload, but we haven't checked task complexity. Impact on approach: If task complexity has increased, we'd need to investigate potential bottlenecks in task processing.

• Given that this is a latency issue, I'm curious about the specific metrics we're seeing. What's the normal latency for task queue operations, and what levels did we observe during the spike?

Why it matters: Understanding the magnitude of the issue helps prioritize our response. Expected answer: Normal latency is 100ms, spiked to 500ms during the incident. Impact on approach: A 5x increase suggests a severe issue requiring immediate attention.

• Thinking about potential external factors, have we observed any similar issues with other services or dependencies that the task queue interacts with?

Why it matters: Latency issues can sometimes propagate from interconnected systems. Expected answer: No reported issues with other services, but we haven't thoroughly checked all dependencies. Impact on approach: If confirmed, we'd need to expand our investigation to include system dependencies.