What caused the sudden 30% increase in latency for Tencent's QQ messaging services during peak hours yesterday?

Introduction

The sudden 30% increase in latency for Tencent's QQ messaging services during peak hours yesterday is a critical issue that demands immediate attention. As we analyze this product problem, we'll follow a systematic framework to identify, validate, and address the root cause while considering both immediate and long-term implications.

Our approach will involve clarifying the situation, ruling out external factors, understanding the product and user journey, breaking down the metric, gathering data, forming hypotheses, conducting root cause analysis, and developing a resolution plan. This structured method will ensure we cover all angles and arrive at a comprehensive solution.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent deployment. Has there been any significant code push or infrastructure change in the last 24-48 hours?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a minor update to the messaging protocol. Impact on approach: If confirmed, we'd prioritize investigating that change.

• Considering the specificity of "peak hours," I'm wondering about usage patterns. Can you provide more details on what constitutes peak hours for QQ messaging and if this latency increase is consistent across all peak periods?

Why it matters: Understanding usage patterns helps identify potential capacity issues. Expected answer: Peak hours are typically 7-10 PM, and the latency increase is consistent during these times. Impact on approach: Consistent issues during peak times might indicate scaling problems.

• Given the magnitude of the increase, I'm curious about user segments. Is this latency increase affecting all users equally, or are there specific demographics or regions more impacted?

Why it matters: Segmented impact could point to localized issues or specific user behaviors. Expected answer: The issue seems to affect urban users more significantly. Impact on approach: We'd focus on investigating urban infrastructure and usage patterns.

• Considering potential measurement anomalies, I'm wondering about our monitoring systems. Have there been any changes to how we measure or define latency recently?

Why it matters: Ensures we're addressing a real issue and not a measurement artifact. Expected answer: No changes to measurement systems or definitions. Impact on approach: Confirms the issue is genuine and not a data anomaly.