Why did Facebook Gaming stream latency increase to 15 seconds?

Introduction

Facebook Gaming's stream latency increase to 15 seconds represents a significant challenge that could impact user experience and platform competitiveness. I'll approach this issue systematically, focusing on identifying the root cause, validating hypotheses, and developing both immediate and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Considering the technical nature of streaming, I'm wondering about recent infrastructure changes. Have there been any significant updates to our content delivery network (CDN) or server architecture in the past month?

Why it matters: Infrastructure changes often impact latency directly. Expected answer: Yes, we recently migrated to a new CDN provider. Impact on approach: If confirmed, I'd focus on CDN-related hypotheses and solutions.

• Looking at user feedback, I'm curious about the geographic distribution of this issue. Is the increased latency consistent across all regions, or are some areas more affected than others?

Why it matters: Helps identify if it's a global issue or localized to specific regions. Expected answer: The issue is more pronounced in certain regions. Impact on approach: I'd prioritize investigating region-specific factors and solutions.

• Considering potential changes in user behavior, has there been a significant shift in the types of content being streamed or viewed on the platform recently?

Why it matters: Different content types may have varying bandwidth requirements. Expected answer: There's been an increase in high-definition and interactive streams. Impact on approach: I'd explore how content type changes might be affecting overall latency.

• Thinking about system health, have we observed any unusual spikes in server load or network traffic coinciding with the latency increase?

Why it matters: Unusual load could indicate capacity issues or potential DDoS attacks. Expected answer: There have been intermittent traffic spikes. Impact on approach: I'd focus on scaling solutions and traffic management strategies.