Why did Prime Video streaming quality drop to 480p for 45% of users?

Introduction

The sudden drop in Prime Video streaming quality to 480p for 45% of users is a critical issue that demands immediate attention and a thorough root cause analysis. This problem not only affects user experience but also poses a significant risk to customer retention and brand reputation. I'll approach this issue systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the scale of the issue, I'm thinking this might be a system-wide problem. Can you confirm if this is affecting users across all regions and devices?

Why it matters: Understanding the scope helps narrow down potential causes. Expected answer: Yes, it's affecting users across all regions and devices. Impact on approach: If localized, we'd focus on regional infrastructure; if global, we'd look at core systems.

• Considering the specific quality drop to 480p, I'm wondering about our adaptive bitrate streaming system. Has there been any recent change to our ABR algorithms or content delivery network?

Why it matters: ABR systems are crucial for managing streaming quality. Expected answer: No recent changes to ABR algorithms, but there was a CDN configuration update last week. Impact on approach: If confirmed, we'd prioritize investigating the CDN update's impact.

• Given the 45% user impact, I'm curious about our monitoring systems. Did we receive any alerts before users reported issues, and if so, what metrics triggered them?

Why it matters: Assesses the effectiveness of our monitoring and alert systems. Expected answer: Some latency alerts were triggered, but not at a level that usually indicates widespread issues. Impact on approach: If monitoring failed to catch this, we'd need to review and improve our alert thresholds.

• Thinking about potential external factors, have we seen any unusual spikes in user activity or changes in viewing patterns recently?

Why it matters: Helps distinguish between internal issues and external pressures on our systems. Expected answer: There's been a 20% increase in concurrent streams due to a popular new series launch. Impact on approach: If confirmed, we'd investigate our capacity planning and scaling mechanisms.