Why has Netflix My List ordering randomized for 35% of users?

Introduction

Netflix's My List feature, a cornerstone of personalized content curation, has unexpectedly randomized for 35% of users. This issue potentially disrupts user experience and engagement, warranting immediate investigation. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Considering the specificity of 35%, I'm thinking this might be a targeted test. Is this a planned A/B test that went awry?

Why it matters: Distinguishes between intentional changes and unintended issues. Expected answer: No, this wasn't a planned test. Impact on approach: If unplanned, we'll focus on technical issues or unintended consequences of recent changes.

• Given the randomization, I'm wondering about recent backend changes. Have there been any updates to the recommendation algorithm or content sorting systems in the past week?

Why it matters: Identifies potential technical triggers for the issue. Expected answer: Yes, there was a minor update to the recommendation system. Impact on approach: We'll prioritize investigating the recent update and its potential side effects.

• Noticing the specific user percentage, I'm curious about user segmentation. Is this issue affecting a particular user demographic or device type more than others?

Why it matters: Helps narrow down potential causes and affected user groups. Expected answer: The issue seems to be spread across different user segments. Impact on approach: We'll need to look at system-wide changes rather than user-specific factors.

• Thinking about user behavior, has there been any significant change in engagement metrics for the affected users since the randomization began?

Why it matters: Assesses the impact on user experience and potential business implications. Expected answer: Yes, there's been a slight decrease in time spent on the platform for affected users. Impact on approach: We'll prioritize quick resolution to mitigate negative impact on user engagement.