Why did Netflix recommendation relevance drop by 45%?

Introduction

Netflix's recommendation system is a cornerstone of its user experience, driving engagement and retention. A 45% drop in recommendation relevance is a critical issue that demands immediate attention and thorough analysis. I'll approach this problem systematically, examining potential causes, gathering data, and proposing solutions to restore and improve the recommendation system's performance.

Step 1

Clarifying Questions (3 minute)

• Timing of the drop: "I'm thinking this might be a sudden change. Has the 45% drop occurred gradually over time or was it a sudden decrease?"

Why it matters: Helps distinguish between systemic issues and potential one-time events. Expected answer: Sudden drop within the last week. Impact on approach: A sudden drop would focus our investigation on recent changes or events.

• Scope of the issue: "Are we seeing this drop across all user segments and content types, or is it more pronounced in specific areas?"

Why it matters: Identifies whether this is a global issue or limited to certain user groups or content. Expected answer: More pronounced in certain content categories. Impact on approach: Would guide us to investigate specific content recommendation algorithms or data sources.

• Recent changes: "Have there been any significant updates to the recommendation algorithm or content library in the past month?"

Why it matters: Could directly link the issue to recent modifications. Expected answer: A major algorithm update was implemented two weeks ago. Impact on approach: Would focus our investigation on the recent update and its implications.

• User feedback: "Has there been an increase in user complaints or changes in viewing patterns since the drop in relevance?"

Why it matters: Provides insight into user perception and behavior changes. Expected answer: Slight increase in complaints about irrelevant recommendations. Impact on approach: Would incorporate user feedback into our analysis and solution design.

• Measurement accuracy: "Are we confident in the accuracy of our relevance measurement system? Have there been any changes to how we calculate or collect this metric?"

Why it matters: Ensures the problem isn't a result of measurement errors. Expected answer: No changes to the measurement system. Impact on approach: Would rule out measurement issues and focus on actual performance problems.