Why has YouTube recommendation relevance dropped by 40% for new users?

Introduction

YouTube's recommendation relevance dropping by 40% for new users is a critical issue that demands immediate attention. This significant decline in recommendation quality could lead to decreased user engagement, retention, and ultimately, revenue. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the user segment, I'm thinking this might be isolated to new users. Is the 40% drop specific to new users, or are we seeing any impact on existing users as well?

Why it matters: Understanding the scope helps us focus our investigation and potential solutions. Expected answer: The drop is primarily affecting new users, with minimal impact on existing users. Impact on approach: If it's isolated to new users, we'll focus on onboarding and initial recommendation algorithms.

• Considering the magnitude of the drop, I'm wondering about the timeframe. Has this 40% drop occurred suddenly or gradually over time?

Why it matters: The timeline can indicate whether this is due to a specific change or a gradual degradation. Expected answer: The drop occurred relatively suddenly over the past two weeks. Impact on approach: A sudden drop would point us towards recent changes or issues.

• Given the focus on relevance, I'm curious about how we're measuring this metric. Can you clarify the specific components of the recommendation relevance score?

Why it matters: Understanding the metric composition helps us pinpoint which aspects are underperforming. Expected answer: The relevance score is based on click-through rates, watch time, and user feedback. Impact on approach: We'll focus our analysis on these specific components of the relevance score.

• Thinking about potential system changes, have there been any recent updates to the recommendation algorithm or content moderation policies?

Why it matters: Recent changes could be directly responsible for the drop in relevance. Expected answer: There was a minor update to the recommendation algorithm two weeks ago. Impact on approach: We'll prioritize investigating the impact of this recent update.