Why has user engagement with Udemy's course recommendation system suddenly decreased by 20%?

Introduction

The sudden 20% decrease in user engagement with Udemy's course recommendation system is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

I'll approach this problem by first clarifying the context, then ruling out external factors before diving deep into internal causes. We'll break down the metric, gather relevant data, form hypotheses, and conduct a thorough root cause analysis. Finally, we'll develop a comprehensive plan to validate our findings and implement solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a recent change in the recommendation algorithm. Has there been any update to the recommendation system in the past month?

Why it matters: Recent changes could directly impact user engagement. Expected answer: Yes, there was an update two weeks ago. Impact on approach: If confirmed, we'd focus on the algorithm change as a primary hypothesis.

• Considering user segments, I'm curious about the distribution of this engagement drop. Is the 20% decrease uniform across all user types, or are certain segments more affected?

Why it matters: Helps identify if the issue is global or specific to certain user groups. Expected answer: The decrease is more pronounced among new users. Impact on approach: We'd prioritize investigating onboarding and new user experience.

• Thinking about the metric itself, I'm wondering about the exact definition of "engagement" in this context. Can you clarify how engagement is measured for the recommendation system?

Why it matters: Ensures we're analyzing the correct data points. Expected answer: Engagement is measured by clicks on recommended courses and time spent viewing recommendations. Impact on approach: We'd focus on these specific actions in our analysis.

• Considering potential technical issues, have there been any reported problems with the recommendation system's performance or uptime?

Why it matters: Technical issues could directly impact user engagement. Expected answer: No significant technical issues reported. Impact on approach: We'd shift focus from technical problems to user experience and content quality.