Why have edX's course completion rates for self-paced courses dropped 15% last month?

Introduction

The recent 15% drop in edX's course completion rates for self-paced courses is a significant issue that requires immediate attention. As we analyze this product challenge, we'll follow a systematic approach to identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

I'll begin by clarifying key details about the situation, then rule out external factors before diving deep into the product ecosystem, user journey, and metric breakdown. From there, we'll generate data-driven hypotheses, conduct root cause analysis, and develop a comprehensive plan to address the issue.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be seasonality at play. Has this 15% drop been compared to the same period last year?

Why it matters: Seasonal fluctuations could explain the drop and impact our solution approach. Expected answer: Yes, it's been compared and is still significant. Impact on approach: If seasonal, we'd focus on year-over-year trends rather than month-to-month.

• Considering user segments, I'm curious if this drop is uniform across all course types. Are there specific categories or difficulty levels more affected?

Why it matters: Identifying patterns in affected courses could point to content-specific issues. Expected answer: The drop is more pronounced in advanced courses. Impact on approach: We'd investigate factors unique to advanced courses, such as complexity or time commitment.

• Thinking about recent changes, have there been any significant updates to the platform or course structure in the past 1-2 months?

Why it matters: Recent changes could directly impact user behavior and completion rates. Expected answer: A new user interface was rolled out six weeks ago. Impact on approach: We'd focus on how the UI change might have affected user engagement and course navigation.

• Regarding data integrity, has there been any change in how completion rates are calculated or tracked?

Why it matters: Ensures we're comparing apples to apples and not chasing a data anomaly. Expected answer: No changes to the calculation method. Impact on approach: Confirms we should focus on actual user behavior rather than measurement issues.