How can we explain the unexpected 20% decline in lesson completion rates for Preply's mobile app users in the last two weeks?

Introduction

The unexpected 20% decline in lesson completion rates for Preply's mobile app users over the past two weeks is a critical issue that demands immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term and long-term implications for our product and users.

I'll approach this problem by first clarifying key details, ruling out external factors, and then diving deep into our product, user journey, and metrics. From there, I'll generate data-driven hypotheses, conduct root cause analysis, and propose validation methods and solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking this could be related to a recent app update. Has there been any significant update or change to the mobile app in the last 2-3 weeks?

Why it matters: Recent changes often correlate with sudden metric shifts. Expected answer: Yes, there was an update. Impact on approach: If yes, we'd focus on changes in that update; if no, we'd look at other factors.

• Considering user segments, I'm curious about the distribution of this decline. Is the 20% drop uniform across all user segments, or is it more pronounced in specific groups?

Why it matters: Helps identify if the issue is global or segment-specific. Expected answer: The decline varies across segments. Impact on approach: If varied, we'd focus on the most affected segments; if uniform, we'd look at system-wide issues.

• Given the specificity of the metric, I'm wondering about our definition. Has there been any change in how we define or measure "lesson completion" in the last month?

Why it matters: Ensures we're comparing apples to apples in our metrics. Expected answer: No change in definition. Impact on approach: If changed, we'd need to recalibrate our analysis; if not, we can proceed with current data.

• Considering potential technical issues, have we noticed any unusual patterns in app performance metrics, such as load times or crash rates, during this period?

Why it matters: Technical issues can significantly impact user behavior. Expected answer: Some performance anomalies noted. Impact on approach: If yes, we'd prioritize technical investigations; if no, we'd focus more on user behavior and content.