What factors are contributing to the decline in completion rates for UserTesting's mobile app tests in the last quarter?

Introduction

The decline in completion rates for UserTesting's mobile app tests in the last quarter is a critical issue that requires immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term and long-term implications for the product and user experience.

I'll approach this problem by first clarifying key details, ruling out external factors, and then diving deep into the product, user journey, and relevant metrics. From there, I'll generate data-driven hypotheses, conduct root cause analysis, and propose validation methods and solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a seasonal component. Has this decline been observed in previous Q4 periods?

Why it matters: Helps distinguish between cyclical patterns and new issues. Expected answer: No, this decline is unique to this quarter. Impact on approach: If seasonal, we'd focus on optimizing for Q4 specifically.

• Considering user segments, I'm curious about the distribution. Are we seeing a uniform decline across all user types, or is it more pronounced in certain segments?

Why it matters: Identifies if the issue is global or specific to certain users. Expected answer: The decline is more significant among new users. Impact on approach: We'd prioritize onboarding and first-time user experience improvements.

• Thinking about recent changes, have there been any updates to the mobile app or testing platform in the last 3-4 months?

Why it matters: Pinpoints potential technical or UX causes. Expected answer: Yes, a major UI overhaul was implemented 2 months ago. Impact on approach: We'd focus on analyzing the impact of these recent changes.

• Regarding the metric itself, has there been any change in how completion rates are calculated or tracked?

Why it matters: Ensures we're comparing apples to apples. Expected answer: No changes to the metric definition or tracking. Impact on approach: Confirms the issue is with user behavior, not data collection.