You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that the % of people that added more information increased by 8%. However, 7 day retention decreased by 2%. What do you do?

Introduction

The trade-off we're facing involves a new signup flow designed to encourage users to add more profile information. While we've seen an 8% increase in users adding more information, we've also observed a 2% decrease in 7-day retention. This scenario presents a classic product trade-off between user engagement and retention.

In my response, I'll analyze this situation, propose hypotheses, design experiments, and provide a recommendation based on a data-driven approach.

Step 1

Clarifying Questions (3 minutes)

Before diving deeper, I'd like to ask a few questions to better understand the context:

• What's the current 7-day retention rate, and how does it compare to industry benchmarks?

• Why it matters: This helps us understand the severity of the 2% decrease.
• Hypothetical answer: Our current 7-day retention is 35%, which is slightly above the industry average of 32%.
• Impact: If we're already performing well, we might have more flexibility to experiment with engagement tactics.

• What specific profile information are we encouraging users to add, and how is it being used to improve the user experience?

• Why it matters: This helps us understand the value proposition for users.
• Hypothetical answer: We're asking for interests, preferences, and social connections to provide personalized content and recommendations.
• Impact: If the added information significantly enhances the user experience, it might be worth a short-term retention dip.

• Are there any specific user segments that show different patterns in this trade-off?

• Why it matters: This could reveal opportunities for targeted approaches.
• Hypothetical answer: Younger users (18-24) show higher engagement with the new flow but less retention impact.
• Impact: We might consider segment-specific strategies or a more granular rollout.

• What's the typical time frame for seeing the full impact of retention initiatives in our product?

• Why it matters: This helps us determine if we need to extend our observation period.
• Hypothetical answer: We usually see the full impact within 30 days.
• Impact: We might need to design longer-term experiments to fully understand the retention effects.