You're the PM for Facebook Stories. A new feature you just released led to a 5% decrease in the number of active creators but a 10% increase in the number of daily active users. Would you keep or roll back this feature?

Introduction

The trade-off we're facing with Facebook Stories is a decrease in active creators versus an increase in daily active users. This scenario presents a complex challenge that requires careful analysis of our product goals, user behavior, and long-term platform health. I'll walk you through my approach to evaluating this situation and making a recommendation.

Step 1

Clarifying Questions (3 minutes)

• What specific feature was released that led to these changes?

• Why it matters: Understanding the feature helps identify the root cause of the trade-off.
• Hypothetical answer: A new AI-powered content suggestion tool for viewers.
• Impact: This would focus our analysis on the viewer experience and content discovery mechanisms.

• How long has this trend been observed since the feature release?

• Why it matters: Distinguishes between short-term reactions and long-term behavioral shifts.
• Hypothetical answer: The trend has been consistent for two weeks.
• Impact: This timeframe suggests the change is significant and not just a temporary fluctuation.

• Are there any specific creator segments more affected than others?

• Why it matters: Helps identify if the impact is uniform or concentrated in certain user groups.
• Hypothetical answer: Smaller creators with fewer followers are more affected.
• Impact: This would require us to consider strategies to support and retain smaller creators.

• Has there been any change in the quality or engagement of the content being produced?

• Why it matters: Helps assess the overall health of the Stories ecosystem.
• Hypothetical answer: Engagement per story has increased by 15%.
• Impact: This suggests that while there are fewer creators, the content is resonating more with viewers.

• What's the current revenue model for Stories, and how might these changes affect it?

• Why it matters: Aligns our decision with business objectives.
• Hypothetical answer: Revenue is primarily from ads between stories, which has increased due to higher viewer engagement.
• Impact: This would favor keeping the feature, but we'd need to balance short-term gains with long-term ecosystem health.

Step 2

Trade-off Type Identification (1 minute)

This situation falls under the trade-off sub-type (b) - same product with different variations. We're dealing with a single product, Facebook Stories, but the new feature has created a variation that affects different user segments (creators and viewers) in opposite ways.

Identifying this trade-off type informs our approach by highlighting the need to balance the interests of two key user groups within the same product ecosystem. It suggests that our analysis should focus on the interdependencies between creators and viewers, and how changes to one group affect the other.

Step 3

Product Understanding (5 minutes)

Facebook Stories is a feature that allows users to share photos, videos, and animations that disappear after 24 hours. Its core features include:

1. Creation tools (filters, stickers, text overlays)
2. Sharing options (public, friends, custom lists)
3. Viewing interface (tappable progression, reactions, direct messages)
4. Discovery mechanisms (algorithmic recommendations, friend activity)

Key stakeholders include:

• Creators (both individuals and brands)
• Viewers (casual users and heavy consumers)
• Advertisers
• Facebook (the company)

The value proposition of Stories is to provide a casual, immediate way for users to share moments of their lives and for viewers to stay connected with friends and interests. It aligns with Facebook's mission to bring the world closer together by facilitating real-time, authentic connections.

The user flow in the Stories ecosystem typically looks like this:

1. Creators produce content using the Stories creation tools
2. Content is shared and becomes visible to the creator's network or publicly
3. Viewers discover and consume Stories through their feed or active searching
4. Viewers interact with Stories (reactions, messages) or create their own in response
5. The cycle continues with new content creation and consumption

This ecosystem relies on a balance between content creation and consumption. The new feature seems to have shifted this balance, which is at the heart of our current trade-off.

Step 4

Trade-off Agreement and Hypothesis (5 minutes)

The trade-off we're considering is between creator retention and user engagement. Specifically, we're weighing a 5% decrease in active creators against a 10% increase in daily active users.

My hypothesis for why we're seeing these results is that the new AI-powered content suggestion tool is highly effective at surfacing engaging content to viewers, leading to increased user activity. However, this same mechanism may be concentrating visibility on a smaller pool of highly engaging creators, inadvertently discouraging less established creators who are seeing decreased visibility for their content.

Potential impacts

Considering different user types:

• For viewers, the short-term impact is largely positive, with more engaging content readily available.
• For established creators, this change may be beneficial, increasing their reach and engagement.
• For newer or niche creators, the impact is negative, potentially leading to decreased motivation to create content.

For the platform itself, the short-term gains in user engagement are clear, but there's a risk of long-term ecosystem health if creator diversity isn't maintained.

If we were to fully embrace the viewer-centric approach for an extended period, we might see extreme outcomes such as:

• A significant reduction in the number of active creators, leading to a less diverse content ecosystem
• Potential user fatigue from seeing content from a limited pool of creators
• Difficulty in surfacing fresh talent or niche content that could be valuable to specific user segments

Conversely, if we were to prioritize creator retention at the expense of viewer engagement, we might risk:

• Decreased overall platform usage if content quality or relevance suffers
• Reduced ad revenue due to lower user engagement
• Potential loss of competitive edge against other social media platforms that offer more engaging user experiences

Step 5

Key Metrics Identification (4 minutes)

North Star Metric: Daily Active Engagement (DAE) - a composite metric that combines daily active users with meaningful interactions per user.

This North Star aligns with our goal of fostering an active, engaging platform while also capturing the health of our creator ecosystem. It intersects value for viewers (who contribute to daily actives), creators (whose content drives engagement), and the platform (which benefits from both).

Supporting metrics:

1. Creator Retention Rate

   • Importance: Measures the health of our creator ecosystem
   • Stakeholder relation: Directly impacts creators, indirectly affects viewers and platform health

2. Content Diversity Index

   • Importance: Ensures a wide range of content is being produced and consumed
   • Stakeholder relation: Benefits viewers through variety, encourages diverse creators, supports long-term platform health

3. Viewer Satisfaction Score

   • Importance: Measures the quality of the user experience
   • Stakeholder relation: Directly impacts viewers, provides feedback to creators, indicates platform success

4. Creator Satisfaction Score

   • Importance: Gauges the platform's appeal to content producers
   • Stakeholder relation: Directly impacts creators, indirectly affects content quality for viewers

5. Average Engagement per Story

   • Importance: Measures content quality and relevance
   • Stakeholder relation: Indicates success for creators, satisfaction for viewers, and overall platform health

6. New Creator Acquisition Rate

   • Importance: Ensures a fresh influx of content and talent
   • Stakeholder relation: Vital for platform growth, content diversity, and viewer interest

7. Ad Engagement Rate

   • Importance: Tracks the business impact of changes to the Stories ecosystem
   • Stakeholder relation: Critical for platform revenue, indirectly affects resources available for product development

These metrics include both leading indicators (e.g., Creator Satisfaction Score, which can predict future creator behavior) and lagging indicators (e.g., Creator Retention Rate, which confirms the long-term impact of changes).

Step 6

Experiment Design (3 minutes)

To validate our hypotheses about the impact of the new feature, I propose an A/B/C test:

Experiment Hypothesis: Modifying the AI content suggestion algorithm to include a diversity factor will increase creator retention without significantly impacting the gains in daily active users.

• Control Group (A): Current version of the feature
• Treatment Group B: Modified AI algorithm with a moderate diversity factor
• Treatment Group C: Modified AI algorithm with a high diversity factor

Target Audience:

• Size: 10% of our user base for each group (30% total)
• Characteristics: Stratified sample ensuring representation across user activity levels, geographies, and device types

Duration: 4 weeks to account for novelty effects and establish stable behavior patterns

Key considerations for test validity:

• Randomization: Use a hash function on user IDs to ensure consistent group assignment
• Sample size: Calculated to detect a 2% change in our North Star metric with 95% confidence
• Novelty effects: Monitor daily trends and consider extending the test if metrics haven't stabilized

Guardrail metrics to monitor:

• Overall platform usage (to ensure we're not negatively impacting the broader ecosystem)
• Content report rates (to watch for any unintended increases in problematic content)
• Creator churn rate (to catch any sudden drops in creator activity)

Step 7

Data Analysis Plan (3 minutes)

To evaluate the experiment results, we'll analyze:

1. Changes in our North Star metric (Daily Active Engagement) across all three groups
2. Movement in all supporting metrics identified earlier
3. Correlation between creator retention and user engagement metrics

Specific analyses:

• Segment analysis: Compare impacts across creator sizes (e.g., nano, micro, macro influencers) and viewer engagement levels
• Cohort analysis: Track new creators during the experiment period to assess retention and growth
• Funnel analysis: Examine changes in the viewer journey from discovery to engagement

When metrics move in opposite directions, we'll:

1. Quantify the trade-off (e.g., X% loss in metric A yields Y% gain in metric B)
2. Assess the long-term impact using historical data and industry benchmarks
3. Consider the strategic importance of each metric to our overall goals

We'll pay special attention to:

• Any non-linear relationships between creator numbers and engagement
• Threshold effects (e.g., minimum number of creators needed for a healthy ecosystem)
• Unexpected positive outcomes (e.g., higher quality content from fewer creators)

Step 8

Decision Framework (4 minutes)

Decision tree approach:

Red flags that would prevent shipping:

• 10% drop in creator retention

• Any significant decrease in content diversity
• Negative impact on ad engagement rates

For mixed results:

1. Weigh the magnitude of changes in each metric
2. Consider long-term strategic goals (e.g., prioritize ecosystem health over short-term gains)
3. Evaluate the potential for iterative improvements

If guardrail metrics are hit but targets aren't:

1. Investigate the cause of the guardrail metric violation
2. Consider a limited rollout to segments where performance is positive
3. Develop mitigation strategies for the negatively impacted areas

For inconclusive results:

1. Extend the experiment duration if trends are promising but not statistically significant
2. Conduct qualitative research to uncover insights not captured in the quantitative data
3. Consider alternative feature variations based on learnings from the initial test

To ensure alignment, I'd engage:

• Product team to assess feature modifications
• Data science to dive deeper into user behavior changes
• UX research to gather qualitative insights
• Business development to understand revenue implications
• Community management to gauge creator sentiment

Step 9

Recommendation and Next Steps (3 minutes)

Based on our analysis, my recommendation would be to proceed with caution and implement a modified version of the feature that balances the gains in user engagement with the need to support our creator ecosystem.

Next steps:

1. Implement the version of the AI algorithm that showed the best balance between creator retention and user engagement in our A/B/C test.

2. Launch a creator support initiative alongside the feature rollout, including:

   • Educational content on optimizing for the new algorithm
   • Highlighted spaces for emerging creators to gain visibility
   • Feedback mechanisms for creators to report concerns

3. Conduct a series of creator roundtables to gather qualitative feedback and ideas for improvement.

4. Develop a long-term creator acquisition and retention strategy to ensure a healthy content ecosystem.

5. Plan a follow-up experiment testing personalized creator recommendations to viewers based on their engagement patterns.

These steps consider implications for:

• Related features: We'll need to review and potentially adjust other discovery and engagement features to ensure consistency.
• Broader ecosystem: This change may impact how users interact with other Facebook products, requiring coordination with other teams.
• Long-term strategy: We're setting a precedent for balancing algorithm-driven engagement with creator empowerment, which will inform future product decisions.

To drive buy-in and efficient implementation, I'll:

• Present findings and recommendations to leadership, emphasizing both short-term gains and long-term ecosystem health.
• Collaborate with engineering to ensure smooth technical implementation and monitoring.
• Work with marketing and creator relations teams to communicate changes effectively to our creator community.
• Establish regular check-ins with cross-functional teams to track progress and quickly address any issues that arise.

Expand Your Perspective

• How might this trade-off compare to similar challenges faced by other content platforms? For example, YouTube's changes to its recommendation algorithm in 2019 led to similar debates about creator visibility versus viewer engagement.

• Considering the rise of short-form video content, how might this decision impact Facebook's competitive position against platforms like TikTok in the next 2-3 years?

• What alternative approaches could we consider to boost both creator retention and user engagement simultaneously? For instance, could a mentorship program pairing established creators with newcomers create a win-win situation?

Related Topics

• Content Creator Monetization Strategies: Exploring ways to incentivize and reward creators for producing engaging content.

• Algorithmic Fairness in Content Distribution: Examining the ethical implications of AI-driven content suggestion systems.

• User Segmentation and Personalization in Social Media: Tailoring experiences for different user types to maximize engagement and satisfaction.

• Cross-platform Content Ecosystem Management: Strategies for maintaining a healthy balance across multiple interconnected products (e.g., Facebook, Instagram, WhatsApp).

• Long-term Retention Metrics for Social Platforms: Developing and tracking indicators of sustained user engagement and platform health over extended periods.