You're the PM for Facebook Stories. A new feature you just released led to a 5% decrease in the number of active creators but a 10% increase in the number of daily active users. Would you keep or roll back this feature?

Introduction

I'm facing an interesting trade-off with Facebook Stories: a new feature has led to a 5% decrease in active creators but a 10% increase in daily active users. This presents a classic product dilemma where we're seeing opposing movements in two critical metrics. I'll analyze whether to keep or roll back this feature by examining the metrics in context, understanding the ecosystem impacts, and developing a framework for making this decision.

Step 1

Clarifying Questions (3 minutes)

• Looking at the feature implementation, I'm thinking this might be a significant UI or algorithmic change. Could you share more details about what this new feature actually is and how it changed the Stories experience?

• Why it matters: Understanding the feature's nature helps identify the mechanism causing both positive and negative effects
• Expected answer: It's a recommendation algorithm change that surfaces more popular content
• Impact on approach: Would focus analysis on content diversity and creator incentives

• Regarding our business objectives, I'm assuming Stories has dual goals around creator engagement and viewer retention. Where does Facebook Stories currently sit in terms of strategic priorities - is it more focused on creator growth, user engagement, or monetization at this stage?

• Why it matters: Helps weigh the relative importance of the metrics moving in opposite directions
• Expected answer: User engagement is currently the primary focus
• Impact on approach: Would prioritize the 10% DAU increase over creator decrease if engagement is the current priority

• On the user impact side, I'm curious about the quality of engagement we're seeing. Beyond the 10% DAU increase, do we have data on whether session length, interactions per session, or content consumption metrics have changed?

• Why it matters: Helps determine if we're seeing deeper engagement or just more frequent, shallow visits
• Expected answer: Engagement depth metrics have improved alongside DAU
• Impact on approach: Would strengthen case for keeping the feature if engagement quality improved

• Regarding the creator ecosystem, I'm wondering about the nature of the 5% creator decrease. Do we know if this is affecting a particular segment of creators (casual vs. power users) and whether it's impacting content volume or diversity?

• Why it matters: Helps assess long-term ecosystem health implications
• Expected answer: Primarily affecting casual creators while power creators remain stable
• Impact on approach: Would focus on mitigating impacts for casual creators while maintaining the feature

• From a timeline perspective, how long has this feature been live, and do we have enough data to determine if these changes represent a new stable state or if they're still in flux?

• Why it matters: Determines confidence level in the observed metrics and whether more time is needed
• Expected answer: Feature has been live for 2-3 weeks with stable metrics for the past week
• Impact on approach: Would influence whether to make an immediate decision or continue monitoring

Step 2

Trade-off Type Identification (1 minute)

Identify which sub-type of trade-off question you're dealing with:

This is primarily a Type B trade-off: same product with different variations. We're evaluating a feature change within Facebook Stories that's creating tension between creator participation and user engagement.

This identification is crucial because it frames our approach around optimizing a single product experience rather than managing competition between products. The core challenge is finding the right balance within the Stories ecosystem itself, not allocating resources between competing products.

This trade-off type suggests we should focus on:

1. Understanding how the feature change altered the core Stories experience
2. Analyzing whether the creator decrease and user increase are causally linked
3. Determining if we can modify the feature to maintain user gains while recovering creator participation

Step 3

Product Understanding (5 minutes)

Facebook Stories is an ephemeral content format that allows users to share photos and videos that disappear after 24 hours. As a product manager for this feature, I need to understand its core components:

• Core Features:

  • Creation tools (camera, filters, effects, text)
  • Publishing mechanism (to Stories feed)
  • Consumption experience (tappable, sequential viewing)
  • Interaction capabilities (reactions, direct replies)
  • Discovery mechanisms (friend Stories, suggested Stories)

• Key Stakeholders:

  • Creators (both casual and power users who post Stories)
  • Consumers (users who view but may not create Stories)
  • Advertisers (who place ads between Stories)
  • Facebook (platform seeking engagement and monetization)

• Value Proposition:

  • For creators: Low-friction, ephemeral self-expression
  • For consumers: Authentic, timely updates from connections
  • For Facebook: Increased engagement, session frequency, and ad inventory

• Alignment with Company Mission:

  • Stories supports Facebook's mission to connect people by enabling more frequent, casual sharing
  • It counters the "highlight reel" problem of the News Feed by encouraging everyday moments
  • It helps Facebook compete with other platforms that pioneered the Stories format

• User Journey:

  • Creator journey: Capture moment → Apply creative tools → Publish → View analytics → Engage with responses
  • Consumer journey: Discover Stories → View content → React/respond → Continue browsing or exit

The ecosystem is particularly delicate because Stories requires both creators and viewers to thrive. Without enough creators, viewers have limited content to engage with. Without enough viewers, creators lack incentive to publish. This creates a flywheel effect where more creators attract more viewers, which in turn encourages more creation.

The new feature appears to have disrupted this balance by somehow favoring consumption over creation. This could be through changes to the discovery algorithm, the viewing experience, or the creation flow.

Step 4

Trade-off Agreement and Hypothesis (5 minutes)

The core trade-off we're facing is between creator participation and user engagement in Facebook Stories. Based on the metrics, we're seeing a 5% decrease in active creators alongside a 10% increase in daily active users.

My hypothesis for why we're seeing these results:

The new feature likely changed how Stories content is surfaced or consumed, potentially prioritizing popular or engaging content from a smaller subset of creators. This would explain why we're seeing more users engaging (they're finding more relevant content) but fewer creators participating (as visibility may have become more concentrated).

Specific hypotheses about the feature:

1. It may have introduced a more algorithmic approach to Stories ranking rather than purely chronological display
2. It could have changed the discovery mechanism to highlight Stories with higher engagement
3. It might have altered the UI to make consumption easier but creation more complex

Potential impacts

For different user types:

• Power creators: May benefit from increased visibility if their content is favored by the algorithm
• Casual creators: Likely experiencing reduced visibility, leading to lower motivation to create
• Active consumers: Benefiting from more relevant content surfaced to them
• Casual consumers: May be more likely to engage due to improved content quality

If we continue with the feature unchanged:

• Best case: Creator decline stabilizes as the most engaged creators remain, while user growth continues
• Worst case: Creator decline accelerates as more creators become discouraged, eventually leading to content shortages and subsequent user decline

If we roll back the feature:

• Best case: Creator numbers recover while we find alternative ways to drive user engagement
• Worst case: We lose the user engagement gains and struggle to find another growth lever

This represents a classic platform dilemma: optimizing for consumption can sometimes come at the expense of creation, but the platform needs both sides of the marketplace to thrive long-term.

Step 5

Key Metrics Identification (4 minutes)

North Star Metric

For Facebook Stories, the North Star metric should be Daily Stories Interactions Per User. This metric captures both creation and consumption behaviors, aligning with the platform's goal of driving meaningful engagement through Stories.

This North Star intersects value for all sides:

• For creators: More interactions mean more engagement with their content
• For consumers: More interactions indicate they're finding valuable content
• For Facebook: More interactions drive session frequency and ad opportunities

Supporting Metrics

1. Creator Retention Rate (7-day)

   • Why it's important: Measures our ability to keep creators actively posting
   • Stakeholder relevance: Critical for maintaining content supply and diversity
   • Type: Leading indicator of ecosystem health

2. Stories Posted Per Creator

   • Why it's important: Measures creator engagement intensity
   • Stakeholder relevance: Indicates creator satisfaction and platform stickiness
   • Type: Leading indicator of content supply

3. Stories Viewed Per User

   • Why it's important: Measures consumption depth
   • Stakeholder relevance: Indicates content relevance and user satisfaction
   • Type: Leading indicator of user engagement

4. Stories Completion Rate

   • Why it's important: Measures content quality and relevance
   • Stakeholder relevance: Indicates value delivery to both creators and viewers
   • Type: Leading indicator of content-market fit

5. New Creator Activation Rate

   • Why it's important: Measures ecosystem growth and creator funnel health
   • Stakeholder relevance: Indicates platform accessibility and creation incentives
   • Type: Leading indicator of ecosystem sustainability

6. Weekly Retention Rate (All Users)

   • Why it's important: Measures overall product stickiness
   • Stakeholder relevance: Indicates long-term platform health
   • Type: Lagging indicator of overall product value

7. Ad Revenue Per User

   • Why it's important: Measures business impact
   • Stakeholder relevance: Connects user engagement to business outcomes
   • Type: Lagging indicator of monetization effectiveness

These metrics provide a balanced view of both sides of the Stories ecosystem. By monitoring them together, we can detect imbalances early and make adjustments before they impact the overall health of the product.

Step 6

Experiment Design (3 minutes)

To validate our hypotheses about the feature's impact, I'd design an A/B/C test with the following structure:

Experiment Hypothesis: The new feature increases user engagement by surfacing more relevant content but reduces creator participation by concentrating visibility. A modified version could maintain user engagement gains while mitigating creator losses.

Test Groups:

• Group A (Control): Original Stories experience before the new feature
• Group B (Current Feature): The new feature as currently implemented
• Group C (Modified Feature): A modified version of the feature that attempts to balance creator visibility with content relevance

Target Audience:

• 5% of users in Group A (baseline)
• 5% of users in Group B (current feature)
• 5% of users in Group C (modified feature)
• The remaining 85% would continue with the current feature since it's already launched

Duration: 2 weeks minimum to account for weekly usage patterns and allow for stabilization

Key Considerations:

• Randomization: Ensure users are randomly assigned across groups with balanced demographics and usage patterns
• Sample Size: Each group should include at least 100,000 users to ensure statistical significance for both creator and consumer metrics
• Novelty Effects: Run the test long enough to get past initial novelty reactions, especially for Group C

Guardrail Metrics:

• Creator retention should not drop below 90% of control
• Content diversity (unique creators seen per user) should not drop below 95% of control
• User satisfaction (measured via in-product surveys) should not decrease

This experiment design allows us to:

1. Validate the impact of the current feature against the original experience
2. Test if our modified approach can achieve the "best of both worlds"
3. Make a data-driven decision about whether to keep, modify, or roll back the feature

Step 7

Data Analysis Plan (3 minutes)

To evaluate the experiment results effectively, I would analyze the following data:

Primary Analysis

1. Metric Comparison Across Groups

   • Compare all key metrics identified earlier across the three test groups
   • Apply statistical significance testing to ensure differences aren't due to chance
   • Calculate effect sizes to understand the magnitude of differences

2. Segment Analysis

   • Break down results by creator segments:
     • Power creators (top 10% by posting frequency)
     • Regular creators (middle 40%)
     • Casual creators (bottom 50%)
   • Analyze user segments:
     • Heavy consumers (top 20% by viewing time)
     • Regular consumers (middle 60%)
     • Light consumers (bottom 20%)
   • Geographic and demographic segments to identify any disparate impacts

3. Cohort Analysis

   • Track new creators who joined during the test period to measure activation and retention
   • Analyze existing creators' behavior changes over time
   • Examine user retention patterns across cohorts

Correlation Studies

• Measure correlation between creator decline and user growth to determine if they're directly related
• Analyze whether increased user engagement leads to higher creator retention over time
• Examine the relationship between content diversity and user satisfaction

Anomaly Detection

• Look for unexpected patterns in the data that might indicate unintended consequences
• Identify any creator segments that are particularly affected (positively or negatively)
• Monitor for any unusual spikes or drops in key metrics that might indicate external factors

Interpreting Mixed Results

When metrics move in opposite directions, I would:

1. Weight them according to their alignment with our strategic priorities
2. Consider short-term vs. long-term implications
3. Evaluate whether the trade-off is sustainable or will lead to ecosystem collapse
4. Determine if the modified version (Group C) achieves a better balance

For example, if Group C shows only a 7% DAU increase but only a 2% creator decrease, this might represent a better overall outcome than Group B's 10% DAU increase with a 5% creator decrease.

Step 8

Decision Framework (4 minutes)

To make a structured decision about whether to keep or roll back the feature, I'll use the following framework:

Primary Decision Criteria

1. Ecosystem Health Index = (Creator Retention × Content Volume × Content Diversity)
2. User Value Index = (DAU × Stories Viewed Per User × Completion Rate)
3. Business Impact = (Ad Impressions × Ad Engagement Rate)

Decision Tree

Red Flags / Deal-Breakers

• Creator decline exceeding 10% with no signs of stabilization
• Content diversity dropping below 80% of baseline
• New creator activation falling below 70% of baseline
• User satisfaction scores dropping significantly despite engagement increases

Mixed Results Scenarios

• If user metrics improve but creator metrics worsen: Implement creator incentives while keeping the feature
• If both metrics stabilize at new levels: Keep the feature but monitor long-term ecosystem health
• If metrics are inconclusive: Extend testing period or test with larger sample

Cross-Functional Considerations

• Engineering: Assess technical complexity of feature modifications
• Design: Evaluate UX improvements that could better balance creation and consumption
• Data Science: Develop predictive models for long-term ecosystem impacts
• Community: Gather qualitative feedback from creator communities
• Business: Evaluate revenue implications of different scenarios

This framework ensures we're making a holistic decision that considers both immediate metrics and long-term ecosystem health.

Step 9

Recommendation and Next Steps (3 minutes)

Based on the analysis framework I've outlined, my recommendation would be:

Keep the feature but implement targeted modifications to support creators while maintaining the user engagement gains.

This recommendation is based on:

1. The significant 10% DAU increase represents substantial user value
2. The 5% creator decrease, while concerning, may be manageable with the right interventions
3. The Stories ecosystem likely has enough creator capacity to absorb some reduction without immediate content shortages

Next Steps

1. Implement Creator Visibility Enhancements

   • Modify the algorithm to ensure broader creator visibility while maintaining content relevance
   • Develop creator analytics to help them understand how to succeed in the new environment

2. Launch Creator Incentive Program

   • Introduce features that reward consistent creation (e.g., "Creator Highlights")
   • Provide exclusive creative tools for regular contributors

3. Develop Segmented Experience

   • Test personalization that balances algorithm-driven content with chronological friend content
   • Create different experiences for heavy creators vs. casual creators

4. Establish Ecosystem Health Monitoring

   • Implement weekly dashboards tracking creator retention and diversity metrics
   • Set up automatic alerts for concerning trends in the creator ecosystem

5. Research Creator Motivations

   • Conduct qualitative research with churned creators to understand their decision to stop posting
   • Use insights to inform future feature development

This balanced approach recognizes that both creators and consumers are vital to the Stories ecosystem. By keeping the feature that drives user engagement while actively addressing creator concerns, we can maintain a healthy platform that continues to grow and evolve.

Expand Your Perspective

• Industry analogies: YouTube faced a similar challenge when they shifted from chronological to algorithmic recommendations. They ultimately succeeded by creating specialized tools for creators to understand the algorithm and optimize their content. Similarly, TikTok has managed to balance discovery of popular content with surfacing new creators through their "For You" page algorithm.

• Future trend implications: As social platforms mature, the tension between creation and consumption will likely intensify. We might need to evolve toward more explicitly separated experiences for creators versus viewers, similar to how streaming platforms have distinct interfaces for content uploaders versus subscribers.

• Alternative approaches: Instead of modifying the algorithm, we could consider a completely different approach like a tab-based interface that separates "Friends Stories" from "Discover Stories," giving users explicit control over their consumption preferences while ensuring creators maintain a guaranteed audience.

Related Topics

• Direct product strategy connections: Platform ecosystem balancing strategies for two-sided marketplaces, particularly how to manage supply (creators) and demand (viewers) in content platforms

• Technical architecture implications: Recommendation system design that balances content diversity with engagement optimization, including exploration vs. exploitation algorithms

• User experience considerations: Creation flow optimization to reduce friction and increase creator retention, particularly for casual creators who may be more sensitive to reduced visibility

• Cross-functional collaboration frameworks: How product, data science, and community teams can work together to monitor and maintain ecosystem health metrics

• Metrics evolution strategies: Developing composite metrics that capture both sides of a platform ecosystem, moving beyond simple engagement metrics to more holistic health indicators