Why has the average experiment runtime in Optimizely's Full Stack product increased by 35% this quarter compared to last?

Introduction

The recent 35% increase in average experiment runtime for Optimizely's Full Stack product is a concerning trend that requires immediate attention. This analysis will systematically investigate potential root causes, generate data-driven hypotheses, and propose a comprehensive plan to address the issue. We'll examine both internal and external factors, considering technical, user behavior, and product-related aspects to ensure a holistic approach.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a recent product update. Has there been any significant change to the Full Stack product in the last quarter?

Why it matters: Recent changes could directly impact experiment runtime. Expected answer: Yes, there was a major update. Impact on approach: If yes, we'd focus on the update's impact; if no, we'd look at gradual changes or external factors.

• Considering user segments, I'm wondering if this increase is uniform across all customers. Are we seeing this 35% increase consistently across different customer types or usage levels?

Why it matters: Helps identify if it's a global issue or specific to certain user groups. Expected answer: The increase varies across customer segments. Impact on approach: If consistent, we'd look at system-wide issues; if varied, we'd investigate segment-specific factors.

• Thinking about performance metrics, I'm curious about other related KPIs. Have we noticed any changes in other metrics like experiment creation rate or user engagement during this period?

Why it matters: Correlated changes in other metrics could point to broader issues. Expected answer: Some related metrics have also changed. Impact on approach: If yes, we'd investigate potential systemic issues; if no, we'd focus more on runtime-specific factors.

• Considering data integrity, I want to ensure we're comparing apples to apples. Has there been any change in how we measure or define "experiment runtime" in the last quarter?

Why it matters: Ensures the observed increase is real and not due to measurement changes. Expected answer: No changes in measurement methodology. Impact on approach: If changed, we'd need to recalibrate our analysis; if not, we can proceed with our current understanding.