Why has UserTesting's Live Conversation feature seen a 20% drop in usage over the past month?

Introduction

The recent 20% drop in usage of UserTesting's Live Conversation feature over the past month is a concerning trend that requires immediate attention. This analysis will systematically identify, validate, and address the root cause while considering both short-term and long-term implications for the product.

I'll approach this issue by first clarifying key details, ruling out external factors, and then diving deep into product understanding, metric breakdown, and data analysis. From there, I'll form hypotheses, conduct root cause analysis, and propose validation methods and solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a seasonal component. Has this 20% drop been compared to the same period last year?

Why it matters: Seasonal trends could explain the drop and influence our solution approach. Expected answer: No significant seasonal pattern observed in previous years. Impact on approach: If seasonal, we'd focus on anticipating and mitigating annual fluctuations.

• Considering user segments, I'm curious if this drop is uniform across all user types. Have you noticed any particular user segment more affected than others?

Why it matters: Identifying affected segments helps pinpoint specific issues and tailor solutions. Expected answer: Enterprise users showing a larger decrease compared to other segments. Impact on approach: We'd investigate enterprise-specific factors and potentially prioritize solutions for this segment.

• Thinking about recent changes, were there any updates to the Live Conversation feature or related systems in the weeks preceding the drop?

Why it matters: Recent changes could directly correlate with the usage decline. Expected answer: A minor UI update was implemented three weeks ago. Impact on approach: We'd scrutinize the update's impact and consider rolling back if necessary.

• Regarding the metric itself, has there been any change in how we measure or define "usage" for Live Conversation?

Why it matters: Ensures we're comparing apples to apples and not facing a data anomaly. Expected answer: No changes to the metric definition or measurement system. Impact on approach: If changed, we'd need to recalibrate our analysis based on the new definition.