How did 6sense's Predictive Analytics model accuracy fall below 85% for the first time in Q2?

Introduction

The sudden drop in 6sense's Predictive Analytics model accuracy below 85% in Q2 is a critical issue that demands immediate attention. This analysis will systematically investigate the root cause, considering both internal and external factors that could have contributed to this unprecedented decline in performance.

To address this problem, I'll follow a structured approach that includes clarifying the context, ruling out external factors, understanding the product and user journey, breaking down the metric, gathering and prioritizing data, forming hypotheses, conducting root cause analysis, and proposing validation methods and next steps.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might have been a significant change in the data input or model architecture. Has there been any recent update to the data sources or model structure?

Why it matters: Changes in data or model structure could directly impact accuracy. Expected answer: Yes, there was a recent update to include new data sources. Impact on approach: If confirmed, we'd focus on data integration and model recalibration.

• Given the specificity of the 85% threshold, I'm curious about the historical performance. Has the model consistently performed above 85% in previous quarters?

Why it matters: Understanding the historical context helps identify if this is an anomaly or part of a trend. Expected answer: Yes, the model has consistently performed above 85% for the past two years. Impact on approach: If confirmed, we'd look for recent changes or external factors that could have caused this sudden drop.

• Considering the nature of predictive analytics, I'm wondering about the lead time of predictions. Has there been any change in the prediction window that could affect accuracy?

Why it matters: Changes in prediction timeframes can significantly impact model performance. Expected answer: No changes to the prediction window have been made. Impact on approach: If confirmed, we'd focus on other factors affecting model accuracy.

• Given the potential impact on customers, I'm interested in the user feedback. Have we seen any increase in customer complaints or reported inaccuracies?

Why it matters: User feedback can provide valuable insights into real-world model performance. Expected answer: There has been a slight increase in reported inaccuracies from key accounts. Impact on approach: If confirmed, we'd prioritize investigating these specific cases for patterns.

• Considering potential system issues, I'm curious about our monitoring processes. Have there been any alerts or anomalies detected in our model performance tracking systems?

Why it matters: System issues could lead to inaccurate reporting of model performance. Expected answer: No system alerts have been triggered, but there was a brief outage in our monitoring system last month. Impact on approach: If confirmed, we'd investigate the impact of the monitoring outage on our accuracy measurements.