What factors are contributing to the sudden increase in error rates for Everlaw's predictive coding functionality this quarter?

Introduction

The sudden increase in error rates for Everlaw's predictive coding functionality this quarter is a critical issue that demands immediate attention. Predictive coding is a core feature of Everlaw's e-discovery platform, enabling efficient document review and classification. To address this problem, I'll employ a systematic approach to identify, validate, and resolve the root cause while considering both short-term fixes and long-term implications.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a recent change in the system. Has there been any significant update to the predictive coding algorithm or underlying infrastructure in the past quarter?

Why it matters: Recent changes often correlate with performance issues. Expected answer: Yes, there was a major algorithm update. Impact on approach: If confirmed, we'd focus on the update's impact and potential rollback strategies.

• Considering user segments, I'm curious about the error distribution. Are we seeing this increase across all user types, or is it concentrated in specific segments or use cases?

Why it matters: Helps narrow down if it's a global issue or specific to certain users or data types. Expected answer: The issue is more prevalent in cases with large, diverse datasets. Impact on approach: If confirmed, we'd investigate how dataset characteristics interact with the algorithm.

• Thinking about performance metrics, I'm wondering about the magnitude of the increase. Can you quantify the error rate change – for example, has it doubled, tripled, or increased by an order of magnitude?

Why it matters: The scale of the problem influences the urgency and type of response needed. Expected answer: Error rates have increased by approximately 50%. Impact on approach: This would help prioritize the issue and determine the resources needed for resolution.

• Considering potential measurement issues, I'm curious about our error detection methods. Have there been any changes to how we measure or define errors in the predictive coding process?

Why it matters: Ensures we're dealing with a real increase in errors, not a change in measurement. Expected answer: No changes in error measurement or definition. Impact on approach: If confirmed, we can focus on actual performance issues rather than measurement discrepancies.