Why has Ocrolus's document classification accuracy rate dropped by 5% over the last month for mortgage applications?

Introduction

Ocrolus's document classification accuracy rate for mortgage applications has dropped by 5% over the last month, signaling a critical issue that demands immediate attention. This decline in accuracy could have far-reaching implications for the company's core product offering and customer satisfaction. I'll approach this problem systematically, focusing on identifying the root cause, validating hypotheses, and developing both short-term fixes and long-term solutions.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might have been a recent system update. Has there been any significant changes to the document classification algorithm or infrastructure in the past month?

Why it matters: System changes often correlate with performance shifts. Expected answer: Yes, there was a minor update to improve processing speed. Impact on approach: If confirmed, I'd focus on regression testing and rollback options.

• Considering user segments, I'm curious about the distribution of this accuracy drop. Is the 5% decrease uniform across all types of mortgage applications, or is it more pronounced in certain categories?

Why it matters: Uneven distribution could point to specific document types or user segments affected. Expected answer: The drop is more significant in jumbo loan applications. Impact on approach: I'd prioritize investigating factors unique to jumbo loan documentation.

• Given the nature of document classification, I'm wondering about recent changes in input data. Have there been any notable shifts in the types or quality of documents being submitted for mortgage applications?

Why it matters: Changes in input data can significantly impact machine learning model performance. Expected answer: There's been an increase in scanned documents versus digital uploads. Impact on approach: I'd focus on image preprocessing and OCR capabilities.

• Thinking about external factors, I'm considering market dynamics. Has there been a significant increase in mortgage application volume or complexity in the past month?

Why it matters: Sudden changes in volume or complexity can strain system resources and affect accuracy. Expected answer: Mortgage applications have increased by 20% due to a drop in interest rates. Impact on approach: I'd investigate scalability issues and potential resource constraints.