What factors are causing the increased error rate in Icertis's AI-powered contract risk scoring feature this month?

Introduction

The increased error rate in Icertis's AI-powered contract risk scoring feature this month presents a critical challenge that demands immediate attention and a systematic approach to resolution. As we delve into this issue, we'll employ a structured framework to identify, validate, and address the root cause while considering both short-term fixes and long-term strategic implications.

Step 1

Clarifying Questions (3 minutes)

• Looking at the timing, I'm thinking there might be a recent change in the AI model or data pipeline. Has there been any recent update to the AI model or data processing systems?

Why it matters: Recent changes often correlate with performance shifts. Expected answer: Yes, there was a model update two weeks ago. Impact on approach: If confirmed, we'd focus on the model update as a primary suspect.

• Considering user segments, I'm wondering if this is a universal issue or specific to certain contract types. Are we seeing the increased error rate across all contract types or is it concentrated in specific areas?

Why it matters: Helps narrow down the scope of the problem and potential causes. Expected answer: The issue is more prevalent in complex, multi-party contracts. Impact on approach: We'd investigate factors specific to complex contracts and their processing.

• Given the nature of AI systems, I'm curious about the training data. Have there been any significant changes in the volume or nature of contracts being processed recently?

Why it matters: Changes in input data can significantly affect AI model performance. Expected answer: There's been a 30% increase in international contracts over the past month. Impact on approach: We'd examine how the model handles international contracts and if this increase has impacted performance.

• Thinking about system dependencies, I'm wondering about any changes in integrated systems. Have there been any updates or issues with systems that feed data into the risk scoring feature?

Why it matters: External dependencies can impact feature performance. Expected answer: The document parsing system was upgraded last week. Impact on approach: We'd investigate the interaction between the parsing system and the risk scoring feature.