Introduction
Harvey Knowledge answers legal research questions with citations to cases, statutes and other authorities. Lawyers can flag an answer if a citation looks wrong: the case doesn't support the point, the quote is inaccurate, or the citation format is off. Two weeks after Harvey upgraded the underlying model and retrieval pipeline for Knowledge, the citation-flag rate doubled, from 0.6% to 1.2% of research answers. Automated validation shows no increase in non-existent citations.
This is a hard root cause question because "citation problems" in legal AI can come from many layers (model, retrieval, content, interface or user behaviour), and the stakes are high. I'll work through it methodically.
Clarify, rule out external causes, understand the product, decompose the metric, gather data, form hypotheses, find the root cause, validate, decide, and resolve.
Step 1
Clarifying Questions (3 minutes)
Why it matters: Tells us which layers to inspect. Expected answer: A new base model, a change in how passages are chunked and retrieved, and a new answer format that places citations inline rather than in a list at the end. Impact on approach: Three possible causes, all shipped together.
Why it matters: Flag categories narrow the problem. Expected answer: Users choose a reason: "doesn't support the point" rose most, "wrong pinpoint page" also rose; "case doesn't exist" did not. Impact on approach: Focus on support and pinpoint accuracy, not fabrication.
Why it matters: Jurisdictions and practice areas use different sources. Expected answer: Not yet analysed. Impact on approach: Segment early.
Why it matters: A rate can rise because of the mix of questions. Expected answer: Research volume rose 15% as new customers onboarded. Impact on approach: Check mix effects.
Practice similar questions
Subscribe to access the full answer
.png){kind=logo}
