Home / Hallucinations: what the data says and what helps
Fluent, false legal output — invented cases, misquoted statutes, real citations behind wrong conclusions. The peer-reviewed numbers are sobering; the defences are measurable.
Three species recur. Invented authorities — cases and patents that do not exist (SallyIP’s adversarial traps include a fake Federal Circuit decision and a zero-number patent; both refused). Altered quotations — real sources quoted with changed words (caught word-for-word in ablation, 2/2 with zero false flags). Unsupported attachment — real citations behind claims they do not support (the P0 run’s 81.0% unsupported-proposition rate, published with the release blocked). Each species needs a different gate, which is why single-score "accuracy" marketing misleads.
SallyIP’s adversarial runs show what healthy refusal looks like: 14 grounded refusals across 23 ABIGAIL cases where offline evidence could not support an answer, and 6 conservative refusals in 12 live traps. Each refusal is logged with the evidence state that caused it — reviewers see what was missing, not just that the system declined. A tool that has never refused has either never been tested or never admits it. Refusal telemetry is preserved the same way quote telemetry is: de-quoted, recorded, and counted in the next run’s denominators.