Home / Hallucination, measured — not marketed

Hallucination, measured — not marketed

Peer-reviewed research finds general LLMs hallucinate on most tested legal queries. SallyIP measures its own grounding mechanics on frozen benchmarks and publishes the misses alongside the wins.

What peer review found

General LLMs (GPT-3.5/4, Llama 2, PaLM 2): 58–88% hallucination on legal queries (Dahl et al., Stanford HAI 2024).
Purpose-built legal RAG tools: 17–33% — Lexis+ AI ~17%, Westlaw ~33%, CoCounsel-lineage ~17% with 60%+ refusals (Magesh et al., Stanford/JELS 2024–2025; tested versions may differ).
Harvey, Genie AI, Solve Intelligence, DeepIP: no public hallucination measurements.

SallyIP grounding benchmark 100 (run f8dfe146, gemini-flash-lite-latest, 2026-09-09)

Authority recall: 100% (95/95); zero dangling citations: 100% (95/95)
Exact-quote verification: 72.5% (87/120); unverified quotes 22.5% (27/120) — flagged, never silently passed
Frozen P0 full run: citation integrity 100% of scored (98/98) with entailment 66.7% (FAIL) and unsupported-proposition 81.0% (FAIL) — release BLOCKED

Unverified quotes are not deleted quietly: they are de-quoted with telemetry preserved, so the failure stays measurable release after release instead of vanishing into a rephrased answer.

Mechanical verdicts are regression signals, not lawyer review. No smoke test populates a headline; every metric carries sample size and headline eligibility. Practitioner grading: PENDING (0/25) — no legal-correctness score exists for these answers, and none is shown until humans grade.

Data: latest.json · headline-metrics.csv · why integrity ≠ correctness

Can AI hallucinate legal citations?

Yes — studies show most tested legal queries fail on general models, and purpose-built tools still miss a substantial share. SallyIP counters with mechanical citation and quote checks plus published miss rates, and refuses or qualifies when evidence is missing.

All benchmarks