Home / Retrieval benchmarking without fabricated ground truth
Retrieval benchmarking without fabricated ground truth
SallyIP’s offline retrieval bench runs 19 patent-family cases through 120 checks. It passes 99 — and the 21 failures are disclosed as metric and dataset issues, not hidden.
Results (2026-09-10, offline deterministic engine, no model)
- Overall: 99/120 checks pass; adversarial spot checks 4/4 PASS
- Family-resolution 0/19 — a metric bug, disclosed: the bench wipes priority numbers by design, then expects INPADOC-grade family reunion the merger cannot compute. Fix direction: seed real priorities or downgrade the metric to report-only.
- Corpus overlap: two cases target the same application at grant vs publication stage; the engine correctly groups them and the bench counts it as a false merge — a curation issue, not an engine bug.
- BLOCKED, never skipped: recall@k / precision@k / ground-truth rank need live USPTO/EPO searches (provider keys).
Harvest honesty: Google Patents harvesting throttles (503s); the harvester resumes with backoff rather than imputing missing data. Free-tier model 429/503s excluded items from denominators, never imputed.
Adversarial spot checks
Inside the same bench, four adversarial checks pass 4/4: forced merges correctly refused, shared-priority members correctly united, and garbage inputs handled safely. These checks guard the merger logic that the family-resolution metric mis-measures — small, deterministic, and honest about what they cover.
What recall-at-k still needs
True recall and precision against live patent offices need provider keys and live searches — currently BLOCKED, listed openly rather than approximated with the offline corpus. The offline bench measures normalization and merging deterministically; ranking quality against live results awaits the keyed run, and the methodology file says so.
Run references: offline bench 2026-09-10 (benchmarks/cross-bench-report.md §1, §5); retrieval dataset v1-frozen, 19 cases.
All benchmarks · ablation study