Build the Harness, Not the Disclaimer
Description
A constructive sequel to The Calibration Gap. Since verification is a luxury the underserved can't supply, build it into the system: a harness that routes every sensitive (health/legal) claim through reliable search APIs and authoritative sources, verifies to a statistical standard, abstains when it can't confirm, and concentrates its checks on case-law citations (where hallucination clusters) while trusting reliably-relayed public statutes. The grounding cure already exists (RAG slashes hallucination); the move is to make it mandatory and to invert the industry's liability dodge — taking responsibility ON rather than disclaiming it away.
Sources & further reading (8)
- Stanford: Assessing legal AI research tools (hallucinations)https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Hallucinations.pdf
- npj Digital Medicine: clinical safety & hallucination rates of LLMshttps://www.nature.com/articles/s41746-025-01670-7
- arXiv: Reducing hallucination in structured outputs via RAGhttps://arxiv.org/pdf/2404.08189
- PMC: MEGA-RAG — multi-evidence RAG for public-health hallucination mitigationhttps://pmc.ncbi.nlm.nih.gov/articles/PMC12540348/
- Mata v. Avianca, Inc. (fabricated AI citations, sanctions)https://en.wikipedia.org/wiki/Mata_v._Avianca,_Inc.
- Science: Sycophantic AI decreases prosocial intentions and promotes dependencehttps://www.science.org/doi/10.1126/science.aec8352
- LSC Justice Gap Report 2025 — Executive Summaryhttps://justicegap.lsc.gov/resource/executive-summary/
- Frontiers in AI: hallucinations and prompting strategies (2025)https://www.frontiersin.org/journals/artificial-intelligence/articles/10.3389/frai.2025.1622292/full
Script
Cold open
Last time, two people and one chatbot — and the tenant who couldn't afford a lawyer, holding an eviction notice. We said the missing skill was calibration: knowing when to believe the machine. But then we admitted the cruel part — verification is a luxury the people who need it most can't afford. So tonight, a different question. What if we stop asking THEM to verify… and build a machine that verifies itself — and stands behind the answer?
Frame
Right now the industry's posture on anything that matters is a disclaimer. Every health and legal AI ends the same way: this is not advice, consult a professional. That sentence is an abdication — it hands the risk back to the person least equipped to carry it. The proposal tonight is the opposite: a verification HARNESS that takes on more responsibility, not less. Let's build it, one requirement at a time.
Why can't the raw chatbot be trusted on sensitive claims?
Start with why the naked chatbot can't be trusted here. Ask a general-purpose model a real legal question and hallucination runs fifty-eight to eighty-eight percent — confident, fabricated citations. In medicine, over sixty percent without grounding. And this isn't hypothetical: in twenty twenty-three, New York lawyers were fined five thousand dollars for filing a brief with six cases that ChatGPT simply invented. So the harness's first rule is brutal — treat the model as guilty until verified.
Does the cure already exist, just un-wired?
Now the hopeful part: the cure already exists, it's just not wired up by default. Feed a model the right documents and that same medical task drops from over sixty percent hallucination to under two. Retrieval-augmented systems have cut hallucinated steps from twenty-one percent to under seven. One public-health framework cut its error rate by over forty percent just by forcing multi-source evidence. The capability is sitting on the shelf. A harness is what takes it down and makes it mandatory.
What exactly is a verification harness, held to what standard?
So what IS a harness? Every sensitive claim gets broken into atomic facts, and each one is checked against reliable, authoritative sources through a search API BEFORE it reaches you. And not loosely — to a statistical standard: a confidence threshold, independent corroborating sources, and the discipline to abstain. If it can't verify, it doesn't assert; it says 'I couldn't confirm this,' and it reports its own measured error rate. Prompt tricks alone buy about twenty-two points. A harness is held to a number.
Where does law actually break — case law, not statutes?
And here's where law actually breaks — and it's not where you'd guess. The statute itself, the public text of the law, a model usually relays fine; it's published, it's all over its training data. Where it turns to soup is CASE LAW: fabricated citations, invented holdings, rulings it swears are still good law when they aren't. The Stanford study found the hallucinations cluster precisely in citation generation; Avianca was six fake cases in a single filing. So the harness spends its skepticism there — every case it cites gets checked against the real reporter: does it exist, does it say this, is it still binding? The Bar made exactly that a duty in twenty twenty-four.
Turn
And here's the turn. The whole industry is racing to take responsibility OFF the table — the disclaimer, the 'consult a professional,' the model that flatters you forty-nine percent more than a human ever would. The meritorious move is to run the other way: build the harness that takes responsibility ON. It owns its claims. It cites its sources. It publishes its error rate. It gets audited. It is the hard, unglamorous, liability-bearing engineering project that nobody wants — which is exactly why it's the one worth doing. We don't need braver users. We need accountable machines.
Closer
Back to the tenant and the eviction notice. The disclaimer version says: this is not legal advice, good luck. The harness version reads the notice, pins it to their state's actual statute, checks the real deadline against the current code, cites it, flags the one thing it couldn't confirm… and tells them exactly when a human has to step in. One of those takes the problem seriously. The other just takes itself off the hook.