Learn AI Create the Future — with Pranay
Main site

Legal and Compliance AI

Legal RAG and fabricated citations

Legal RAG grounds an AI's answer in retrieved case law, but the generated citations still need checking, because a fluent, correctly-formatted citation can point to a case that never existed.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. Where you have already seen it
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Legal RAG has an AI look up real case law before answering. It can still invent a citation that reads perfectly and points at nothing real.

A student writes an essay citing a source, "according to page 42 of this textbook". The citation looks completely legitimate. A teacher who checks the shelf might find no such book exists.

A legal AI can do the same thing: write "Smith v. Jones, 2019" in a confident, correctly-formatted sentence, when no such case exists. The fix is not trusting the format. It is checking the source.

Why it exists

A chatbot answering a legal question from memory alone is dangerous. It has read millions of legal documents during training. It can blend them into a citation that looks completely real, formatted exactly like a genuine one, while citing nothing.

RAG, retrieval-augmented generation, has the system search a real library of case law first. It then writes its answer using only what it actually found. This should stop the model from citing something it never looked up.

It does not fully solve the problem. The model can still slip in a citation from memory, alongside real ones it retrieved. Both look identical on the page.

How it works

  Question: "Can my landlord evict me without any notice?"

  Step 1: search real case law for relevant matches.
    retrieved: "Rai v. Sharma, 2018"  (similarity 0.55)

  Step 2: the model writes an answer using that case:
    "No. Under Rai v. Sharma, 2018, thirty days notice
     is required. This is also confirmed by Patel v.
     State Housing Board, 2017."

  Step 3: check every citation against what was actually retrieved.
    "Rai v. Sharma, 2018"              -> found in the library. VERIFIED.
    "Patel v. State Housing Board..."  -> never retrieved. FABRICATED.

The second citation was never in the retrieved material at all. It was invented, mixed in seamlessly next to a real one, in the exact same confident tone.

Where you have already seen it

  • Reported chatbot legal mishaps. Several documented court cases involve lawyers submitting AI-generated briefs citing cases that turned out not to exist.
  • Legal research tools. Products advertising "cited sources" for every legal answer are built specifically to make this checking step visible to the user.
  • AI news coverage. "AI hallucinated a fake citation" stories, in law and elsewhere, are all versions of this exact failure.

Remember this

  • Retrieval alone does not guarantee every citation in an answer is real.
  • A fabricated citation can look identical in format to a genuine one.
  • Every citation needs checking against what was actually retrieved, not trusted on sight.

What to learn next

  • What is RAG — the retrieval technique this lesson builds a safety check on top of.
  • Hallucination — the general failure mode fabricated citations are one specific case of.
  • Legal documents as data — structuring the case-law library this retrieval step searches.

Developer — Code and libraries.

This builds a tiny case-law search engine, then checks a generated answer's citations against exactly what that search actually returned.

Setup

bash
pip install scikit-learn

Retrieval, then citation verification

legal_rag_demo.py
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity

# A tiny "case law" library. A real system would hold thousands of real
# judgments; four is enough to show how the mechanics work.
corpus = {
    "Rai v. Sharma, 2018": "A landlord cannot evict a tenant without thirty days written notice under the state tenancy act.",
    "Mehta v. Union Bank, 2020": "A bank must disclose all processing fees before a loan agreement is signed.",
    "Iyer v. Fashion Retail Ltd, 2021": "An employer must provide written reason before terminating an employee of five years or more.",
    "Fernandes v. CityCorp, 2019": "A consumer contract clause is unenforceable if it was never actually shown to the customer.",
}

names = list(corpus.keys())
vectorizer = TfidfVectorizer().fit(corpus.values())
doc_vectors = vectorizer.transform(corpus.values())

def retrieve(query, k=2):
    q_vec = vectorizer.transform([query])
    scores = cosine_similarity(q_vec, doc_vectors)[0]
    ranked = sorted(zip(names, scores), key=lambda x: -x[1])
    return [name for name, score in ranked[:k] if score > 0]

query = "Can my landlord evict me without any notice?"
retrieved = retrieve(query)
print("retrieved:", retrieved)

# Suppose the model's generated answer cites two cases.
generated_answer = (
    "No. Under Rai v. Sharma, 2018, thirty days notice is required. "
    "This is also confirmed by Patel v. State Housing Board, 2017."
)

citation_pattern = re.compile(r"[A-Z][a-zA-Z]+ v\.? [A-Za-z ]+, \d{4}")
found_citations = citation_pattern.findall(generated_answer)

print("\ncitations in the generated answer:")
for citation in found_citations:
    verified = citation.strip() in retrieved
    print(f"  {citation!r:40} verified={verified}")
Output
retrieved: ['Rai v. Sharma, 2018']

citations in the generated answer:
  'Rai v. Sharma, 2018'                    verified=True
  'Patel v. State Housing Board, 2017'     verified=False

The retrieval step correctly found the one real, relevant case. The generated answer still smuggled in a second citation that was never retrieved at all. Checking citation in retrieved, rather than trusting the model's output, is what actually catches this.

Line by line

retrieve only returns matches with score > 0, so a query with no genuinely relevant case returns an empty list instead of a forced, low-quality match. An empty retrieval result should mean "say I don't know", not "cite the closest thing anyway".

citation_pattern looks for the shape of a legal citation, Name v. Name, Year, not its truth. Format-matching finds candidate citations; it says nothing about whether they are real.

citation.strip() in retrieved is the actual safety check: is this specific citation one the system verifiably looked up. This is a strict, exact-string check here for clarity; a production system typically needs fuzzier matching for near-identical formatting differences.

Common mistakes

Trusting any citation that "looks right". As shown above, a fabricated citation can be formatted identically to a real one. Format alone proves nothing about whether a source exists.

Checking citations against the whole library, instead of what was actually retrieved. A citation might be a real case, in your full library, that the retrieval step never actually surfaced for this specific question, meaning the model still did not use it correctly.

Skipping verification when retrieval seems to have "found something". Verification is not optional cleanup. It is the step that separates "grounded in something real" from "not visibly fabricated", two very different guarantees.

Try it yourself

Add "Rai v. Sharma, 2019" to the generated answer, a one-digit-off variant of a real citation. Rerun the exact-match check above.

It should fail verification, correctly, since the year does not match any retrieved case. Then consider what a fuzzier matching approach would need to handle this kind of near-miss without accepting genuinely wrong citations too easily.

What to learn next

Researcher — Mathematics and papers.

The retrieval-generation gap

RAG systems reduce, but do not eliminate, hallucination, because generation remains a free-text process conditioned on, not strictly constrained by, retrieved context. Formally:

text
answer = generate(query, retrieved_docs)

Nothing about generate guarantees every factual claim, including citations, in answer traces back to retrieved_docs. The model can still draw on parametric knowledge, information encoded in its weights from pretraining, alongside the retrieved context, and blend the two indistinguishably in its output.

Measuring the problem

Dahl et al. (2024), Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models, tested general-purpose LLMs on legal queries without retrieval grounding. Hallucination rates for case citations and legal facts ranged roughly 58% to 82%, depending on model and query type, and rose further for less prominent courts and older cases.

Magesh et al. (2024), evaluating commercial legal-AI products that specifically claim RAG-based grounding, found hallucination rates dropped substantially compared to ungrounded generation, but did not reach zero. Roughly one in six queries still contained at least one unsupported or incorrect legal claim, across the systems tested, despite retrieval grounding being marketed as the core safety mechanism.

Citation verification as post-hoc entailment checking

The verification step in the developer block is a simplified, string-matching version of a more general technique: checking whether a generated claim is entailed by its cited source, using the natural language inference methods covered earlier in this course.

text
verified(claim, source) = entails(source, claim)

A production system typically needs this NLI check in addition to simple citation-existence checking. A citation can point to a real, retrieved case that does not actually say what the answer claims. That is a subtler, more common failure than outright invention.

Complexity

Retrieval is O(log N) per query with an approximate nearest-neighbour index over N documents, or O(N) with the naive cosine-similarity scan used in the developer block. Citation-existence verification is O(c) for c citations, a cheap string or set lookup. Entailment-based verification is more expensive: one NLI forward pass per claim-source pair, which becomes the dominant cost for a thoroughly-checked answer.

Key references

  • Dahl, M. et al. (2024). Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models. arXiv:2401.01301
  • Magesh, V. et al. (2024). Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools. arXiv:2405.20362
  • Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv:2005.11401 — the original RAG paper.

Current state and open problems

Retrieval grounding is now standard in serious legal-AI products, and has measurably reduced, without eliminating, citation hallucination compared to ungrounded generation, per the Magesh et al. findings above.

The open problem is exactly the gap those numbers expose. A marketed grounding mechanism does not guarantee grounded output. Independent auditing of legal-AI tools' real citation accuracy, rather than trusting vendor claims, remains active and contested, with real professional consequences, as reported court sanctions for fabricated citations already show.

What to learn next