Beyond RAG: Your AI Is Hallucinating Its Sources
Let’s say you’re using an AI tool for document review. You ask it a question about your company’s annual leave policy. It gives you an answer and cites “Section 3.2 of the Employee Handbook.” That’s exactly what you wanted, an answer with a source. You can verify it. You can trust it.
Except you check Section 3.2 and it’s about company holidays, not annual leave. These aren’t the same thing at all. So what happened? Did the AI misread the section? Did it confuse two documents? Did it pull from an outdated version?
None of the above. The AI never read Section 3.2. It never read any section of anything. It doesn’t know what sections are. What it did was generate an answer from fragments, the same fragments we talked about in the last article. Then it guessed where that answer probably came from.
The citation is not a lookup. It’s a hallucination that looks like a lookup.
How Citations Actually Work in RAG
Most people assume that when an AI tool cites a source, the process went something like this: The system searched your documents, found the relevant passage, read it, and reported back where it found the answer. You can think of it like a research assistant pulling a book off a shelf, opening to the right page, and handing it to you with a sticky note.
That’s not even close to what happens.
Here’s the actual process. The system takes your question, converts it to a vector, and grabs the top five or ten document fragments that are mathematically closest. Those fragments get passed to the AI along with your question. The AI generates a response based on whatever is in that prompt window.
Now the system needs to tell you where the answer came from. Depending on the implementation, this happens one of a few ways. Sometimes the AI itself is asked “which sources did you use?” and it generates a citation the same way it generates everything else: by predicting what text should come next. Sometimes the system maps the response back to whichever chunks were retrieved and says “the answer came from these fragments.” Sometimes a separate algorithm tries to match the AI’s claims against the source fragments after the fact.
None of these approaches do what you think they do.
In the first case, the AI is generating the citation using the same mechanism it uses to generate everything else: statistical prediction. It’s not looking up a source. It’s predicting what a citation should look like. When it says “Section 3.2,” it might be repeating text it saw in a fragment, or it might be inferring a section number because the content felt like it belonged there, or it might be fabricating one because the response seemed like it needed a citation and the model obliged.
In the second case, the system can tell you which chunks were retrieved, but it cannot tell you which chunks the AI actually used when composing its answer. Five chunks went in. The AI might have leaned heavily on chunk 2, ignored chunks 1 and 4 entirely, and blended chunks 3 and 5. The citation says “From these five documents.” Which one actually supported the claim? The system doesn’t know. It knows what went in, not what came out.
In the third case, a matching algorithm tries to verify the AI’s claims against the source text after the response is generated. This can catch obvious errors, like citing a document that clearly discusses a different topic, but it relies on text similarity to do the matching. Subtle misattributions, where the citation almost supports the claim but doesn’t quite, sail right through.
All three approaches share a deeper problem. Remember that the original document was chopped into fragments during chunking. Section headers, page numbers, document structure, the relationship between a heading and the paragraphs beneath it: all of that was destroyed. When any part of the system tries to tell you “this came from Section 3.2 of the Employee Handbook,” it’s reconstructing that location from fragments that no longer carry structural information. The section number is an inference, not a fact.
This Is Not a Rare Edge Case
If you’ve only used AI tools casually, you might think this happens occasionally. It might be a glitch, or an edge case that gets caught in QA.
It’s not occasional. Research on RAG citation accuracy consistently shows significant error rates. Studies from Stanford’s HELM project and others have documented that AI systems regularly attribute claims to the wrong source, cite passages that don’t support the claim, or generate citations to documents that don’t exist in the corpus at all.
This isn’t surprising when you understand the architecture. There is no part of a RAG system that is engineered to guarantee citation accuracy. The retrieval step finds fragments by mathematical similarity. The generation step produces text by statistical prediction. And the citation step, regardless of which method is used, is either the AI guessing, the system reporting what went in without knowing what was used, or a post-hoc check that catches some errors but not all. At no point does anything in the pipeline do what a human researcher does: find the document, read the passage, confirm it supports the claim, and then report the source.
When “Close Enough” Isn’t Close Enough
For a customer service chatbot answering, “What are your return policy hours?”, citations don’t even matter. The customer doesn’t care about the source document. If the answer is roughly right, nobody gets hurt. The stakes are low.
But what if you put this system in a law firm? An associate asks the AI to find every document that references a specific contractual obligation. The AI returns a list with citations. The associate, under time pressure, trusts the citations and includes them in a filing. One of those citations points to a document that says the opposite of what the AI claimed. Even worse, it points to a privileged document from a different client matter that the AI pulled from because the vector database doesn’t respect access boundaries. (That’s a whole separate problem, and it’s coming later in this series.)
What if you put it in an insurance company? A claims analyst asks the AI to summarize the medical evidence supporting a claim. The AI cites three medical reports. Two are correct. The third is from a different claimant’s file. The summary looks authoritative. The analyst approves the claim based on evidence that doesn’t belong to it.
What about using it in a compliance environment? A regulatory filing requires citation to specific internal policies. The AI generates the filing with citations that look exactly right, with correct document names and correct section numbers. But the actual content of those sections doesn’t support the claims in the filing.
In every one of these scenarios, the AI did what it was built to do. It generated a response and attached citations. The problem is that there is nothing in the architecture that ensures those citations are accurate. And this system is being sold to industries where the difference between a correct citation and a plausible one can mean a malpractice suit, a regulatory violation, or a wrongful claim decision.
The Question You Should Be Asking
When you’re evaluating an AI document tool, there’s one question that tells you whether the citations can be trusted:
Can you look at the cited document and see the exact text that triggered the match?
A full-text search engine returns a document because the words you searched for are actually in it. The citation points to that document because the search engine matched your query to real text on a real page. You can open the document, find those words, and verify the match yourself. The citation is anchored to something concrete.
A vector-based system returns document fragments because they are numerically similar to your query. The citation points to a fragment because of math, not because of specific text you can go read and confirm. And even if the right fragment was retrieved, the AI might restate it, summarize it, or misrepresent what it says. The citation still looks correct because it points to the right fragment, even when the AI’s interpretation of that fragment is wrong. There is nothing concrete to verify against.
That’s the difference. One gives you a citation you can check by reading the document. The other gives you a citation that looks right but has no anchoring to the actual words on the page.
If a vendor can’t explain how their citation pipeline works, or changes the subject when you ask, that tells you something. If they say “the AI identifies the relevant passages” without explaining whether those passages were found by matching real text or by vector similarity, that tells you even more.
This is the second article in the Beyond RAG series, a three-part exploration of why AI document tools fail and what the alternative looks like. Next: The Fix Isn’t a Better Prompt.