How to Test Your LLM for Hallucinations

To test an LLM for hallucinations, run its outputs through groundedness and faithfulness checks—comparing each claim against a trusted source (reference-based) or the provided context (reference-free)—score them with metrics like a faithfulness rate, set a pass threshold (e.g. 95%+), and automate the suite in CI so every model or prompt change is re-tested before release.

Hallucination is the failure mode that most often erodes trust in a production LLM: the model states something fluent, confident, and wrong. This guide walks through what hallucination actually is, how to detect it systematically, which metrics to track, and how to wire the whole thing into your delivery pipeline so regressions never reach users.

What is an LLM hallucination?

A hallucination is any model output that is not supported by its source material or by verifiable fact. The text usually reads well—that is precisely what makes it dangerous. The model isn't lying in any intentional sense; it is generating statistically plausible tokens that happen to diverge from truth or from the context it was given.

It helps to split hallucinations into two types, because they require different tests.

Intrinsic vs extrinsic hallucinations

• Intrinsic hallucination — the output contradicts the provided source. In a RAG (retrieval-augmented generation) system, this is when the answer disagrees with the documents that were retrieved. Example: the context says a refund window is 30 days, the model answers "60 days."
• Extrinsic hallucination — the output adds claims that are not present in and cannot be verified against the source. The model invents a citation, a statistic, an API parameter, or a policy that simply isn't in the context. Extrinsic errors are harder to catch because there's nothing to directly contradict—you're testing for unsupported additions, not conflicts.

A thorough test plan checks for both. Intrinsic errors are caught by contradiction detection; extrinsic errors are caught by groundedness (every claim must trace back to a source).

How do you evaluate an LLM for hallucinations?

There is no single test. In practice you combine a few complementary methods, trading off accuracy, cost, and how much ground-truth data you have.

Reference-based vs reference-free evaluation

• Reference-based evaluation compares the output to a known-correct answer ("gold" label) or to a trusted knowledge source. It's the most rigorous approach, but it requires you to build and maintain a labeled dataset.
• Reference-free evaluation has no gold answer. Instead it checks the output for internal consistency and groundedness against the provided context. This is what you use for open-ended generation where a single correct answer doesn't exist, and it's the default for most RAG faithfulness checks.

Groundedness and faithfulness checks

These are the workhorses of hallucination testing for RAG and tool-using systems:

• Groundedness asks: is every factual claim in the answer supported by the retrieved context? You decompose the answer into atomic claims and verify each one against the source passages.
• Faithfulness asks: does the answer stay consistent with the source without contradicting it or adding unsupported detail? It's effectively groundedness plus contradiction detection.

A useful, evaluation-friendly definition of faithfulness is the share of claims in an answer that are supported by the context: faithful claims / total claims.

LLM-as-judge

You can use a separate, capable model to grade outputs—classifying each claim as supported, contradicted, or unsupported, or scoring overall faithfulness. LLM-as-judge scales far better than human review and correlates reasonably well with it, but it has caveats: judges have their own biases, can be inconsistent, and shouldn't grade their own model's output unscrutinized. Mitigate this by using a strong judge model, giving it the source context explicitly, asking for claim-level verdicts with quoted evidence, and periodically calibrating against a human-labeled sample.

Retrieval grounding (for RAG systems)

If your LLM uses retrieval, a large fraction of "hallucinations" are actually retrieval failures—the right document never made it into the context. Test the retrieval layer separately: measure whether the passages needed to answer the question were actually retrieved (context recall) and whether retrieved passages were relevant (context precision). A model can't be grounded in a source it never received.

A comparison of hallucination evaluation methods

Method	What it measures	Needs gold data?	Cost	Best for
Exact/semantic match to reference	Output matches a known-correct answer	Yes	Low	Closed-form QA, fact lookups
Groundedness check	Each claim is supported by source context	No	Medium	RAG, summarization
Faithfulness (claim-level)	Supported claims vs. contradictions/additions	No	Medium	RAG, open-ended generation
LLM-as-judge	Holistic or claim-level correctness score	Optional	Medium	Scale, nuanced/open-ended tasks
NLI / contradiction detection	Output contradicts the source	No	Low–Medium	Catching intrinsic errors
Self-consistency sampling	Agreement across multiple sampled answers	No	High	Flagging uncertain outputs
Retrieval grounding (context recall/precision)	Whether correct context was retrieved	Partial	Low	Diagnosing RAG root cause

Most teams layer a cheap, deterministic check (contradiction/NLI, retrieval metrics) under a more expensive claim-level faithfulness or LLM-as-judge pass.

What metrics and pass thresholds should you use?

Pick a small set of metrics, define them precisely, and freeze a threshold before you look at results so the bar isn't moved to fit the model.

• Faithfulness / groundedness rate — share of outputs (or claims) fully supported by the source. This is usually the headline metric.
• Hallucination rate — the inverse: share of outputs containing at least one unsupported or contradicted claim. Track it per category if you can.
• Contradiction rate — share of outputs that directly conflict with the source (intrinsic).
• Abstention / "I don't know" rate — how often the model correctly declines when the answer isn't in context. A model that never abstains will hallucinate on out-of-scope questions.

Thresholds are risk-dependent and should be set per use case—the numbers below are illustrative starting points, not industry standards:

• High-stakes domains (medical, legal, financial): for example, target a faithfulness rate of 99%+ and treat any contradiction as a hard fail.
• General knowledge or support assistants: for example, 95%+ faithfulness with a small allowance for unsupported-but-harmless additions.
• Always pair a primary threshold with a no-regression rule: the new build must not score worse than the current production baseline on the same test set.

Report a confidence interval, not just a point estimate—on a 200-example set, a few percentage points of movement may be noise rather than a real regression.

How do you automate hallucination testing in CI?

The goal is to make hallucination testing a gate that runs automatically, the same way unit tests do. A practical setup:

1. Build a versioned eval set. Curate representative prompts—including known edge cases and previously-found failures—with source context and, where possible, gold answers. Version it alongside your code.
2. Run the suite on every change. Trigger it on prompt edits, model or version upgrades, retrieval/index changes, and dependency bumps—not just code commits.
3. Score against frozen thresholds. Compute faithfulness, hallucination, and contradiction rates; compare to both the absolute threshold and the production baseline.
4. Fail the build on regression. If a metric drops below threshold, block the merge or deploy and surface the specific failing examples with their claim-level verdicts so an engineer can triage quickly.
5. Keep a golden regression set. Every real hallucination you find in production becomes a permanent test case, so the same bug can never silently return.

This is where a dedicated suite earns its keep. An automated platform like LLMQA runs these groundedness, faithfulness, and adversarial checks continuously and packages the result as a cryptographically signed, verifiable certificate—useful evidence when you need to show auditors, customers, or leadership that a given model version was tested before it shipped. Hallucination is also only one axis; mature programs test alongside jailbreaks, persona drift, bias, and compliance, and map results to frameworks such as the OWASP LLM Top 10 and the NIST AI Risk Management Framework.

Frequently asked questions

Can you ever fully eliminate LLM hallucinations?

No. Hallucination is an inherent property of probabilistic language models, so the realistic goal is to measure, reduce, and bound it—not to claim zero. Strong retrieval grounding, prompt constraints that allow the model to abstain, and continuous testing against a threshold keep the rate low and prevent regressions.

What's the difference between faithfulness and factual accuracy?

Faithfulness measures whether an answer is supported by the provided context—it can be faithful to a source that is itself wrong. Factual accuracy measures whether the answer matches real-world truth, which usually requires reference-based evaluation against a trusted knowledge base. Test both: faithfulness catches RAG grounding failures, accuracy catches bad source data.

Do I need labeled data to test for hallucinations?

Not necessarily. Reference-free methods—groundedness checks, contradiction detection, and LLM-as-judge against the provided context—work without gold answers, which is why they're the default for open-ended and RAG tasks. Labeled data raises rigor and is worth building for high-stakes use cases, but you can start testing immediately without it.

Key takeaways

• Hallucinations come in two flavors: intrinsic (contradicts the source) and extrinsic (unsupported additions)—test for both.
• Groundedness and faithfulness checks are the core methods; use reference-based evaluation when you have gold data and reference-free when you don't.
• LLM-as-judge scales evaluation but must be calibrated against human-labeled samples.
• For RAG, diagnose retrieval grounding separately—many hallucinations are retrieval failures, not generation failures.
• Define metrics and frozen pass thresholds (faithfulness rate, hallucination rate, contradiction rate) per use case, and enforce a no-regression rule.
• Automate the suite in CI so every model or prompt change is re-tested, and turn every production failure into a permanent regression test.