How to measure your chatbot's hallucinations and reduce them

Your chatbot makes things up. It’s not a possibility: it’s a statistical certainty. Even in production, language models fabricate information at rates typically reported between 3% and 20% on mixed tasks, and higher on edge cases or poorly covered domains.^[1] The useful question isn’t whether it hallucinates, but how much —and you only find that out by measuring it. This guide shows you how to measure your chatbot’s hallucination rate and, afterward, how to bring it down.

The order matters: first you measure (to get a baseline), then you apply reduction tactics, and then you measure again to confirm you improved. Without the measurement, you’re optimizing blind.

Step 1 — Define what counts as a hallucination in your case

Before measuring you need a clear criterion. A hallucination is a statement that is false or not supported by the sources the system is supposed to use. That second part is key: in a chatbot that answers from your company’s documentation, an answer can be “true in the world” but still a hallucination if it doesn’t come from your approved sources. Define which of the two matters to you —general factual accuracy, faithfulness to the source, or both— because that determines how you’ll evaluate.

Step 2 — Design test cases that expose them

Hallucinations don’t appear evenly: they get triggered in specific situations. Your test cases have to seek them out deliberately, not just cover the easy stuff. Include:

• Questions about concrete data (figures, dates, policies, conditions) whose correct answer you know and can verify.
• Out-of-scope or ambiguous questions, where the model has “too much freedom” to make things up —a vague prompt is one of the direct causes of hallucination.^[2]
• Specialized or infrequent domain cases, where the model has less signal and tends to fill in the gaps.
• Loaded questions that assume a false fact, to see whether the model corrects it or goes along with it.

Prioritize coverage over volume: 50 well-chosen cases that target your real risks are worth more than 500 generic ones.

Step 3 — Measure with a hallucination evaluator

This is where you get the number. Because the answers are free text, you can’t compare them word by word; you use an evaluator —typically an AI judge (LLM-as-a-judge)— that compares each answer against the source or the expected answer and determines whether there’s fabrication. The metrics you’ll see:

• Hallucination rate: what percentage of answers contain an unsupported statement. It’s your headline number.
• Groundedness: how anchored the answer is in the retrieved context, in RAG systems.
• FActScore: for long answers, it breaks the text into atomic facts and measures what fraction is supported by a reliable source.^[3]

One detail that isn’t minor: the evaluator can also be wrong. That’s why it’s worth measuring the detector’s own accuracy (is it over-flagging hallucinations that aren’t?) and, ideally, using a judge that’s calibrated against human judgment —if you don’t verify the evaluator, you don’t know whether your hallucination rate is real or an artifact of the judge.

Step 4 — Apply the reduction tactics

With the baseline measured, you attack the problem. The levers, from highest to lowest impact:

1. Anchor to sources with RAG. It’s the most effective mitigation: making the model retrieve reliable documents before answering, instead of “remembering.” Various production reports indicate that RAG on its own reduces hallucinations by roughly 40% to 71%, and combined with guardrails it can go much higher in well-built architectures.^[4] But RAG is only as good as what it retrieves: if it brings back poor or outdated documents, the problem returns.
2. Add grounding guardrails. Modules that review the answer before showing it and block or edit unsupported statements. They verify that each claim is supported by the source; anything new that isn’t gets flagged as ungrounded.^[5]
3. Improve prompts with a grounding focus. System instructions that explicitly require relying on the context and citing the source reduce “free guessing.” Narrowing the scope and giving specific context —“summarize the Q2 2025 report” instead of “tell me about the latest project”— removes the model’s freedom to make things up.^[2:1]
4. Post-hoc verification (self-verification). Techniques where the model generates the answer and then checks each statement against the context, in a second pass.^[5:1]
5. Design for “I don’t know.” A system that can abstain or ask for clarification when uncertain is more reliable than one that always answers.

An honest expectation: none of these brings the rate to zero. As long as the model is probabilistic, some portion of hallucination is structural. The goal is to lower it to a level tolerable for your case —and keep it there.

Step 5 — Monitor over time

The hallucination rate isn’t fixed: it changes when the provider updates the model, when your knowledge base goes stale, or when the queries change. That’s why measurement isn’t a one-off pre-launch event but a continuous process. In fact, one industry recommendation is to allocate between 30% and 40% of an AI project’s time specifically to testing and mitigating hallucinations —it’s not overhead, it’s a central part of the work.^[6] Reuse your cases as regression tests: every time you change something, you rerun them and confirm the rate didn’t get worse.

How ArtificialQA solves it

Sustaining this cycle —measure, reduce, measure again, monitor— by hand is unfeasible. ArtificialQA connects to your chatbot —via URL or API, without writing code— and subjects it to a hallucination evaluator that measures, over your representative cases, when the answer departs from the truth or from the source. Because its judges are calibrated, the rate you get is reliable and not an opaque number; and because you can rerun the same plans, you measure the effect of each reduction tactic and watch for degradation over time.

The result is that you stop guessing how much your agent makes up and start managing it with a number. Because with hallucinations the question is never whether they’ll happen —they will— but how many you’re willing to tolerate, and whether you’ll find out in a test or your customer will in production.

Frequently asked questions

How do you measure a chatbot’s hallucination rate? By designing test cases that expose fabrication (concrete data, ambiguous questions, specialized domain) and evaluating the answers with a detector —typically an AI judge— that compares them against the source or the expected answer. The headline metric is the percentage of answers with any unsupported statement.

What’s the best way to reduce hallucinations? Anchoring answers to reliable sources with RAG is the highest-impact tactic (it reduces them by roughly 40% to 71% depending on the case). It’s complemented by grounding guardrails, grounding-focused prompts, post-hoc verification, and designing the system so it can say “I don’t know.”

Does RAG eliminate hallucinations? Not entirely. It reduces them significantly by anchoring the answer to real documents, but it’s only as good as what it retrieves: if retrieval fails or brings back poor data, the model makes things up again. As long as the model is probabilistic, some portion is structural.

What is groundedness? It’s the measure of how anchored an answer is in the retrieved context, in RAG systems. A well-grounded answer relies on the source; a new claim that doesn’t come from the context is considered ungrounded.

How often should you measure hallucinations? Continuously. The rate changes when the model, the knowledge base, or the queries are updated, so it’s best to monitor in production and reuse the cases as regression tests on every change.