AI quality metrics: accuracy, tone, completeness and hallucinations explained

“Is the chatbot doing fine?” is a question you can’t answer —at least not usefully— until you break it down. “Fine” is not one single thing: an AI answer can be factually correct but with terrible tone, or friendly but incomplete, or fluent but with a fabricated fact in the middle. Measuring AI quality means evaluating several dimensions at once, each with its own metric. This is the reference guide to which ones they are, what each measures and when it matters most.

One piece of advice before we start: not every metric applies to every case. Choosing the wrong ones brings two problems —false confidence (high scores on irrelevant metrics while the real problems go unnoticed) and alert fatigue (low scores on metrics that don’t matter for your case).^[1] Use this guide to pick the right combination, not to measure everything.

Factual accuracy

What it measures: whether what the AI says is correct. It’s the most basic metric and the most critical in any system that delivers information —and the one that, when it fails, produces the most expensive incidents.

Accuracy is not trivial to evaluate because it depends on the domain: an assistant can have 95% accuracy in general language and fail dramatically with an industry’s jargon or internal acronyms, which makes it essential to evaluate it in your context and not in the abstract.^[2] It’s best measured by query category, not as a single average that hides where it’s really failing.

Hallucinations

What it measures: how often the AI asserts things that aren’t true or that aren’t backed by its sources. It’s a cousin of accuracy, but with a nuance: a hallucination is an invention presented with confidence, and that’s why it’s especially dangerous —the system doesn’t hesitate when it invents.

It’s the dimension that most worries organizations: hallucinations and reliability are among the top barriers to adopting generative AI, cited by 60% of executives in a global survey.^[3] In source-anchored systems (RAG), it specifically measures whether the answer stays faithful to the retrieved material.

Tone and empathy

What it measures: whether the style, formality and emotional load of the answer are appropriate for the context.^[4] An answer can be perfect in content and still damage the relationship if it sounds cold at a delicate moment or too informal in a serious one.

This dimension is often underrated, but it carries weight: in customer service, how matters as much as what, and in Spanish-speaking markets especially —AI that feels like a wall built to avoid a person generates rejection. Tone is hard to evaluate with classic automatic metrics and is one of the cases where an AI judge (LLM-as-a-judge) contributes most, because it can assess nuances that a word comparison doesn’t capture.

Completeness

What it measures: whether the answer addresses everything that was asked, not just part of it. It matters especially in multi-part questions.^[4:1] If someone asks “what are the pros and cons of X, and which one suits me?”, an answer that covers only the pros is accurate and relevant but incomplete.

Completeness has an internal tension with conciseness: for a simple question, a “complete” answer can be unnecessarily long.^[4:2] That’s why it’s best to calibrate the completeness expectation to the complexity of the query, not to demand exhaustiveness across the board.

Relevance

What it measures: whether the answer actually addresses what was asked, without wandering off. It’s central in search, RAG and chatbot systems, where the user expects a direct answer.^[4:3] An AI can give correct and complete information about something that was not what was asked.

Handoff to a human (escalation)

What it measures: whether the agent hands control over to a person when the case requires it —an explicit request, a sensitive situation, something beyond its scope. In sectors such as healthcare or banking, this is among the most critical dimensions: an agent that doesn’t hand off in time turns a manageable problem into a serious one. In healthcare, indeed, a too-low handoff rate can be a warning sign, not a sign of success.

Context and source adherence

What it measures: whether the agent sticks to the documentation and policies it was given, instead of “filling in” with what the model thinks it remembers. It’s the key dimension when the system must answer only from approved sources —typical in banking, insurance and any regulated domain.

How to choose and combine the metrics

The table summarizes when each dimension weighs most:

The practical principle: map your main query types and, for each one, identify which quality dimension is the priority. If 40% of your queries are about the status of an order, the factual accuracy of the shipping data is your priority; if most are sizing questions, completeness and relevance weigh more.^[2:1] Start with two or three dimensions that really matter for your case, not with all seven at once.

How they are measured in practice

Historically, evaluating these dimensions required the team to read conversations by hand —accurate but impossible to scale. Today the standard approach combines automatic evaluation with AI judges and human review of samples: an evaluator model reviews the answers against the criteria you define (accuracy, completeness, tone, policy adherence) and scores thousands of conversations a human wouldn’t get through.^[5] The key —and what distinguishes a reliable evaluation— is that this judge is calibrated against your team’s judgment, so its scores are trustworthy and not an opaque figure.

That is exactly what ArtificialQA does: it connects to your agent —by URL or API, no code— and evaluates it across these dimensions with calibrated judges, leaving a record of each measurement. Instead of “I think the bot is doing fine”, your team gets a number per dimension, on your own cases, that can be tracked over time. Because AI quality is not one thing —it’s several— and each is managed only when it’s measured.

Frequently asked questions

What are the main AI quality metrics? The most important are factual accuracy, hallucinations, tone and empathy, completeness, relevance, handoff to a human and context/source adherence. Which to prioritize depends on the use case.

What is the difference between accuracy and hallucinations? Accuracy measures whether the answer is correct; hallucinations measure when the AI invents information presenting it with confidence. They are related, but a hallucination is especially dangerous because it comes wrapped in confidence.

How do I choose which metrics to measure? Map your main query types and identify, for each one, which quality dimension is the priority. Start with two or three that matter for your case, not with all of them. Measuring irrelevant metrics generates false confidence and alert fatigue.

Why is tone a quality metric? Because an answer can be correct in content and still damage the relationship with the customer if the style is inappropriate for the context. In customer service, the how weighs as much as the what.

How are these dimensions measured without reviewing everything by hand? With automatic evaluation through AI judges (LLM-as-a-judge) calibrated against the team’s judgment, complemented by human review of samples. That way thousands of conversations a human wouldn’t get through are scored.