From chatbots to agents: what changes in quality control
Moving from chatbots that respond to agents that act completely changes quality control. What gets harder and why.
For years, “conversational AI” meant chatbot: the user types, the system responds, the end. In 2026 that definition became outdated. The average company already operates around 12 AI agents, and the projection is 20 by 2027.[1] And an agent is not a faster chatbot: it’s a distinct category of software, one that breaks almost everything we knew about how to assure its quality.
If your team learned to test chatbots and now has to handle an agent, this article explains what changes, why the old method falls short, and what to measure instead.
The difference that changes everything: from responding to acting
A chatbot —even a powerful one, based on an LLM— works in a question-answer pattern: it receives a message, generates a response, waits for the next one. It doesn’t take actions in external systems or plan multiple steps.[2]
An agent operates in a loop: it observes, reasons, acts, evaluates whether the task is complete, and if not, tries again.[2:1] You give it a goal —“resolve this customer’s complaint”— instead of a specific instruction, and it decides how: it queries a database, calls an API, retrieves a document, chains the steps together. It has five traits a chatbot doesn’t: goal-oriented thinking, tool use, multi-step reasoning, memory between interactions and autonomous decision-making when something turns out differently than expected.[3]
That autonomy is precisely what makes it valuable —and what makes it hard to test. As a McKinsey partner summed it up, agency is not just another feature: it’s a transfer of decision rights. The question stops being “is the model accurate?” and becomes “who answers when the system acts?”.[4]
Why chatbot QA isn’t enough for agents
When you test a chatbot, you evaluate a response: is it correct, does it have a good tone, does it not hallucinate? That’s still necessary with an agent, but it’s no longer sufficient, for three reasons that come directly from its architecture.[5]
Failures hide in the path, not in the final response. An agent takes many steps without supervision. It can arrive at a correct final answer having done something improper along the way —querying the wrong data, calling a tool it shouldn’t have— and a test that only looks at the final result won’t catch it. The canonical example comes from the Tau²-Bench benchmark, which simulates customer service domains: an agent that books the correct flight but waives a fare that couldn’t be waived fails the task, even though the result looks fine.[6] That maps exactly to how regulated industries deploy agents.
The same input can produce a different path each time. Because agents have state, memory and receive variable results from their tools, the same task can take a different trajectory on each run. A single test that passes tells you very little.[5:1]
Assigning blame is hard. With retrievers, tools, planners and subagents all in play, an end-to-end score tells you that something broke, not which component broke.[5:2]
The cost of not seeing these failures is concrete. In March 2025, an AI agent at a fintech went into an uncontrolled loop during a transaction reconciliation: it ran eleven days and accumulated 47,000 dollars in costs before anyone noticed.[7] And according to a PwC survey, 79% of organizations have already adopted AI agents, but most cannot trace failures across multi-step flows or measure quality systematically.[8] It’s no coincidence that Gartner projects that more than 40% of agentic AI projects will be cancelled by the end of 2027, often because teams don’t have the evaluation infrastructure to detect failures before production.[7:1]
The new things to evaluate
Testing an agent doesn’t replace response-quality QA: it extends it. To the usual dimensions —accuracy, tone, hallucinations, escalation— are added others specific to agents. The 2026 technical consensus organizes them into three levels:[5:3]
| Level | What it evaluates | Question it answers |
|---|---|---|
| End-to-end | The final result of the task | Did the agent achieve the goal? |
| Trajectory | The sequence of steps and decisions | Did it get there by a correct path? |
| Component | Each tool, retriever or subagent | Which specific piece failed? |
Within those levels, the evaluations that gain prominence with agents are:
- Tool calling: does the agent call the correct tool, with the correct parameters, in the correct number of steps? A single task can unfold into dozens or hundreds of calls to APIs, databases and other services.[5:4]
- Planning quality: does the sequence of steps it chose make sense, or does it take unnecessary detours?
- Policy adherence: does it respect the written business rules, as in the non-waivable fare example?
- Escalation to a human: when and how it hands over control —as critical in an agent that acts as in a chatbot that responds.
What this means for your team
The underlying change is one of mindset. With chatbots, QA validated responses. With agents, QA has to validate behavior: not just what the system said, but what it did, in what order and why. It’s no coincidence that the 2026 quality assurance discourse describes QA as the “accountability layer” of AI —the place where you define what it means for an agent to work well and verify that it actually does.
The good news is that the core of the craft doesn’t change: it’s still about designing good test cases, defining what “correct” means and measuring it systematically. What changes is the surface to cover. And that requires tools built to evaluate agents, not chatbots —able to look at the trajectory, not just the final response.
That’s what ArtificialQA does: it connects to your agent —by URL or by API, no code— and evaluates it on the dimensions that matter in a system that acts, including correct escalation to a human and adherence to business rules. A QA or business team can thus test an agent with the same rigor with which it always assured software quality, adapted to an era in which AI no longer just converses: it decides and acts.
Because that is, in the end, the question that defines the moment. When your AI only responded, an error was a bad answer. Now that it acts, an error is a bad decision —executed before anyone sees it. The only defense is having tested it beforehand.
Frequently asked questions
What is the difference between a chatbot and an AI agent? A chatbot responds in a question-answer pattern. An agent operates in a loop: it observes, reasons, uses tools, makes decisions and executes multi-step tasks autonomously, pursuing a goal instead of responding to a specific instruction.
Why is it harder to test an agent than a chatbot? Because failures hide in the trajectory of steps (not just in the final response), the same input can produce different paths each time, and with many components in play it’s hard to attribute which piece failed.
What is evaluated in an AI agent that isn’t evaluated in a chatbot? Beyond the response-quality dimensions, you evaluate tool use, planning quality, adherence to business policies and the trajectory of decisions, across three levels: end-to-end, trajectory and component.
What is trajectory evaluation? It’s evaluating the sequence of steps and decisions the agent took to reach a result, not just the result itself. It lets you detect when an agent reaches a correct answer by an improper path.
Have chatbots stopped being relevant? No. For many cases (FAQs, answers from documents), a chatbot or a RAG system solves things well and at lower cost and complexity than an agent. The agent is worthwhile when the task involves multiple steps and actions in real systems.
Product Manager of ArtificialQA at QAlified, with 15+ years in software testing and automation. She works at the intersection of quality and AI: designing and evaluating approaches to test non-deterministic systems and ensure their behavior in production.
