Bias and fairness in AI: the two-applications experiment

The first AI discrimination case the EEOC (the U.S. equal employment opportunity agency) ever settled was not uncovered by a sophisticated audit. It was uncovered with the simplest experiment in the world: a person rejected by iTutorGroup’s hiring software submitted the same application again, identical in every respect except for the date of birth, which was more recent. The second time, they got the interview.^[1]

That pair of twin applications ended in a federal lawsuit, a USD 365,000 settlement for more than 200 applicants rejected on age grounds, and a five-year consent decree.^[2] And it left behind, in passing, this article’s methodological lesson: bias is tested with counterfactual pairs —the same input twice, changing only the protected attribute. If the output changes, you have a finding.

What that applicant did by hand once, a QA team can do systematically, at scale, and before launch. It is one of the most concrete tests of Responsible AI, because fairness is the principle that is easiest to declare and hardest to demonstrate. This is the protocol.

First, conceptual precision: what bias means in a conversational agent

In generative systems, bias is rarely an explicit rule like iTutorGroup’s (which was deliberately programmed, a nuance worth keeping when citing the case).^[3] It tends to be statistical and subtle, inherited from the training data, and it shows up in ways no log will surface: the agent is more formal with some names than others, more skeptical toward certain written accents, more likely to offer the premium product depending on how the customer sounds, more dismissive in one language than another.

It helps to separate two planes, because they are tested differently:

• Unequal treatment (process fairness): given equivalent queries, does the agent respond with the same quality, tone, completeness and willingness to help? This plane applies to any conversational agent.
• Unequal outcomes (decision fairness): when the agent influences decisions (approve, recommend, prioritize, escalate), do the outcomes differ by protected attribute? This plane applies to agents with consequences, and it is the one that triggers legal risk.

The two-applications protocol, systematized

Step 1 — Choose the attributes and the risk in your domain. Gender, age, origin (names, demonyms, language variants), implied socioeconomic status, disability. Not all of them apply to every agent: an HR assistant and a credit chatbot have different risk maps. The guiding question: in what decisions or treatment could this agent differentiate improperly?

Step 2 — Build counterfactual pairs (or families). Take your real test cases and clone them changing only the attribute: the same credit query signed by “Juan” and by “Yenifer”; the same complaint written in neutral Spanish and in a strongly marked regional variant; the same question from a 25-year-old and a 70-year-old customer when age is visible in the context. Everything else —amount, history, underlying wording— stays identical. The method’s strength lies in that sameness: any difference in output is attributable to the attribute.

Step 3 — Evaluate the differences, not just the answers. Each member of the pair is scored on the usual dimensions (accuracy, completeness, tone, willingness to help, escalation) and the analysis looks at the delta within the pair: did the “Yenifer” version get shorter answers, more hedging, fewer offers? An isolated case is noise —models are non-deterministic and no two runs are identical— which is why you work with families of pairs and look for the pattern: systematic differences, in the same direction, across many cases.

Step 4 — Define what “passed” means. The fairness criterion needs a threshold too: for example, that the average score difference between variants of an attribute does not exceed a defined margin on any dimension, and that no prohibited behavior (denying a service, changing a condition) appears associated with the attribute. Like the rest of the test plan, it is re-run on every model or prompt change, because bias drifts too.

The three mistakes that invalidate fairness testing

Testing with artificial lab cases. “Are you racist?” asks the model about its self-image, not about its behavior. No commercial model answers that one badly, and none reveals its real biases there. The test material is your real business cases, cloned.

Looking only at the decision and not at the treatment. An agent can approve the loan for both variants… having demanded twice as much justification from one of them. Unequal treatment erodes customers just as unequal decisions do, and it is also the early signal of the second.

Doing it once. The iTutorGroup case teaches this too: the decree did not require the company “not to discriminate” in the abstract, but sustained monitoring, records and compliance procedures.^[2:1] A bias test snapshot expires with the next model change; what is defensible is the series.

Where the platform fits

The counterfactual protocol is, at bottom, a particular use of the same evaluation engine: cases with context and criteria, multidimensional evaluators, systematic comparison of results. In ArtificialQA you implement it by creating the counterfactual case families inside a fairness plan, evaluating each variant with the same calibrated judges (tone, completeness, willingness, escalation) and using the comparison across cases and runs to detect the systematic deltas, with the results recorded as evidence. Which is exactly what a regulator, an auditor or an enterprise customer will ask for when they ask how you know your agent treats everyone the same.

Frequently asked questions

What is counterfactual bias testing? It is evaluating a system with pairs of inputs that are identical except for a protected attribute (name, age, gender, language variant) and measuring whether the outputs differ systematically. It is the most direct method for attributing a difference in treatment to the attribute rather than to another variable.

Does a different result in one pair prove there is bias? No: models are non-deterministic and no two runs are identical. The valid finding is the pattern —differences in the same direction, sustained across families of cases and repeatable between runs.

Does this apply if my agent doesn’t make decisions (it only informs)? Yes, on the treatment plane: quality, tone, completeness and willingness must be equivalent for equivalent queries. Unequal treatment damages the experience and the brand even when no formal “decision” is at stake.

Does fairness testing protect me legally? It reduces risk and produces evidence of diligence, but no technical test replaces legal analysis: obligations vary by jurisdiction and by use case (hiring, credit and health have specific rules). Build the fairness plan together with your legal team.

How often should it be repeated? On every model, prompt or data change, and on a schedule as part of monitoring. Bias can appear or disappear with a provider update, just like any other quality dimension.

This article is informational and does not constitute legal advice. Verify your specific obligations with your legal or compliance team.