How Should CX Leaders Evaluate AI Agents Before Customers Do?
By Christi AkinwumiJune 26, 2026
Most customer experience teams still review 2 percent of their AI conversations and call it quality assurance. That model was built for a roomful of human agents handling a few hundred calls a day. It does not survive contact with a fleet of AI agents handling thousands.
Yet that is the system most enterprises still run. A quarterly review. A sample. A heat map. A board update that says quality looks fine. Meanwhile the AI agent has already shipped a confidently wrong refund policy to 4,000 customers before anyone notices.
This is the AI evaluation gap. It is the next big control failure in enterprise CX, and the clock on it started running the moment AI agents stopped being chatbots.
The Old QA Model Is Out of Math
Sample-based quality assurance works when the population is small, the variance is low, and the cost of each failure is bounded. A 2 percent sample of 500 calls per day gives a reasonable read on a team of human agents. The failures are slow. The reviewers can keep up.
That math does not work for AI agents. A single AI workflow can handle 10,000 conversations in a day. The variance is much higher because user input is open-ended. The failures are fast and silent. By the time a quarterly review surfaces a pattern, the affected customer count is already in the thousands.
The deeper issue is structural. Sample review measures what the agent already did. Evaluation measures what the agent will do, before it reaches a customer. The shift from review to evaluation is the same shift that software engineering went through 20 years ago when continuous testing replaced manual QA. CX is now living through that change. Most teams have not noticed.
Gartner’s 2026 Hype Cycle for Agentic AI flags the operating-model gap directly. Gartner also expects over 40 percent of agentic AI projects to be canceled by the end of 2027. The cancellations rarely happen because the model failed. They happen because the team could not prove the system was safe to scale.
What an AI Evaluation Actually Is
In plain language, an evaluation is a structured test. You give the AI system a defined input. You compare the output to a rubric or a known-good answer. You record whether it passed. You run that test on every change.
That is the whole shape. The skill is in the details.
Anthropic’s Develop Tests and Evaluations guide walks through the practical primitives. OpenAI’s evals framework on GitHub provides a working code reference. Both treat evaluation as a first-class engineering discipline rather than an afterthought.
For CX leaders, the operational form is what matters. An evaluation system has three pieces. A golden dataset of representative examples. A rubric that defines what good looks like for each behavior. A runner that scores every change against every example. Build those three, and you have an eval program. Skip any of them, and you have a dashboard.
The Three Layers Most CX Teams Miss
A useful frame, drawn from the NIST AI Risk Management Framework, is that AI risk is layered. The same is true of evaluation. Different layers catch different failures, and most CX teams only run one.
Layer one: input evaluation. Did the agent understand what the customer asked for? Did it classify the request correctly? Did it route it to the right tool or flow? Input failures look like the agent answering a billing question with a shipping policy. They are easy to detect when you write a rubric for them. They are invisible if you do not.
Layer two: behavior evaluation. Did the agent follow the rules of how it should respond? Tone, disclosure, escalation thresholds, citation rules, refusal patterns. Behavior failures are the ones that show up on social media. They are also the ones that violate the EU AI Act transparency rules for AI-mediated interactions and trigger reputational risk well before legal risk.
Layer three: outcome evaluation. Did the customer’s problem get resolved? Measured against ground truth, downstream signals, or human review. Most teams run only this layer, only on a sample, and only after the fact. That is the gap that ate the quarterly review.
The teams that catch the most failures early are the ones running all three layers continuously, with rubrics tied directly to deployment gates. The teams that catch the fewest are the ones running one layer manually, with no rubric, on a sample size that statistics gave up on years ago.
For a deeper look at how rubric design and conversation testing meet, see our piece on conversational AI testing methods.
Writing a Rubric That Actually Works
The hardest part of building an evaluation program is not the tooling. It is the rubric. A good rubric translates policy into a binary or graded score that any reviewer, human or model, can apply consistently.
Bad rubric: “The agent should be helpful and on-brand.”
Better rubric: “The agent’s response cited the correct policy. The response disclosed AI involvement on the first turn. The response offered a clear escalation path. The response stayed under 120 words. The response did not invent product names that are not in the approved product list.”
The difference is testability. The first rubric requires a judgment call every time. The second rubric produces the same answer across reviewers. That consistency is what makes the evaluation usable as a deployment gate.
Anthropic’s prompt engineering guidance makes the same point about instructions. Precision in language is the determinant of reliability in production. The rubric is just the eval-time version of the same skill. The team that already writes precise instructions, often a conversation design team, is the team best positioned to write good rubrics.
For behavior evaluation at scale, the LLM-as-judge pattern lets one model score another model’s output against the rubric. It is cheaper and faster than human review. It does require periodic human calibration, because models score themselves and each other generously without it. Stanford’s AI Index 2025 tracks how evaluation maturity correlates with successful production AI in the enterprise, and the maturity curve runs straight through this calibration step.
Adversarial Evals Are Not Optional Anymore
Standard evals check whether the AI does the right thing on representative input. Adversarial evals check whether the AI does the wrong thing on input designed to break it. Prompt injection. Jailbreak attempts. Edge cases that look reasonable and are not. Inputs that test whether the agent will violate policy if asked in the right tone.
For CX agents, adversarial evaluation has stopped being a research exercise. It is now an operational requirement. The OWASP Top 10 for LLM Applications lists prompt injection as the number-one risk in the category. NIST published an Adversarial Machine Learning taxonomy that maps the threat surface in detail. Anthropic and OpenAI both publish red-team reports as part of their model release process.
The CX-specific failure modes look like this. A customer copy-pastes a message that tells the agent to ignore its prior instructions and issue a refund. A support ticket includes hidden text that instructs the agent to reveal another customer’s information. A reasonable-sounding question pushes the agent into giving advice it is not authorized to give. Each of these failure modes has live examples. Each one is testable. None of them are caught by sampling 2 percent of conversations after the fact.
The teams shipping the most reliable CX AI today run an adversarial eval suite on every meaningful change. They keep the suite small enough to run fast and broad enough to cover the categories that matter. They update it whenever a new failure mode shows up in production. That tight loop, sometimes called eval-driven development, is what separates the resilient programs from the brittle ones.
How Evaluation Connects to Governance
Most CX leaders treat evaluation and governance as different conversations. They are not. Evaluation outputs are the audit evidence that governance frameworks ask for.
ISO/IEC 42001 is the AI management system standard. It requires documented evidence that AI systems perform within defined risk tolerances. The EU AI Act requires risk management, monitoring, and post-market surveillance for high-risk systems. The NIST AI Risk Management Framework lays out the same logic in a US context. All three need the same artifact: a continuous, documented stream of evidence that the AI is behaving as specified.
Sample QA cannot produce that artifact at scale. Continuous evaluation can. The same eval that catches a tone violation in production also goes into the audit binder when the regulator arrives. That dual-use property is why mature programs build evaluation and governance on the same backbone. We covered the broader picture in our piece on ISO 42001 enterprise AI governance for CX.
The OECD State of Implementation of the AI Principles makes the same point at a policy level. Organizations that operationalize their AI principles through measurement outperform organizations that publish them and stop there.
The Team That Runs Evaluations
CX teams that take evaluation seriously end up with a small group focused on it. The shape is consistent across the organizations we work with.
A CX operations lead owns the business outcomes. They define what the agent must achieve, what risks the agent must avoid, and what the deployment gates are. They translate executive priorities into evaluation requirements.
A conversation designer or content engineer writes the rubrics. They are the people closest to the language the agent uses and the policy the agent is supposed to follow. They turn “be helpful” into testable behaviors. This is also where the skills gap conversation designers are seeing becomes most acute. Designers who only know dialog flows struggle here. Designers who can write precise rubrics do not.
An evaluation engineer or QA analyst with new skills builds and maintains the eval system. They construct golden datasets, run the eval pipeline, monitor for regressions, and feed results into the release process. In many organizations this person was a QA analyst, content analyst, or data analyst whose role expanded. The McKinsey State of AI survey consistently shows that organizations capturing real AI value invest in role reshaping rather than waiting for new technical hires. Evaluation engineering is one of the clearest examples.
In smaller organizations, two people can wear these hats. In larger organizations, they need to be different people sitting on the same team. The pattern that fails is leaving evaluation to whoever wrote the prompt.
What Anthropic’s Outcomes Feature Is Pointing Toward
Anthropic shipped automated grading inside Claude Managed Agents earlier in 2026, the Outcomes feature we covered in our analysis of the managed-agent stack. The signal in that feature is bigger than the feature itself.
Vendors are baking continuous evaluation into the platform. The teams that already have rubrics and golden datasets get the value immediately. The teams that do not get a dashboard they cannot interpret.
The same pattern is showing up across the industry. Microsoft’s Responsible AI Standard treats measurement as a core engineering practice rather than a compliance afterthought. Google’s Responsible AI practices emphasize ongoing evaluation across the model lifecycle. Deloitte’s State of Generative AI in the Enterprise survey ranks evaluation maturity among the most consistent predictors of successful CX AI deployment.
The CX organizations that build the rubric and golden-dataset muscle now will be ready when the platform side does what platforms always do, which is offer the measurement layer to whichever buyer can describe what to measure.
What Leaders Should Do in the Next 90 Days
The temptation is to design a perfect program for every workflow. Resist it. The teams that ship a working eval program ship one workflow first. Three actions get you there.
Pick one high-volume AI workflow. Refund handling. Order status. Account verification. Pick the one with the most volume and the clearest policy. That becomes the pilot.
Write five rubrics that matter. Not 50. Five. Each rubric covers one behavior that is easy to score and important to your customers. Disclosure on first turn. Correct policy citation. Escalation when uncertain. Tone within brand guidelines. No invented product names. The list will change. Start with five.
Build a golden dataset of 50 to 100 examples. Pull representative real conversations, redact them, and label the ideal outcome for each. This is the work most teams skip. It is also the work that pays compounding interest. Every future eval runs against this baseline. Every model update gets compared to it. Every prompt change gets scored.
Then wire the evaluation into a deployment gate. No change reaches production without a passing eval. That single rule turns the program from a reporting tool into a control point. Our piece on the 30-minute AI CX audit is a useful first read for surfacing where the gaps are today.
What to Stop Doing
- Stop calling a 2 percent sample review quality assurance. It is a sample. Call it that.
- Stop letting prompt changes ship without a rubric pass. The rubric is the unit test.
- Stop treating evaluation as a downstream reporting function. It is an upstream control point.
- Stop running one layer of evaluation and reporting all-green to the board. Three layers, or none.
- Stop hiring AI QA roles that look exactly like contact-center QA roles. The job is different.
The Bottom Line
The CX organizations that catch failures before customers do are not smarter. They have a better operating model. They built rubrics. They built golden datasets. They wired evaluation into deployment. They treat continuous measurement as the substrate of every AI decision.
The organizations that have not are still running a quality model designed for a smaller, slower world. The math has changed. The agents are faster. The volume is higher. The failure modes are silent. Sample review will not catch what comes next.
Build the eval program now, or read about your own failure in someone else’s case study.
ICX helps enterprise CX teams design AI evaluation programs that hold up under operational, regulatory, and reputational pressure. Visit the services page or book a discovery call to discuss what rubric design, golden datasets, and deployment gates look like for your AI workflows.
Key Takeaways
- Sample-based quality review was built for a slower, smaller world. It cannot keep up with AI agents handling thousands of conversations a day with high input variability and silent failure modes.
- Real AI agent evaluation is structured, continuous, and rubric-driven. It tests inputs, behaviors, and outcomes against a defined baseline on every meaningful change.
- Three evaluation layers matter for CX: input evaluation catches misclassification, behavior evaluation catches policy violations, and outcome evaluation confirms problem resolution. Most teams run only the third, on a small sample.
- Rubric design is the hardest and highest-leverage part of evaluation. “Be helpful” is not a rubric. Five binary or graded behaviors that any reviewer can score consistently are.
- Adversarial evaluation is no longer optional. Prompt injection, jailbreaks, and edge-case inputs are operational risks that sample review does not catch.
- Evaluation and governance are the same workstream. The same outputs that catch quality drift satisfy ISO 42001, the EU AI Act, and NIST AI RMF audit requirements.
- The 90-day starting move is small and concrete. One workflow, five rubrics, 50 to 100 golden examples, and a deployment gate that no change can bypass.
Sources
- Anthropic. Develop Tests and Evaluations
- Anthropic. Prompt Engineering Overview
- OpenAI. Evals: Framework for Evaluating LLMs
- NIST. AI Risk Management Framework (AI RMF 1.0)
- NIST. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
- ISO/IEC. ISO/IEC 42001: AI Management System
- European Parliament and Council. EU AI Act, Article 50: Transparency Obligations
- European Union. Artificial Intelligence Act, Full Text
- OWASP. Top 10 for Large Language Model Applications
- Stanford HAI. AI Index Report 2025
- Gartner. 2026 Hype Cycle for Agentic AI
- Gartner. Gartner Predicts Over 40 Percent of Agentic AI Projects Will Be Canceled by End of 2027
- McKinsey & Company. The State of AI: How Organizations Are Rewiring to Capture Value
- Deloitte AI Institute. State of Generative AI in the Enterprise
- OECD. State of Implementation of the OECD AI Principles: Three Years On
- Microsoft. Responsible AI Standard
- Google. Responsible AI Principles and Practices
- Harvard Business Review. AI in the Enterprise: Strategy and Governance
- MIT Sloan Management Review. Artificial Intelligence and Business Strategy
Human Review & AI Assistance
This article was developed using AI-assisted research, analysis, and drafting workflows. A human reviewer evaluated the content before publication. Sources were reviewed for accuracy at the time of publication. While every effort has been made to ensure accuracy, readers should independently verify information before making business, legal, financial, regulatory, or technical decisions.
Frequently asked questions
What is an AI evaluation, in plain language?
An AI evaluation, or eval, is a structured test of how an AI system behaves on a defined set of inputs. It compares the system's output against a rubric or a ground-truth answer. A simple eval might check whether the AI gave the correct refund policy. A behavior eval might check whether the AI stayed within tone and disclosure rules. Anthropic's evaluation guide and OpenAI's evals framework provide the technical primitives. The skill that matters most is writing the rubric that captures what good looks like for your customers.
Why is sample-based QA not enough for AI agents?
Sample QA was designed for human agents handling hundreds of conversations a day. Reviewing 2 percent gave a statistically reasonable read on quality. AI agents handle thousands of conversations a day, with much more variability in input and much faster failure modes. By the time a quarterly sample review surfaces a problem, the problem has already affected thousands of customers. Continuous, automated evaluation closes that gap. It also produces an audit trail that a sample review cannot.
Who in a CX organization should own AI evaluation?
Three roles need to be involved. A CX operations lead owns the business outcomes the evals measure. A conversation designer or content engineer writes the rubrics that translate policy into measurable behavior. An evaluation engineer or QA analyst with new skills builds and maintains the evaluation systems. In smaller organizations these roles overlap. In larger organizations they sit on the same team. The role that does not work is the traditional QA analyst doing manual reviews without rubric or tooling support.
What are the three layers of AI agent evaluation CX teams need?
Input evaluation checks whether the user request was understood, classified, and routed correctly. Behavior evaluation checks whether the agent followed disclosure, tone, escalation, and policy rules. Outcome evaluation checks whether the customer's problem was actually resolved, measured against ground truth or human review. Most teams run only the third layer, and they run it on a small sample. Running all three, continuously, is the operating standard the better-performing CX organizations are converging on.
How does AI evaluation connect to AI governance and compliance?
Evaluation outputs are the audit evidence that compliance frameworks require. ISO/IEC 42001, the EU AI Act, and the NIST AI Risk Management Framework all call for documented evidence that an AI system performs within defined risk tolerances. Without continuous evaluation, that documentation is mostly theory. With it, the same outputs that catch quality drift also satisfy the auditor. Evaluation is the bridge between AI operations and AI governance, not a separate workstream.
How should CX leaders start in the next 90 days?
Start small and concrete. Pick one high-volume AI workflow, define five behaviors that matter most for that workflow, write a rubric for each, and build a golden dataset of 50 to 100 representative examples. Run the eval on every model update and every prompt change. Use the results to gate deployment. That single workflow becomes the template for the rest of the program. Most teams that try to design a perfect program for everything at once never ship the first eval.
