How to QA AI Writing Without Code
In this article
You need three things to QA AI writing without code. First, a short rubric that says what “good” means. Second, a set of real test questions. Third, a spreadsheet that logs what fails. Check every name, number, quote, and link by hand. When a draft is wrong, fix the cause. A second AI can help you grade. But a person signs off.
QA stands for quality assurance. For AI writing, QA is a habit. You check AI drafts against a written standard. You do it before a customer sees them. You also track the mistakes so they stop coming back. You do not need an engineer for this. You need one hour and a clear idea of what you want. You also need a person who will really read the drafts.
Why AI writing needs QA
AI writes fast and sounds sure of itself. That is the problem. A confident wrong answer is easy to miss. These mistakes have been public and expensive.
Tech site CNET used AI to write money articles. It published 77 stories that way. In January 2023, CNET issued corrections on 41 of the 77 stories. The Verge reported the corrections.
In May 2025, the Chicago Sun-Times ran a summer reading list. It came from a content partner. NPR reported the result. Ten of the 15 books on it did not exist. The writer said he used AI. He called it a “huge mistake on my part.”
In October 2025, Deloitte made a deal about a report. It agreed to partly refund the Australian government. The report cost A$440,000. The AP said the report had a made-up quote. The quote was from a federal judge. It also cited reports that did not exist.
You do not need code to catch errors like these. A person only had to look up the books. Then check the quote and click the links.
Step 1: Write a rubric before you read anything
A rubric is a short scoring list. It turns “I’ll know good when I see it” into rules. Anyone on your team can apply them the same way.
Anthropic makes Claude. Read its guide to building evaluations. It says good success criteria are “specific” and “measurable.” Do not use “sounds good” as a check. Write rows that you can mark pass or fail.
Here is a starting rubric for most small businesses:
- Accurate. Every fact matches a source that you trust.
- Answers first. The answer is in the first two sentences.
- On voice. It sounds like your brand, not like a press release.
- Follows the rules. It uses no banned words. It makes no promises that you cannot keep. It gives no prices that you did not approve.
- Knows its limits. It hands off to a person for refunds and legal questions. It also hands off upset customers.
Want to mark these five rows for one draft? Use our free AI Writing QA Scorecard.
Step 2: Build a test set from real questions
A test set is a list of real questions. Each question has the answer that you would accept. Collect 20 to 50 questions. Take them from your inbox and chat logs. You can also ask the people who answer the phone. Use a spreadsheet. Write the accepted answer next to each question.
Add a few hard questions on purpose. Anthropic’s guide says, “Don’t forget to factor in edge cases!” Here are three for a business. Think of an angry customer. Think of a question your policy does not cover. Think of a refund the AI cannot give.
Run the whole test set every time something changes. A change can be a new prompt or a new AI model. It can also be an updated source document, such as a policy page. Do you only check after a customer complains? Then you are not doing QA. You are doing cleanup.
Step 3: Fact-check names, numbers, quotes, and links
Look back at the three stories above. Every error was a name or a number. Or it was a quote or a source. In those four spots, AI often fills a gap. It uses something that sounds right.
Check those four spots by hand, every time:
- Names. Search every person, book, product, and company. Make sure each one exists and has the right spelling.
- Numbers. Trace every stat and price to its source. Can you not find the source in five minutes? Cut the number.
- Quotes. Find the exact words in the original. Close is not good enough.
- Links. Click every link. Make sure the page loads. It must say what the sentence claims.
Can another AI check the work?
Yes, with limits. Anthropic’s guide says human grading is the most flexible and high quality. But it is “slow and expensive.” AI grading is “fast and flexible,” the guide says. The guide gives one rule for AI grading: “test to ensure reliability first then scale.” The steps below follow that rule.
Use a second AI without code in four steps:
- Paste the rubric and the draft. Put both into a chat with a second AI.
- Ask for a grade on each row. The second AI marks each rubric row pass or fail. Ask for a one-sentence reason for each grade.
- Grade 10 drafts yourself. Grade 10 of the same drafts by hand. Compare your grades with the AI grades.
- Hand off the first pass. Do you agree on almost all 10 drafts? Then the second AI can do the first pass on the rest.
A second AI is good at catching tone problems. It also catches missing steps and broken rules. It is weak at catching made-up facts. It can make the same kind of mistake. A person must check names and money. A person must check legal terms and health.
Step 4: Log every failure and fix the cause
A QA log is a spreadsheet of every mistake you find. With a log, you stop fixing the same error weekly. Use these columns:
- Date and the question that failed.
- What went wrong, in one line.
- Type: wrong fact, wrong tone, made-up content, or missed handoff.
- Cause: the prompt, a source document, or the AI model.
- Fix and the date that you retested.
The cause column matters most. Say the AI keeps quoting an old return policy. If you rewrite the answer, the error comes back. Fix the old policy page instead. We cover this in why context engineering is the missing layer in enterprise AI. Your AI can only be as right as what it reads.
Want to score one draft and build a log row in your browser? Use our free AI Writing QA Scorecard. It uses the same five rubric rows. It uses the same four risk spots.
Who should QA AI writing at your business?
At Intelligent CX Consulting, our view is clear. One person knows your customers and your voice best. That person should own AI QA. That is usually a support lead or a content writer. It is rarely IT. An engineer can tell you that a score went up. A person close to your customers knows more. Did the answers get better? Or did they only get longer?
First you build your rubric and test set. After that, plan on about 30 minutes a week. Compare that to one wrong answer that spreads online. The cost is small.
We make the case for this role in What a Content Writer Does Now. Does your AI run a customer chat? Start with how to choose your chatbot’s brain. Then you will know what bot you are testing. Next in this series: how to hire a content writer in 2026.
FAQ
What does QA mean for AI writing? QA means quality assurance. For AI writing, QA means checking AI text against a clear standard. You do this before customers see it. QA also tracks mistakes so you can fix their causes.
How do I fact-check AI-generated content? Check the four places where AI most often goes wrong. These are names, numbers, quotes, and links. Search each name. Trace each number to a source. Find each quote in the original. Click every link. Cut anything that you cannot confirm.
Can I use AI to check AI writing? Yes, for a first pass. Give a second AI your rubric. Ask it to mark each row pass or fail. It gives a reason. Spot-check its grades. A person must still check facts, money, and legal terms.
How many test questions do I need? Start with 20 to 50 real questions from your customers. Add a few hard edge cases. Each time you find a failure, add a new question. The test set grows with what you learn.
How often should I test my AI writing? Run your full test set after every change. A change can be to the prompt, the AI model, or a source document. Between changes, spot-check live AI writing once a week. A weekly spot-check catches most problems early.
Frequently asked questions
What does QA mean for AI writing?
QA means quality assurance. For AI writing, QA means checking AI text against a clear standard. You do this before customers see it. QA also tracks mistakes so you can fix their causes.
How do I fact-check AI-generated content?
Check the four places where AI most often goes wrong. These are names, numbers, quotes, and links. Search each name. Trace each number to a source. Find each quote in the original. Click every link. Cut anything that you cannot confirm.
Can I use AI to check AI writing?
Yes, for a first pass. Give a second AI your rubric. Ask it to mark each row pass or fail. It gives a reason. Spot-check its grades. A person must still check facts, money, and legal terms.
How many test questions do I need?
Start with 20 to 50 real questions from your customers. Add a few hard edge cases. Each time you find a failure, add a new question. The test set grows with what you learn.
How often should I test my AI writing?
Run your full test set after every change. A change can be to the prompt, the AI model, or a source document. Between changes, spot-check live AI writing once a week. A weekly spot-check catches most problems early.
Get an email when we publish a new post.