Test AI features that vary every run: 20 fixed cases, written pass rules, repeat runs

Today in AI (2026-10-22)

Anthropic's Claude Duet watches your apps and does tasks, asking before acting: Your customers will soon expect software that takes action on its own with an approval step, not just a chat box that answers questions.
OpenAI's models and Codex are coming to Google Cloud: You can mix OpenAI and Gemini models inside one cloud account, which makes switching AI providers easier and less risky.
PHP patches two AI-found security holes already being exploited: If your product or website runs on PHP, ask your developer to upgrade today, because attackers already have working exploits.

If your AI feature gives a different answer every time, you can't test it by trying it once and nodding. A single good answer tells you almost nothing, because the same input might fail on the next run.

The fix is to stop checking answers and start measuring rates. You build a fixed set of test questions, write down what counts as a pass, run each question many times, and track the percentage that pass. Engineers call this an "eval" (short for evaluation). It is a spreadsheet and a script, not a research project.

Here is the setup, step by step.

Why one run proves nothing

An AI model works by picking likely words, not by looking up a stored answer. Ask it the same question twice and you can get two different responses. Usually both are fine. Sometimes one is wrong, rude, or invented.

That means a demo that went well is a sample of one. The question a founder needs answered is "how often does this go wrong?" You can only answer that by running the same input many times.

There is a second trap. Models change. Providers update them, and your own prompt edits shift behavior too. Anthropic just shipped its first official API version since 2023 and says it plans to release versions more often, so developers can handle breaking changes before a new model requires them (source). The direction is clear: the thing underneath your product will keep moving. A repeatable test is how you notice when it moved.

The evaluation setup

Three parts: a test set, pass criteria, and repeat runs.

1. Build a test set of real inputs

Start with 20 cases. That is enough to catch obvious problems and small enough to build in an afternoon.

Pull 10-12 from real or realistic user requests. If you have no users yet, write what you expect them to type, including sloppy phrasing and typos.
Add 4-5 hard cases: vague requests, requests with missing information, and long messy inputs.
Add 3-4 "should refuse or escalate" cases: things the feature must not do, like giving legal advice, revealing another customer's data, or answering outside its scope.

Store them in a spreadsheet with two columns: the input, and a note on what a good response looks like. Add to it every time a real failure shows up in production. That list becomes your most valuable asset, because it records every way your product has broken.

2. Write pass criteria before you run anything

Decide what "pass" means in writing, in advance. If you judge after seeing the output, you will talk yourself into accepting weak answers.

Good criteria are checkable statements, not feelings:

Must include the order number from the input.
Must not mention a price that isn't in our price list.
Must ask a follow-up question when the date is missing.
Must stay under 120 words.
Must refuse and point to a human when asked for a refund over the limit.

Avoid "sounds helpful" or "good quality." Those can't be scored the same way twice.

Some criteria a computer can check automatically (length, required fields, banned words). Others need judgment, like "is the tone appropriate?" For those, you can have a second AI model grade the answer against your written rule, then read a sample yourself to confirm the grader agrees with you. Treat the AI grader as a helper, not the final word.

3. Run each case several times

Run every test case 5-10 times and record pass or fail each time. With 20 cases and 5 runs, that is 100 results. Your score is the share that pass.

Now you can see things a single run hides:

A case that passes 5 of 5 is solid.
A case that passes 3 of 5 is flaky, and flaky is where users get hurt.
A case that passes 0 of 5 is a clear bug you can fix.

Set a release bar before you start. For example: no case in the "must refuse" group is allowed to fail even once, and the overall pass rate must not drop from the last release. You choose the numbers based on how costly a mistake is in your product. A typo in a marketing draft is cheap. A wrong answer about a customer's money is not.

Common mistakes

Testing only the happy path. Your 20 cases must include inputs that try to break the feature.
Changing the pass rules after seeing results. Write them first and keep them fixed between runs, or you can't compare versions.
Running once and calling it done. The whole point is repetition.
Trusting the AI grader blindly. Read a sample of its verdicts every time you change the grading rule.
Never updating the test set. Each real failure should become a new case.
Skipping the check after "small" changes. A one-line prompt tweak or a model upgrade can shift behavior. Rerun the set every time.

What has changed recently

Two shifts make this more urgent.

First, AI features are doing more than answering questions. Anthropic's new Claude Duet watches your connected apps and does tasks on its own, asking for approval before certain actions (source). When a feature acts rather than talks, a wrong answer becomes a wrong action. Your "must refuse or ask first" test cases matter more, and you should test that the approval step actually fires.

Second, you will have more models to choose from. OpenAI's models and Codex are coming to Google Cloud's Model Garden (source). More choice means you will want to compare providers on your own cases instead of trusting a benchmark. Your 20 cases, run against two models, tell you which one fits your product. If you also want to check the security side of AI-built features, my post on testing data access with two accounts covers that gap.

What to do this week

Open a spreadsheet and write 20 test inputs for your AI feature, including 3-4 it must refuse.
Next to each, write the pass rule as a checkable statement.
Ask your developer (or your coding agent) to run each input 5 times and save the results with pass or fail.
Set your release bar in writing, then fix the cases that fail most often.
Rerun the whole set before every release and every model or prompt change.

Do this once and you will know, with a number, how reliable your feature is. Most founders shipping AI features can't say that.