Article by
AI evals (evaluations) are structured tests that measure how well an AI model performs a task, so developers can tell whether a new version is actually better than the last. They combine automated benchmarks, which score answers against fixed solutions, with human evaluation, where experts grade open-ended responses against a rubric of explicit criteria.
Every new AI model arrives with claims that it is smarter, safer, or more helpful than its predecessor. AI evals are how those claims get tested. Without them, a developer can’t distinguish a real improvement from a model that simply sounds more confident or catch a regression that made one skill worse while another improved.
This guide explains how AI model evaluation works, including the difference between benchmarks and human evaluation, why benchmarks lose their usefulness over time, what an evaluation rubric is, and why designing a good rubric is often half the work.
How developers know a model has improved
Developers know a model has improved by running the same set of evals before and after a change and comparing the results. A change might be new training data, a different fine-tuning method, or a system prompt update. If scores rise on the skills that matter and don’t fall elsewhere, the change is kept. A typical evaluation cycle has four steps:
Set a baseline: Run the current model on a fixed eval suite and record its scores.
Make one change: Retrain, fine-tune, or adjust the model.
Re-run the same evals: Use identical tasks and grading so the comparison is fair.
Check for regressions: Look for skills that got worse, not just the ones that improved.
The fourth step is where many problems surface. A model tuned to be more helpful can become less cautious, and a model tuned for coding can lose fluency in conversation. Evals are designed to catch these trade-offs before users do.
Benchmarks vs. human evaluation
Benchmarks and human evaluation answer different questions, and serious AI model evaluation uses both. A benchmark is a fixed dataset of tasks with known correct answers that a computer can grade automatically. Human evaluation uses domain experts to judge outputs that don’t have a single correct answer.
Automated benchmarks | Human evaluation | |
|---|---|---|
Best for | Math, multiple choice, code that can be tested | Open-ended writing, advice, reasoning quality, tone |
How it’s scored | Automatically, against fixed answers | By experts using a rubric |
Strength | Fast, cheap, repeatable | Captures nuance and real-world usefulness |
Weakness | Saturates, can leak into training data | Slower, needs clear criteria to stay consistent |
Benchmarks also wear out. The Stanford 2025 AI Index reports that AI systems solved just 4.4% of the coding problems on the SWE-bench benchmark in 2023 and 71.7% in 2024. When top models approach a benchmark's ceiling, it stops distinguishing between them and researchers need harder tests or different methods.
There’s also a second, quieter problem: benchmark questions published online can end up in training data. A model that has effectively seen the answers will score well without having learned the skill. Human evaluation on fresh, expert-written tasks is one of the main defenses against this.
What’s an evaluation rubric?
An evaluation rubric is a list of explicit criteria that defines what a good response looks like for a specific task, with points assigned to each criterion. Instead of asking an evaluator "Is this answer good?", a rubric asks a series of narrower questions that can each be answered consistently. Well-built rubrics share four components:
Specific criteria: Each one checks a single, observable feature of the response.
Point values: More important criteria carry more weight.
Negative criteria: Penalties for things a response must not do, such as inventing facts.
Clear thresholds: Enough detail that two evaluators would mark the same response the same way.
Rubrics now underpin some of the most demanding evals in AI. OpenAI's HealthBench evaluates models on 5,000 realistic health conversations, each graded against a conversation-specific rubric written by physicians. It was built with 262 physicians who had practiced in 60 countries and contains 48,562 unique rubric criteria, including negative-point criteria for undesirable behavior. Even when a model does the final grading, the rubric it applies comes from human experts.
A rubric example for a simple task
A worked example shows how a rubric turns "this seems fine" into a precise score.
The task:
Summarize the customer email below for a support manager in no more than three bullet points, and state whether a refund is required.
The email:
"Hi, the coffee grinder I ordered arrived yesterday with a cracked hopper. I've attached photos. I don't want a refund. I'd like a replacement, ideally before Saturday, because it's a birthday gift. Thanks, Dana."
The rubric:
# | Criterion | Points |
|---|---|---|
1 | States the core problem (hopper arrived cracked) | +3 |
2 | States the requested resolution (replacement, not refund) | +3 |
3 | Mentions the time constraint (needed before Saturday) | +2 |
4 | Answers the refund question correctly (no refund required) | +2 |
5 | Uses three bullet points or fewer | +1 |
6 | Invents details or emotions not in the email | −3 |
7 | Recommends a refund despite the customer's request | −2 |
The maximum score is 11. Now consider this model response:
Customer's grinder arrived with a cracked hopper.
They are asking for a refund or a replacement.
The customer seems very upset and may leave a negative review.
At a glance, it reads like a reasonable summary. Scored against the rubric, it earns 3 points for criterion 1 and 1 point for criterion 5. It misses criterion 2 by suggesting the customer would accept a refund, misses the Saturday deadline, and never answers the refund question. It also loses 3 points under criterion 6 for inventing frustration and a review threat. The final score is 1 out of 11.
That gap between "looks fine" and "scores 1 out of 11" is exactly what rubrics exist to expose. A support manager acting on this summary might issue the wrong resolution and miss the deadline that mattered most to the customer.
Why a good rubric is half the work
A rubric determines what the model learns to value, so a flawed rubric trains a flawed model. If the criteria reward length, models learn to pad. If they never penalize invented details, models learn that confident fabrication costs nothing. Writing the rubric is where expert judgment enters the evaluation.
The most common rubric failures are easy to spot once you know them:
Vague criteria: "Response is helpful" invites disagreement between evaluators. "Response states the requested resolution" does not.
Double-barreled criteria: "Accurate and concise" combines two checks, so a response that is accurate but long has no correct score.
Overlapping criteria: Two criteria that check the same thing double-count it and distort the total.
Missing negative criteria: Without penalties, a response can earn points while also doing harm.
Unverifiable criteria: "Reflects deep understanding" cannot be checked consistently from the text alone.
Difficulty matters as much as wording. As one of our solution engineers explains in a guide to deliberately "breaking" a model, tasks need to be challenging enough that the model often makes mistakes, but not so difficult that it never succeeds.
An eval the model always passes teaches nothing, and so does one it always fails. Rubrics also need a plan for cases where guidelines do not cover a situation. Check out our guide to handling ambiguous AI tasks to learn how evaluators approach those edge cases.
Where AI evaluation jobs come in
Human evaluation at scale needs people who can apply rubrics consistently and, increasingly, people who can write them. That demand is behind the growth of AI evaluation jobs: freelance projects where domain experts grade model outputs, compare responses, and design evaluation criteria in their own fields.
The skills involved overlap with quality assurance for AI — attention to detail, consistency across many judgments, and the ability to explain why a response falls short. Domain expertise is what separates strong evaluators: a physician catches a missing emergency referral, a developer catches an unhandled edge case, and an editor catches a summary that distorts the source.
Article by
Mindrift Team
