LLM Evals: Testing AI Features Before You Ship

Your AI feature worked perfectly in the demo. Responses were crisp, the format was right, the tone was exactly what you wanted. You shipped it.

Two weeks later, a user pastes an input your prompts never anticipated, and the output is confidently wrong. You have no way to know how often this is happening.

That’s the gap evals fill.

What Evals Actually Are

An eval is an automated test for an AI feature output. Unlike unit tests, there’s usually no single correct answer — so evals measure quality on a scale, not just pass/fail. The core question is: does this output meet the bar you’d accept in production?

Most teams skip evals and substitute vibe checks: they run a few manual tests, the results look good, and they ship. This works until scale or edge cases reveal that “looks good in five examples” is not the same as “works reliably across ten thousand inputs.”

The gap between “looks good” and “works correctly”

Language models are good at sounding confident. An output can be fluent, well-formatted, and completely wrong at the same time. Without a systematic way to measure correctness, you’re relying on the assumption that your manual review caught the representative cases.

It rarely does.

What an eval test suite contains

A basic eval suite has three things: a set of test inputs, expected output criteria for each input, and a scorer that measures whether the criteria were met. The scorer can be as simple as checking that a required field is present in the JSON, or as complex as asking a second model to rate the response on a rubric.

Three Types of Evals That Matter

Not every feature needs all three. Start with the simplest type that gives you real signal and add complexity only when you hit its limits.

Deterministic checks

These are the cheapest and most reliable evals. They check things that should always be true regardless of input: the output is valid JSON, a required field is present, the response is under a token limit, a prohibited phrase doesn’t appear. Write these first. They catch entire categories of failures in milliseconds.

Good candidates: structured output validation, safety filter checks, format compliance.

LLM-as-judge

When the quality you care about can’t be reduced to a rule, you can use a second model to score the first model’s output. You give the judge an input, the output, and a rubric (“rate this summary for accuracy and conciseness on a scale of 1–5”). The judge returns a score and usually an explanation.

LLM-as-judge is flexible and cheap at scale compared to human review. Its weakness: judge models tend to be lenient toward fluent, confident-sounding text even when it’s wrong. Mitigate this by giving the judge explicit rubric criteria and calibrating its scores against a small set of human-labeled examples first.

For AI development projects we ship at quickdev, LLM-as-judge is the workhorse eval method for anything involving summarisation, classification, and tone — tasks where rules-based checks can’t capture what “good” actually means.

Human-in-the-loop spot checks

Automated evals catch known failure modes. Human review catches unknown ones. The practical pattern is sampling: route a small percentage of production outputs (1–5%) to a review queue where someone on your team scores them weekly. When you see a new failure mode, you add a deterministic or judge-based eval to catch it automatically going forward.

This is how your eval suite grows without becoming a maintenance burden.

Building Your First Eval Pipeline

You don’t need an eval framework, a separate service, or a third-party tool to start. You need a spreadsheet and a script.

Start with your failures, not your goals

The best place to find eval test cases is the inputs that already gave you trouble. Did a user complain about a specific output? Add that input. Did a prompt change break something unexpected in QA? That’s a test case. Starting from real failures means your eval suite is immediately capturing meaningful signal.

The golden dataset

Collect 20–30 real inputs that span the range of what your feature handles. Include normal cases, obvious edge cases, and anything that made you nervous when you shipped. For each input, write down the criteria an acceptable output must meet. That document is your golden dataset.

Keep it in version control alongside your prompts. When you change a prompt, run the eval suite against the golden dataset before you ship. A regression on even three or four cases is a strong signal to investigate before the change goes live.

Running evals in CI

Once you have a scorer script and a golden dataset, plugging them into CI is straightforward: run the eval job on every pull request that touches a prompt or model config, fail the build if the score drops below a threshold, and post the score delta as a comment. This makes eval regressions visible in the same place as other code review signals.

For teams using DevOps and cloud infrastructure, this typically means a GitHub Actions job that calls your LLM provider, scores the outputs, and writes results to a dashboard. The whole pipeline can be set up in a day.

When Evals Are Enough (and When They’re Not)

Evals measure what you’ve thought to measure. They don’t measure what you haven’t. A feature can pass all its evals and still fail in production — because real users send inputs that your test set never anticipated.

Feature-level vs system-level evals

Feature-level evals test a single AI step in isolation: does the summariser produce good summaries? Does the classifier label this category correctly? These are fast to run and easy to debug.

System-level evals test the full pipeline: does the end-to-end flow produce a useful result for the user? These are slower, more expensive, and closer to what users actually experience. You need both.

Connecting evals to production monitoring

Evals catch problems before you ship. Production monitoring catches problems after. The two should be connected: when production logs surface a new failure pattern, that pattern goes back into the eval suite as a new test case.

This feedback loop is what turns a fragile AI feature into a reliable one. Not a perfect prompt — a tested, monitored system.

---

Yaniv Amrami is founder of quickdev. He has built and shipped AI-powered products for Israeli startups since 2023 and has helped teams replace vibe-check QA with structured eval pipelines that catch regressions before they reach users.