How to Evaluate LLM Output Without Fooling Yourself

A parse rate jumped from 0.74 to 0.98 right after I added a retry loop to an extraction pipeline. It looked like a clean win, and it was the wrong conclusion. The first-attempt parse rate, before any retry, was already 0.98: the model-serving stack had changed underneath. What the retries really fixed was a different number, the schema-valid rate, which went from 0.20 to 0.88.

That's the core problem with deciding how to evaluate LLM output: the numbers move for many reasons, and the flattering explanation is the easiest one to believe. These rules come from a research project on local models for German job-ad extraction that produced more than 55 auditable evaluation runs and no retracted numbers. Each rule exists because of a specific near-miss.

Rule 1: numbers are claims, so re-derive them

Never repeat a result from memory, yours or anyone else's. Query the stored results first, and for any number someone will rely on, recompute it from the raw outputs through a second path.

Two near-misses made this rule:

• A remembered "F1 of 0.51" turned out to hold only for the 56% of documents that parsed at all. On all documents, it was lower. That's a denominator trap: the right number for the wrong set.
• "Even the bigger model didn't improve F1" was simply wrong; it improved by 0.22.

Both were nearly sent to a supervisor. "Verified" now means two independent paths agree to the last reported decimal.

Rule 2: attribution before celebration

When a metric improves after a change, check what else changed before crediting your change: the model version, the serving stack, the data, the prompt template, the hardware. The retry-loop story at the top is the textbook case. Log first-attempt metrics alongside final ones, so you can always tell drift from mechanism.

Reporting your own confound is not a weakness in a write-up. A reviewer will find it if you don't, and finding it yourself is what makes the rest of the numbers credible.

Rule 3: compare on identical items, or don't compare

An A/B claim is only valid on the same items. If one system was evaluated on 50 documents and the other on 6, restrict the comparison to the 6 they share. In the project, a headline of "0.684 against 0.461 on the same six documents" required recomputing the baseline on exactly those six. Comparing against the baseline's 50-document average would have been quietly wrong, in the flattering direction.

Rule 4: instructions don't bind models; checks in code do

Telling a model "only change the labels" is a request. If correctness depends on it, check it mechanically.

In the project, a judge model was told to only re-type entities in an extraction, and not to add, remove or rename them. A check compared the set of (id, text) pairs before and after each review. The judge broke its one rule in 35 of 49 reviews. The pipeline stayed correct because the check rejected those outputs, and the 71% violation rate became one of the project's findings precisely because the check existed to measure it.

The general pattern, which also defends against prompt injection:

Rule 5: decide the threshold before you look

What reliability does the use case need? Decide before anything writes to a database. In the project, an early run of Gemma 4 26B on the 50-document development sample, before any retry logic, returned valid JSON for 74% of documents at an entity F1 of 0.466 (the final numbers are in Gemma vs Mistral vs Qwen). That's useful for assisted annotation, where a person corrects pre-filled output. It's not enough for autonomous writes. The numbers didn't change; the threshold decided what they were good for.

The same project showed that size sets a floor but doesn't rank models: with the same prompt on 50 job ads, Llama 3.1 8B reached an entity F1 of 0.150 against 0.292 for Gemma 4 26B, yet on another matched set of 50 ads Gemma also beat NVIDIA's far larger Nemotron 3 Super (0.459 against 0.212). Measure on your own data before choosing by parameter count.

Rule 6: keep evaluation data clean

List what the system you're evaluating has already seen. If that includes the answers, isolate it. In the project, when asked to do an extraction by hand in a session whose context already held the gold labels and the failure analyses, the right move was to refuse in that session and run a fresh, cold-context agent per document instead, then verify from the logs that each agent read exactly one input file.

Do the same in code: make the test, development and example pools disjoint by assertion, so the evaluation script refuses to start if they overlap. Leakage prevention belongs in code, not in discipline.

Rule 7: store outputs that answer tomorrow's questions

Store the complete output for every item, not a snippet. Early runs in the project stored 200-character snippets and are now permanently unauditable. Later runs stored the full JSON per document, which made it possible to test a new merging heuristic offline in seconds instead of re-running the model for hours.

Two companions to this rule: never silently change what a metric means (new semantics get a new name, so months of logged runs stay comparable), and report every number with its denominator, how it was verified and its caveats, in a table like this:

What about LLM-as-a-judge?

Using a model to grade outputs is useful for scale, and it's an evaluated system in its own right. Treat the judge like any other model step: give it a narrow job, check its invariants in code (Rule 4), and compare its scores with human judgments on a sample before trusting it on thousands of items.

FAQ

What's the minimum evaluation setup for an LLM feature?

A fixed set of real examples with expected results, a script that runs them, and the results stored per item. Run it before and after every prompt, model or setting change. Twenty to fifty good examples beat a thousand vague ones.

How do I know if an improvement is real?

Compare on the same items, check what else changed, and re-derive the number from stored outputs. If the improvement survives all three, it's probably real.

Should I trust benchmark scores when choosing a model?

As a first filter. Your own task, your own data and your own failure modes decide. A model can lead a public benchmark and still fail your output format a third of the time.

What should an evaluation report contain?

Every number with its denominator, how it was verified and its caveats, the comparisons made on identical items, and what you couldn't control. One page someone can absorb is worth more than an exhaustive document they won't read.