Good Metric, Narrow Oracle: When the Weaker Model Scored Higher
Recently I arrived at a result that makes a good case study for how much the oracle matters when testing software. In the case of AI evaluation this becomes especially pertinent since the outputs are high-dimensional and allow for multiple interpretations.
1. A surprising result
I was comparing different LLMs as the summariser in an evaluation suite I built for a customer-review analysis system, so that I could determine which model produced the highest-quality summaries. For this, I had decided to use the faithfulness metric: a well-regarded metric that checks whether the claims in a summary are supported by the source text. The surprising outcome was that the weaker, cheaper model sometimes scored higher in faithfulness than the larger, more expensive one.
Why was this happening? My setup had one configuration pairing the strong summariser (i.e., the more capable model) with a strong LLM judge; another paired the weak summariser with the same strong judge. So we had the same review, the same grader, and the same threshold for both configurations. The only variable that differed was the model writing the summary.1
The faithfulness metric could take values between 0 and 1, where 1 means the summary is perfectly faithful to the source. I used a threshold of 0.7 to determine whether a summary was considered faithful or not.
With one particular case, the strong summariser scored an average of 0.76 across three runs, while the weak one scored 1.00 across all. With a passing threshold of 0.7, both summaries passed; that’s fine, but there was a clear difference in the scores, and the weaker model was a clear winner. If that was true across the board, it would be a strong argument for using the weaker model, right?
This case definitely wasn’t a one-off. Across 16 test cases, the weak summariser beat the strong one on five of them.
So: the weaker model was not winning the evaluation, but it was winning on specific cases, and the margin on this particular one — 0.24 points — was the largest of the five. That’s a concrete signal that something was going on here.
So I opened up the strong model’s summary to observe which of its individual claims the judge had marked unsupported. I realised that the metric was just doing what it had been designed to do, which turned out to be the real problem.
2. Looking under the hood
The review in question read like this:
Placed the order on March 1st. The estimated delivery was March 5th. On March 8th I was told the item hadn’t shipped yet. It shipped March 12th and arrived March 19th — nearly three weeks after I ordered. By then I had already bought a replacement elsewhere. I returned it immediately. The refund still hasn’t appeared after 14 days.
The faithfulness metric works by decomposing a summary into atomic statements, then checking each one against the source. If the source supports it, then the statement passes; if the statement goes beyond what the source actually says, the statement fails. The score is the fraction of statements that pass.
Here is the full breakdown for both models on this case.
Let’s look at the two failures:
“The estimated delivery window for the customer’s order was 5 days” — the source gives an order date and an estimated delivery date, but does not state a duration. The model converted the delivery estimate into a span of time rather than a single arrival date. However, neither the “5 days” nor that framing appears in the source.
“The customer has been waiting over 14 days for a refund” — the source says the refund “still hasn’t appeared after 14 days.” The word “still” already signals an ongoing, unresolved wait, so “has been waiting” is a fair read. The problem is one word: “over.” The source pins the elapsed time at 14 days; “over 14 days” claims that more than 14 have passed. It’s a small shift, but it asserts a quantity that is not supported by the source. The model could have said “has been waiting for 14 days” and it would likely have passed, but the addition of “over” made it fail.
Note that these are not wild hallucinations. A careful human reader might write both sentences, and the strong model produced several more of the same kind across the other reviews. What does this all say about the strong model’s summarisation ability?
3. Is something wrong with the metric?
Let’s look at why these failed. “5 days” genuinely isn’t there in the source. “Over 14 days” genuinely isn’t there either. In both cases, something got added that wasn’t in the source.
The weak model did the opposite: each statement stayed within the source’s factual bounds. “Placed an order on March 1st.” “Nearly three weeks to arrive.” “Refund has not appeared after 14 days.” The judge passed all six, earning the 1.00.
So is the metric wrong? It helps to run through what that could mean:
A miscomputed score? Five of seven statements pass, and 5/7 = 0.71, which is exactly the reported score.
A badly placed threshold? Both summaries pass it, so the gap between their scores is not related to where the threshold sits.
A biased judge? The same judge scored both summaries, and the disputed verdicts can be checked statement by statement: the two failed claims added "5 days" and "over 14 days", neither of which appears in the source. Also, the judge is the same model as the strong summariser, so we would expect that any self-preference would have favoured the model that lost instead of the one that won.
These checks make the score defensible: the metric is doing its job. The problem that remains concerns the part of quality outside its field of view.
Reading those six statements as a standalone summary, they capture the order date, the arrival delay, the shipping update, the replacement, the return, and the missing refund. What they fail to convey is the force of the grievance: a promised March 5th delivery, no shipment by March 8th, dispatch only on March 12th, and arrival after the customer had already given up. That restraint makes the summary faithful, but a support agent working from it would have too little information to understand what actually went wrong. In contrast, the strong model gives a richer sense of severity and consequence, even though it also takes a few unsupported steps.
This means that the faithfulness metric captured exactly what it was designed to capture, but didn't fully capture what we care about. It didn't capture what would make the summary useful and informative, and not just faithful.
4. Measurement has a direction
An evaluation is only as good as the oracle behind it. An oracle is a tool that helps you decide whether a result is a problem; it is a heuristic, which makes it useful, incomplete, and capable of pointing the wrong way.2
Faithfulness is a narrow oracle that recognises one kind of problem. Look at the shape of that check (Fig. 1). It runs in one direction. It takes each claim in the summary and asks whether the source supports it. Every statement the oracle examines comes from the summary.
This single direction is the root of the problem, as it provides only one angle on the relationship between source and summary. It can tell you whether everything in the summary is true, but it has no way to ask the reverse question — let’s call this coverage: did the summary actually keep the things that mattered in the source? For example, a summary could omit the estimated March 5th delivery, the March 8th “not shipped” update, the March 12th dispatch, or the March 19th arrival. Faithfulness would stay perfectly happy as long as the summary did not add anything that isn’t in the source.
This is why the weak model’s 1.00 is quite easy to earn, which is a small instance of Goodhart’s law: when a measure becomes a target, it stops being a good measure. The surest way to make every claim supportable is to make fewer claims, and to keep each of them close to the wording of the source.
For summaries, though, the consistency we care about has two directions: calling a summary consistent with its source is to say the two agree from both perspectives. Faithfulness measures one of those perspectives.
And it leaves the other one unmeasured.
5. Engineering a better oracle
The point about oracles is that often no single one is enough, so a tester carries several and allows them to cover for each other. Anyone who has written automated checks has met this pattern before: think about a happy-path check that passes as it should while the corresponding error handling is left unchecked, or one that confirms the response came back while it overlooks a missing field.
So, while faithfulness may be one good oracle, a thin summary that omits what the reader needs calls for an additional one: does the summary stay useful and informative enough to serve the reader’s task? That question is an instance of what James Bach and Michael Bolton call the Purpose heuristic — whether a result is consistent with what the product is for.3
As this quality criterion resists direct measurement, we can reach for a proxy: a coverage metric that runs from source to summary, asking whether the key claims survived. In this way, it catches omissions that contribute to a summary failing its purpose. Paired with faithfulness, these two can guard against opposite pressures:
faithfulness catches overreach — claims the source doesn’t support;
coverage catches omission — source content the summary did not capture.
Faithfulness and coverage can be represented as two measurable directions (Fig. 2). Together they give us a stronger measure of quality, but a residual remains between that composite score and human judgement.
This residual is easy to observe. To conceptualise it, consider the edge case: a summary identical to the source scores 1.00 on both metrics. Although nothing was actually summarised — no condensing, no prioritising, no highlighting of what matters — neither oracle can see that. What fills the gap to a better evaluation is the judgement of knowing what matters, at what level of detail, and for whom; and even that is only ever an approximation.
6. Fit the oracle to the risk, then keep questioning it
Every oracle is a heuristic, and which heuristic you choose depends on the risk you’re guarding against.
Seeing a stronger model sometimes perform worse on faithfulness eventually led me to look for a more appropriate oracle better suited to the task: one that combines faithfulness with coverage. My working hypothesis is that such an oracle would give the stronger model a fairer test. My defensible conclusion is that faithfulness on its own cannot answer the question I was asking.
Keep in mind that criteria written specifically for this case would likely serve even better;4 a composite of standard metrics is merely the floor.
On the other hand, if we’re only interested in faithfulness — and there are indeed times when a generic metric like this makes sense, e.g., when sampling production traces5 — the weak model can be just fine.
In every case, and as with any test and monitoring infrastructure, treat the evaluation itself as a living thing: something to probe, distrust, and revise. Choose the oracle that fits the risk and gives prompt feedback; reach for more when one dimension leaves too much above the plane. Above all, keep human judgement in the loop, and remember that no metric is a substitute for it.
The strong summariser was Claude Sonnet (claude-sonnet-4-6); the weaker, cheaper one was Llama 3.2, run locally through Ollama. Both configurations used the same judge, claude-sonnet-4-6. Full code, dataset, and evaluation harness: https://github.com/gsiam/review-sentiment-eval. The analysis this series draws on: https://github.com/gsiam/review-sentiment-eval/blob/main/docs/model-configuration-analysis.md. Interactive explorer of the scores discussed here: https://gsiam.github.io/review-sentiment-eval/explorer/measurement-validity/
Cem Kaner, "The Oracle Problem and the Teaching of Software Testing," 2012. https://kaner.com/?p=190
Michael Bolton, “FEW HICCUPPS,” 2012. https://developsense.com/blog/2012/07/few-hiccupps
Hamel Husain and Shreya Shankar, "Should I use 'ready-to-use' evaluation metrics?" in "LLM Evals: Everything You Need to Know," 2026. https://hamel.dev/blog/posts/evals-faq/#q-should-i-use-ready-to-use-evaluation-metrics
Hamel Husain and Shreya Shankar, “How can I efficiently sample production traces for review?” in “LLM Evals: Everything You Need to Know,” 2026. https://hamel.dev/blog/posts/evals-faq/#q-how-can-i-efficiently-sample-production-traces-for-review