Green that lies

The best-behaved framework I have tested still hands the leaderboard a survivor mean, and calls it stderr 0.000

Inspect is the one that does most of this right, and the entry is here for that reason rather than in spite of it. Its default is safe: fail_on_error defaults to None, and a run that hits a sample error stops with status: error and publishes no results at all. I verified that first, because a gate that only ever fires has not been tested. It also does something no other tool in this register does — it writes down what it lost, in results.completed_samples beside results.total_samples.

The gap is one step downstream, and it is a gap you walk into by doing the sensible thing. A long agent run cannot use that default: one flaky container and hours of work are discarded, so real evals set a tolerance. With the tolerance above the actual error rate — the only setting that survives contact with a real harness — ten samples go in, seven raise inside the solver, three are scored, and this is what comes back:

fail_on_error=None  →  status=error, no results   (correct)
fail_on_error=0.10  →  status=error, no results   (correct)
fail_on_error=0.75  →  status=success   accuracy 1.000   stderr 0.000
                       completed_samples 3  ·  total_samples 10

Two things travel out of that run. The counts, which stay in the log file. And the number, 1.000, which is what a leaderboard row, a system card, or a CI threshold actually consumes. Nothing downstream is obliged to open the log. The stderr is the sharper end of it: 0.000 is a claim of certainty, computed over three items, at the exact moment the run is least certain it measured anything.

I ran a second, identical evaluation with nothing failing. Ten of ten scored, accuracy 1.000, stderr 0.000, status: success. The two headline lines are byte-for-byte the same. That is the finding: not that Inspect discards the count, but that the count it keeps never reaches the place where someone decides whether a model is ready.

So this entry ships with the fix rather than the complaint. A gate reads any .eval log and replaces the point estimate with the interval the metric could occupy over the set that was asked for — the survivors’ sum is fixed, each lost sample lands somewhere in [0, 1], and nothing in the arithmetic assumes anything about why a sample died:

always_right.accuracy        reported 1.0000
denominator                  3 scored of 10 asked for   [OVER SURVIVORS]
over the set that was asked for, the same run is somewhere in [0.3000 .. 1.0000]

It carries seven controls that probe the gate itself in both directions, including a negative control that feeds the checker a deliberately wrong answer and demands the word DEAD back — and it exits 4, refusing to give any verdict, if one of them stops working. Both fixtures are real Inspect logs, generated against mockllm/model at no API cost, so anyone can rerun the whole thing in a minute without a key.

The full measurement, both runs and the source →

Three leaderboard rows were scored over 1543 items and ranked against rows scored over 1,223

public validation 20 + private validation 300 + test 1223 = 1543

The TaxEval v2 page documents its test set as 1,223 samples, and lists two validation sets alongside it. Three model rows do not fit 1,223 under any epoch count; they fit 1543 exactly — which is the sum of all three components the page itself lists, and is prime, so it is not 1,223 counted some whole number of times. Those rows sit in the same sorted column as 128 rows that do fit 1,223.

No row on the site publishes a sample count. Every row publishes accuracy and a standard error, and for a proportion those two printed numbers are a system of two equations in two integers, which has a unique solution over the range the printed precision brackets. The counts are therefore not estimated. They are solved. The control that rules out rounding: each model is scored twice, in two tabs, over the same set — and the recovered count agrees in 128 of 128 cases, and 47 of 47 on their GPQA board. A rounding artefact would wander between tabs. It does not.

Separately on that GPQA board, where the page states 198 questions, one model solves to 197 — and 197 is prime.

The denominator was recovered from the decimal itself, and 34 rows do not match the set they claim

Same method, run against a published benchmark table: 34 rows out of 813 recover a denominator that is not the set size the page states. Two of the recovered values are 197 and 1319, both prime, so neither can be explained as the documented set scored a whole number of times.

The finding has a corollary that matters more than the rows: rounding a score to four decimal places or fewer destroys the denominator. Below that precision the recovery is no longer unique, and a reader who tries anyway will get a confident wrong answer. Publishing one integer per row costs nothing and removes the need for anyone to invert an error bar to learn what a score was computed over.

92 items failed, so the run reported 8 items at perfect accuracy

The denominator counted survivors. Of 100 items, 92 failed before they could be scored and left the set entirely; the remaining 8 all passed. The dashboard therefore reported a complete run of 8 items at accuracy 1.000 — correct arithmetic over the wrong set, and no field anywhere in the view says the run was asked for 100.

A suite that collected zero tests passes its own threshold

passRate = 0/0 = NaN  ·  NaN < threshold → false → gate does not fire

The pass rate over an empty suite is not a number, and every comparison against not-a-number is false — including the one that is supposed to fail the build. So the single check whose job is to notice that quality dropped is structurally incapable of firing on the case where quality is unmeasured. A misconfigured glob, a renamed directory, a bad filter: the run goes green and reports nothing missing.

A metric returned False and the reporting layer wrote None over it

The judged verdict was negative. On the way from the metric to the report, the value became None — and a missing verdict is not treated as a failure, it is treated as nothing to report. The run reads as passing. The layer that overwrote the answer is the layer whose only job is to display it, which is why nobody inspecting the metric ever sees the problem.

100% over a suite that shrank from ten cases to three

A pytest status hook reported every collected case as passing, over a suite that had lost seven of its ten cases before execution. The summary line is honest about the three and silent about the seven, and there is no printed total to compare it against. A green wall after a refactor that quietly stopped collecting most of the suite looks exactly like a green wall after a good run.

LangChain confirmed it in writing the next day:

We agree that the pytest plugin can present an incomplete test run misleadingly, and we're tracking a product-quality fix to report incomplete suites accurately. LangChain, 14 August 2026

The same reply contains the sentence I now quote more often than the finding itself. Declining to treat it as a security matter, they noted that the behaviour “does not alter pytest's underlying test results or exit status” — which is precisely correct, and precisely the reason a reader believes the green. The exit status is clean because the work that would have failed was never attempted.

The flush buffer was doing double duty as the row counter

reported = rows % 128  ·  128 rows → “0 rows processed”

Rows are flushed in blocks of 128, and the number surfaced to the user is what is left in the buffer rather than what went through it. A run whose size is an exact multiple of the block size therefore reports that it processed nothing — and a run of 130 reports 2. The buffer is an implementation detail that became the published denominator.

A bare except swallowed the row that belonged in fourth place

One submission’s row never reached the public leaderboard: the code path that assembles the table catches every exception without distinguishing them, so a single malformed field removes an entrant instead of reporting an error. The board looks complete. There is no count of rows attempted next to the count of rows shown, so the absence cannot be seen from the page.

A broader read of the same leaderboard turned up seven separate defects, including one submission appearing as two rows, and independently verified entries ranking below unverified ones because the sort ignores the verification flag.

A skipped case counted toward the success total for eleven months

A case that never ran was tallied with the cases that passed. The behaviour sat in released code for eleven months, which is the part worth dwelling on: this class of defect does not announce itself, does not page anyone, and does not appear in a bug tracker, because the observable symptom is a number that looks fine.

A monitor that stays green over a set that shrank to nothing

All three points I raised were confirmed. Their own summary of the behaviour is the clearest statement of the whole class:

…stays green until someone re-runs it. Weights & Biases, 11 August 2026

That is the definition of the problem. A status that can only be falsified by re-doing the work is not a status; it is a memory of a status, presented as a current one.