Green that lies

Three leaderboard rows were scored over 1543 items and ranked against rows scored over 1,223

public validation 20 + private validation 300 + test 1223 = 1543

The TaxEval v2 page documents its test set as 1,223 samples, and lists two validation sets alongside it. Three model rows do not fit 1,223 under any epoch count; they fit 1543 exactly — which is the sum of all three components the page itself lists, and is prime, so it is not 1,223 counted some whole number of times. Those rows sit in the same sorted column as 128 rows that do fit 1,223.

No row on the site publishes a sample count. Every row publishes accuracy and a standard error, and for a proportion those two printed numbers are a system of two equations in two integers, which has a unique solution over the range the printed precision brackets. The counts are therefore not estimated. They are solved. The control that rules out rounding: each model is scored twice, in two tabs, over the same set — and the recovered count agrees in 128 of 128 cases, and 47 of 47 on their GPQA board. A rounding artefact would wander between tabs. It does not.

Separately on that GPQA board, where the page states 198 questions, one model solves to 197 — and 197 is prime.

The denominator was recovered from the decimal itself, and 34 rows do not match the set they claim

Same method, run against a published benchmark table: 34 rows out of 813 recover a denominator that is not the set size the page states. Two of the recovered values are 197 and 1319, both prime, so neither can be explained as the documented set scored a whole number of times.

The finding has a corollary that matters more than the rows: rounding a score to four decimal places or fewer destroys the denominator. Below that precision the recovery is no longer unique, and a reader who tries anyway will get a confident wrong answer. Publishing one integer per row costs nothing and removes the need for anyone to invert an error bar to learn what a score was computed over.

92 items failed, so the run reported 8 items at perfect accuracy

The denominator counted survivors. Of 100 items, 92 failed before they could be scored and left the set entirely; the remaining 8 all passed. The dashboard therefore reported a complete run of 8 items at accuracy 1.000 — correct arithmetic over the wrong set, and no field anywhere in the view says the run was asked for 100.

A suite that collected zero tests passes its own threshold

passRate = 0/0 = NaN  ·  NaN < threshold → false → gate does not fire

The pass rate over an empty suite is not a number, and every comparison against not-a-number is false — including the one that is supposed to fail the build. So the single check whose job is to notice that quality dropped is structurally incapable of firing on the case where quality is unmeasured. A misconfigured glob, a renamed directory, a bad filter: the run goes green and reports nothing missing.

A metric returned False and the reporting layer wrote None over it

The judged verdict was negative. On the way from the metric to the report, the value became None — and a missing verdict is not treated as a failure, it is treated as nothing to report. The run reads as passing. The layer that overwrote the answer is the layer whose only job is to display it, which is why nobody inspecting the metric ever sees the problem.

100% over a suite that shrank from ten cases to three

A pytest status hook reported every collected case as passing, over a suite that had lost seven of its ten cases before execution. The summary line is honest about the three and silent about the seven, and there is no printed total to compare it against. A green wall after a refactor that quietly stopped collecting most of the suite looks exactly like a green wall after a good run.

LangChain confirmed it in writing the next day:

We agree that the pytest plugin can present an incomplete test run misleadingly, and we're tracking a product-quality fix to report incomplete suites accurately. LangChain, 14 August 2026

The same reply contains the sentence I now quote more often than the finding itself. Declining to treat it as a security matter, they noted that the behaviour “does not alter pytest's underlying test results or exit status” — which is precisely correct, and precisely the reason a reader believes the green. The exit status is clean because the work that would have failed was never attempted.

The flush buffer was doing double duty as the row counter

reported = rows % 128  ·  128 rows → “0 rows processed”

Rows are flushed in blocks of 128, and the number surfaced to the user is what is left in the buffer rather than what went through it. A run whose size is an exact multiple of the block size therefore reports that it processed nothing — and a run of 130 reports 2. The buffer is an implementation detail that became the published denominator.

A bare except swallowed the row that belonged in fourth place

One submission’s row never reached the public leaderboard: the code path that assembles the table catches every exception without distinguishing them, so a single malformed field removes an entrant instead of reporting an error. The board looks complete. There is no count of rows attempted next to the count of rows shown, so the absence cannot be seen from the page.

A broader read of the same leaderboard turned up seven separate defects, including one submission appearing as two rows, and independently verified entries ranking below unverified ones because the sort ignores the verification flag.

A skipped case counted toward the success total for eleven months

A case that never ran was tallied with the cases that passed. The behaviour sat in released code for eleven months, which is the part worth dwelling on: this class of defect does not announce itself, does not page anyone, and does not appear in a bug tracker, because the observable symptom is a number that looks fine.

A monitor that stays green over a set that shrank to nothing

All three points I raised were confirmed. Their own summary of the behaviour is the clearest statement of the whole class:

…stays green until someone re-runs it. Weights & Biases, 11 August 2026

That is the definition of the problem. A status that can only be falsified by re-doing the work is not a status; it is a memory of a status, presented as a current one.