A model emitted a function call on 228 of the 240 questions that exist to catch exactly that. It was scored correct on all 228.

The Berkeley Function-Calling Leaderboard is the standard public scoreboard for tool use, and its Hallucination Measurement block is worth 10 % of every model’s headline score. The check behind that block does not ask whether a model abstained. It asks whether the text it produced could be parsed. For three of the 109 models on the board those are opposite answers, and the block supplies between 79 % and 100 % of what they scored. Everything below is read out of BFCL’s own published code, CSVs and result files.

Free · no key required · every file below is public · 26 August 2026 · the rest of the register →

Finding 1 — the page describes its own headline number wrongly

From the leaderboard page, fetched today, verbatim:

Overall Accuracy is the unweighted average of all the sub-categories. For details on score composition, please refer to our blog.

The code says otherwise. From eval_runner_helper.py, the last thing it computes before writing the CSV:

# TODO: @HuanzhiMao adjust the weights
total_overall_accuracy = calculate_percentage_weighted_accuracy(
    [
        overall_accuracy_non_live,
        overall_accuracy_live,
        total_irrelevance,
        overall_accuracy_multi_turn,
        overall_accuracy_agentic,
    ],
    [10, 10, 10, 30, 40],
    display_na_if_category_missing=False,
)

I rebuilt the published Overall Acc column both ways from the five block scores that sit in the same CSV rows, for all 109 models:

The worst single row is Amazon-Nova-2-Lite: published 27.10, unweighted 51.14. A reader who takes the page at its word and reconstructs a model’s score from its parts is out by twenty-four points.

Then the sentence sends that reader for “details on score composition” to the V3 blog, which opens the relevant section like this:

Note: This leaderboard composition is now outdated. For the most updated composition, please refer to the BFCL V4 blog.

The correct composition is in the V4 blog, which the leaderboard page does not link: “Overall Score = (Agentic × 40%) + (Multi-Turn × 30%) + (Live × 10%) + (Non-Live × 10%) + (Hallucination × 10%)”. The same post also says “Within-Category Calculation: Unweighted Average” — which is where the word on the leaderboard is very likely a survivor from. It is true inside a category and false across them, and the leaderboard sentence is about across.

Finding 2 — the hallucination check tests the parser, not the model

Irrelevance is the category that asks a model to not call a function: the user’s request cannot be served by any tool on offer, and the right behaviour is to decline. Here is how a run is judged, from eval_runner.py:

except Exception as e:
    # Decode failed, which means the model output is not in valid function call format
    contain_func_call = False
    decode_error = str(e)

# irrelevance test means no function call outputted
if "irrelevance" in test_category:
    success = not contain_func_call
else:
    success = contain_func_call

The comment is honest about what it assumes: unparseable output is treated as “no function call”. That assumption is doing all the work. Tried to call a tool and produced malformed syntax and correctly declined to call a tool arrive at this branch as the same event, and the branch calls both of them a pass.

Falcon3-1B-Instruct (FC) — rank 106, and the cleanest case

Its score file records {"accuracy": 0.954, "correct_count": 229, "total_count": 240} on non-live irrelevance. I pulled the 229 entries it was credited for and looked at what the model actually emitted:

Two of the 228, as stored in BFCL’s own result file:

id=irrelevance_0   '<tool_call>\n<tool_call name="determine_body_mass_index">\n  {\n
                     "type": "dict",\n    "properties": {\n      "weight": 70.0,\n
                     "height": 1.75\n    }\n'
id=irrelevance_1   '<tool_call>\n<tool_call name="math.solve" arguments={"a": 1, "b": 2,
                     "c": 3}/>\n</tool_call>'

That is a hallucinated tool call, named and argumented, scored as an abstention.

The eleven entries Falcon was marked wrong on are the ones where its syntax happened to come out valid. The metric is running backwards: within this category, the better a model formats a call it should not have made, the worse it scores.

Llama-3.1-Nemotron-Ultra-253B-v1 (FC) — rank 108, and the limit case

Every entry of this model’s result files holds an empty list — 240 of 240 irrelevance, 884 of 884 live irrelevance, 400 of 400 simple_python, 16 of 16 live relevance, with no other value anywhere. Four score files from that same run, unedited:

BFCL_v4_simple_python      {"accuracy": 0.0, "correct_count":   0, "total_count": 400}
    error: ["Invalid syntax. Failed to decode AST. expected string or bytes-like
            object, got 'list'"]
BFCL_v4_live_relevance     {"accuracy": 0.0, "correct_count":   0, "total_count":  16}
    error: ["Invalid syntax. Failed to decode AST when it should have. expected
            string or bytes-like object, got 'list'"]
BFCL_v4_irrelevance        {"accuracy": 1.0, "correct_count": 240, "total_count": 240}
BFCL_v4_live_irrelevance   {"accuracy": 1.0, "correct_count": 884, "total_count": 884}

One failure mode, one run, three verdicts. The first two files name the failure in an error string. The last two are silent, and they are the ones that reach the score: irrelevance is 100.00 %, and 10 % of 100.00 is the model’s entire published Overall Accuracy of 10.00 %. Every other block is 0.00.

Where it lands on the board

Exactly three models of 109 score 0.00 on relevance detection — never once recognising a call it should make — while scoring high on irrelevance. That combination is the signature, and it is the whole population of it:

The last row is the one that makes it matter. Gemma-3-1b is a one-billion-parameter model that genuinely does the task some of the time — it calls functions correctly on a fifth of the non-live set and recognises a legitimate call 37.5 % of the time. It is ranked below all three, including below a model that produced an empty result on every single entry of the benchmark. The bottom of this leaderboard is upside down.

The control: does the metric misbehave everywhere, or only there?

Only there — and saying so is the point. A criticism that cannot be bounded is not a finding. The top ten rows of the same board, computed the same way:

Every one of them contributes 10.9–13.4 % of its score from the block — its nominal 10 % weight, give or take — and every one of them recognises real calls 62–94 % of the time. For a model that emits parseable syntax, the check measures what its name says. The defect is not “BFCL is broken”. It is that the check silently changes meaning at exactly the point where a model’s formatting collapses, and there is nothing in the published number that says which regime a row is in.

The gate

The failure is a missing distinction, so the fix is to restore it rather than to blacklist today’s three offenders — a blacklist loses to the next model. Three states, not two:

• ABSTAINED — decoded successfully, and the decoded call list is empty. This is the behaviour the category exists to reward, and only this should score.
• MALFORMED — decoding raised. The model produced something; whether it was trying to call a tool is unknown from the score alone, so it must not be silently credited. Report it, count it separately, leave it out of the numerator.
• CALLED — decoded to one or more calls. Fails, as it does today.

The distinction costs nothing to compute: the exception is already caught and its message is already stored in decode_error. It is thrown away one line later, when the branch collapses to a boolean. And the second, cheaper half of the fix has no code in it at all — publishing a per-row relevance figure beside the irrelevance one would have made all three of these rows obvious on sight, because irrelevance 100.00 / relevance 0.00 is not a profile any real abstention policy can produce.

Reproduce it

No API key and no account. Everything is a public file:

# the five leaderboard CSVs, as served to the page
https://github.com/HuanzhiMao/BFCL-Result   → 2025-12-16/

# the model outputs and the score files, per model, per category
2025-12-16/result/tiiuae_Falcon3-1B-Instruct-FC/non_live/BFCL_v4_irrelevance_result.json
2025-12-16/score/tiiuae_Falcon3-1B-Instruct-FC/non_live/BFCL_v4_irrelevance_score.json
2025-12-16/score/nvidia_llama-3.1-nemotron-ultra-253b-v1/non_live/BFCL_v4_simple_python_score.json

# the scoring code
gorilla/berkeley-function-call-leaderboard/bfcl_eval/eval_checker/eval_runner.py
gorilla/berkeley-function-call-leaderboard/bfcl_eval/eval_checker/eval_runner_helper.py

Line 1 of a *_score.json is the category summary; every line after it is an entry the run marked wrong. So the set credited as correct is the entries absent from lines 2 onward — which is how the 228-of-229 count above is built, with no judgement of mine anywhere in it.

What this page does not claim

• No claim that the weights are undocumented. They are stated exactly right in the BFCL V4 blog. The finding is that the leaderboard page contradicts them in its own description and links to a post that announces itself as outdated.
• No claim of hidden behaviour. The decode branch does what its code comment says. I am arguing about what the resulting number means to a reader, not about whether the code matches its own intent.
• No claim that excluding relevance from the score is an error. That is BFCL’s documented design — the V4 blog lists Live Relevance without a weight. I use the relevance column only as a diagnostic, because it is what separates a real abstainer from a broken one.
• No claim about why Nemotron’s outputs are empty. Not one entry carries a zero token count — the irrelevance set runs 11–111 output tokens per entry, simple_python 18–92, live_relevance 20–299 — so text was generated and then lost somewhere between the model and the stored result: model, adapter or harness, I cannot tell from outside and do not need to. The claim is only about what the score files then record.
• Ministral-8B is the weakest of the three examples and I have not leaned on it. It is credited 240/240 and 884/884, but its outputs are a mixture: 84 of the 240 are the empty string, while many others are genuine prose refusals (“I’m sorry, but I can’t provide real-time data…”) that deserve to be scored correct. Only its relevance score of 0/16 puts it in the table at all.
• No claim about the top of the leaderboard. The control section is there precisely because the ranking of every competitive model appears unaffected.
• I did not run any model. Every number here is read from BFCL’s published result and score files for the 2025-12-16 snapshot, cross-checked against the live leaderboard CSVs.