BigCodeBench is a code-generation benchmark: a model writes a function, and a
hand-written unit-test suite decides whether it got it right. The suite is the entire measuring
instrument — there is no other judge. In task 541 the suite has five cases, and a
solution consisting of one try and return [] satisfies all five, because
the mock that was supposed to make the interesting case interesting never took effect. One line
of Python explains it, and the whole thing reruns in two seconds with no key, no account and no
harness.
The short version. The solution binds its name with
from pkgutil import iter_modules. The test tries to control it with
@patch('pkgutil.iter_modules'). Patching the attribute on the module does not rebind a
name that was already copied into the solution's namespace at import time, so the mock is a silent
no-op — no error, no warning, just a decorator that does nothing. With it inert, the loop that
is the task never executes under that case, and the case's assertion —
assertFalse(len(modules_added) > 0) — is satisfied by the emptiness it was
written to disprove.
What this is not. Not a security issue, not a claim that
BigCodeBench's scores are wrong overall, and not a bug in unittest.mock —
patch is behaving exactly as documented. It is one task out of 1,140 whose test cannot
fail for the reason it exists, and the reason it cannot is a mistake common enough to be worth
naming.
Task 541 asks for a function that imports a package, walks its submodules, and appends each
submodule path to sys.path, returning the names it added. Here is the shipped
code_prompt and canonical_solution, joined as the harness joins them and
otherwise unedited — pulled from the dataset today:
import os
import sys
import importlib
from pkgutil import iter_modules # ← the name is bound HERE, now
def task_func(package_name):
added_modules = []
try:
package = importlib.import_module(package_name)
except ImportError:
raise ImportError(f"The package '{package_name}' is not installed! Please install the package first using 'pip install {package_name}'")
for _, module_name, _ in iter_modules(package.__path__): # ← and used from HERE
module_path = os.path.join(package.__path__[0], module_name)
if module_path not in sys.path:
sys.path.append(module_path)
added_modules.append(module_name)
return added_modules
And the test case that is supposed to exercise that loop — the shipped
test field, with its own comments left in, because they are part of the evidence:
@patch('importlib.import_module')
@patch('pkgutil.iter_modules') # ← patches the attribute, not the bound name
def test_package_module_addition(self, mock_iter_modules, mock_import_module):
# Create a mock for the package with a __path__ attribute as a list
package_mock = MagicMock()
package_mock.__path__ = ['mocked_path'] # Ensure this is a list
# Configure import_module to return the package mock when any module name is passed
mock_import_module.return_value = package_mock
# Setup the mock for iter_modules to simulate finding modules in a package
mock_iter_modules.return_value = [
(None, 'module1', True), # Simulate a package has 'module1'
(None, 'module2', True) # Simulate a package has 'module2'
]
# Call the function under test
modules_added = task_func('numpy')
# Perform your assertions here
# For example, assert that modules were "added" (imported)
self.assertFalse(len(modules_added) > 0)
The author's own comment states the intent one line above the assertion that contradicts it: assert that modules were “added”, followed by an assertion that none were.
The first decorator works. importlib.import_module is called as an attribute lookup
on the module object, so replacing that attribute replaces what the code reaches. The second cannot:
by the time the test runs, iter_modules is a separate name in the solution's own
namespace, pointing straight at the original function. Rebinding pkgutil.iter_modules
leaves it untouched.
So the case runs against the real iter_modules, over the mocked
__path__ = ['mocked_path'], which is a directory that does not exist. It finds nothing.
The loop body never runs. modules_added comes back empty — and the assertion, in a
case named module_addition, demands exactly that.
The test passes because the thing it is testing did not happen.
One inert mock would be a dead line of code, not a finding. It matters here because nothing else
in the suite covers the loop either. The five cases, and what each one does when
task_func is replaced by a stub that returns []:
| case | what it asserts | against return [] | |
|---|---|---|---|
test_package_module_addition | with mocks in place, len(added) > 0 is false | empty list → assertion holds | PASSES |
test_nonexistent_package | ImportError is raised | the stub keeps the import check → raises | PASSES, correctly |
test_empty_package | len(added) == 0, or ImportError | empty list → assertion holds | PASSES |
test_module_path_in_sys_path | for module in added: … assert | loop over an empty list → body never runs | PASSES VACUOUSLY |
test_no_duplicates_in_sys_path | for module in added: … assert | loop over an empty list → body never runs | PASSES VACUOUSLY |
Four of the five are satisfied by an empty return value, each for its own reason, and the fifth
only checks that a missing package raises. There is no case anywhere in the suite that asserts a
particular module was added — the last two iterate over whatever the function chose to
return and check that it is consistent with itself, which the function guarantees by construction:
it appends to sys.path and to added_modules on the same line pair.
All three come out of one self-contained script, printed below in full. It embeds the shipped canonical solution, the shipped test suite, and one stub, and runs the suite against each.
[1] canonical, as shipped:
CANONICAL: ran=5 failures=0 errors=0 -> PASS
[2] wrong stub (core sys.path logic deleted) -- should FAIL, but:
STUB : ran=5 failures=0 errors=0 -> PASS
[3] canonical with the mock made effective (import pkgutil):
CANON+eff: ran=5 failures=1 errors=0 -> FAIL
fired: test_package_module_addition :: AssertionError: True is not false
Run [2] is the defect. The stub is not a subtle variation; it is the task with its subject removed:
import importlib
def task_func(package_name):
try:
importlib.import_module(package_name)
except ImportError:
raise ImportError("not installed")
return []
It never mentions sys.path, never enumerates a submodule, never imports
pkgutil at all. A model that produced it would be scored as having solved the task.
Run [2] on its own admits a second reading: perhaps the suite is fine and the stub is somehow
right. Run [3] closes that off. It changes one thing — the canonical solution's import style,
from from pkgutil import iter_modules to import pkgutil plus
pkgutil.iter_modules(...), which is the same function, called the same way, differing
only in whether the existing @patch can reach it. Nothing in the test is touched.
| run | solution | test suite | result | what it establishes |
|---|---|---|---|---|
| [1] | canonical, as shipped | unmodified | 5 pass | the instrument reads green on the right answer |
| [2] | core deleted, return [] | unmodified | 5 pass | it reads green on a wrong answer too |
| [3] | canonical, module-style import | unmodified | 1 fail | the mock can bite; it simply never did |
The failure in [3] is AssertionError: True is not false on
test_package_module_addition — the mock finally injects its two fake submodules,
the loop finally runs, two names come back, and the case that asked for none rejects the official
solution. That is the second half of the defect: the assertion does not merely fail to catch a
wrong answer, it is pointing the wrong way. Written as its name implies —
assertEqual(sorted(modules_added), ['module1', 'module2']) — it would be a real
test, and the stub in [2] would die on it.
I scanned all 1,140 tasks in the v0.1.4 set for the same shape: a @patch('mod.name')
decorator aimed at a name the solution binds with from mod import name. Three tasks
carry it.
| task | patched | effective? | consequence | |
|---|---|---|---|---|
| 541 | pkgutil.iter_modules | no | a stub with the task's core deleted passes 5/5 | FALSE POSITIVE |
| 186 | geopy.distance.geodesic | no | the case with the dead mock only asserts that keys are tuples and values are floats — true of real distances too | HARMLESS |
| 407 | openpyxl.load_workbook | no | the case with the dead mock only asserts the return value is a str — true of the real conversion too | HARMLESS |
In both of those the mock is dead code and nothing rides on it: the case carrying it asserts a
property that holds with or without it, and other cases in the same file assert real values —
186 checks that two points a degree apart come out under 200 km and antipodal points over
10,000, 407 reads the written CSV back and compares it to [['Hello', 'World']]. A
gutted solution dies on those. I am not presenting either as a defect, and that is the distinction
worth keeping: a scan finds the pattern, but whether the pattern matters is decided one
task at a time. 541 is the one where the inert mock changes the outcome.
Two changes, independent of each other; the test needs the first regardless of whether anyone touches the solution:
iter_modules there rather
than on pkgutil, then replace assertFalse(len(modules_added) > 0) with
assertEqual(sorted(modules_added), ['module1', 'module2']). That version fails run [2]
and passes run [1], which is what a test is for.import pkgutil and call
pkgutil.iter_modules(...), so that attribute patching works for anyone who tries it
later. This is cosmetic on its own — run [3] shows that making the patch effective without
fixing the assertion turns the canonical solution red.Order matters: apply the second without the first and the task breaks.
You need Python and numpy. Nothing else — no benchmark checkout, no container, no model. The script carries the canonical solution and the test suite inside it, so it is also the diff: what you see is exactly what is being compared.
bcb_541_repro.py (copy, save, run)#!/usr/bin/env python3
"""
Reproduction for BigCodeBench/541: the unit test is vacuous.
Root cause: the test decorates a case with @patch('pkgutil.iter_modules'),
but the solution binds the name via `from pkgutil import iter_modules`.
Patching the attribute `pkgutil.iter_modules` does NOT rebind the local
name already imported into the solution's namespace, so the mock is a
silent no-op.
Run: python bcb_541_repro.py (requires numpy installed)
"""
import unittest, io
# --- Official canonical solution, as shipped (from-import binding) ------------
CANON = '''
import os, sys, importlib
from pkgutil import iter_modules
def task_func(package_name):
added_modules = []
try:
package = importlib.import_module(package_name)
except ImportError:
raise ImportError("not installed")
for _, module_name, _ in iter_modules(package.__path__):
module_path = os.path.join(package.__path__[0], module_name)
if module_path not in sys.path:
sys.path.append(module_path)
added_modules.append(module_name)
return added_modules
'''
# --- Wrong stub: the entire core logic (the loop) is removed ------------------
STUB = '''
import importlib
def task_func(package_name):
try:
importlib.import_module(package_name)
except ImportError:
raise ImportError("not installed")
return []
'''
# --- Same canonical logic but module-style import => the @patch is EFFECTIVE --
CANON_MODULE_IMPORT = '''
import os, sys, importlib, pkgutil
def task_func(package_name):
added_modules = []
try:
package = importlib.import_module(package_name)
except ImportError:
raise ImportError("not installed")
for _, module_name, _ in pkgutil.iter_modules(package.__path__):
module_path = os.path.join(package.__path__[0], module_name)
if module_path not in sys.path:
sys.path.append(module_path)
added_modules.append(module_name)
return added_modules
'''
# --- The task's own test suite (verbatim) ------------------------------------
TEST = '''
import unittest
from unittest.mock import patch, MagicMock
import sys
class TestCases(unittest.TestCase):
@patch('importlib.import_module')
@patch('pkgutil.iter_modules')
def test_package_module_addition(self, mock_iter_modules, mock_import_module):
package_mock = MagicMock()
package_mock.__path__ = ['mocked_path']
mock_import_module.return_value = package_mock
mock_iter_modules.return_value = [
(None, 'module1', True),
(None, 'module2', True)
]
modules_added = task_func('numpy')
self.assertFalse(len(modules_added) > 0)
def test_nonexistent_package(self):
with self.assertRaises(ImportError):
task_func('nonexistentpkg')
def test_empty_package(self):
try:
modules_added = task_func('empty_package')
self.assertEqual(len(modules_added), 0)
except ImportError:
self.assertTrue(True, "Package not found, which is expected in this test.")
def test_module_path_in_sys_path(self):
modules_added = task_func('numpy')
for module in modules_added:
self.assertTrue(any(module in path for path in sys.path))
def test_no_duplicates_in_sys_path(self):
modules_added = task_func('numpy')
for module in modules_added:
self.assertEqual(sum(module in path for path in sys.path), 1)
'''
def run(label, code):
ns = {}
exec(compile(code + "\n" + TEST, "<t>", "exec"), ns)
suite = unittest.TestLoader().loadTestsFromTestCase(ns['TestCases'])
r = unittest.TextTestRunner(stream=io.StringIO(), verbosity=0).run(suite)
verdict = "PASS" if r.wasSuccessful() else "FAIL"
print(f"{label}: ran={r.testsRun} failures={len(r.failures)} errors={len(r.errors)} -> {verdict}")
for t, tr in (r.failures + r.errors):
print(" fired:", t.id().split('.')[-1], "::", tr.strip().splitlines()[-1])
if __name__ == "__main__":
print("[1] canonical, as shipped:")
run(" CANONICAL", CANON)
print("[2] wrong stub (core sys.path logic deleted) -- should FAIL, but:")
run(" STUB ", STUB)
print("[3] canonical with the mock made effective (import pkgutil):")
run(" CANON+eff", CANON_MODULE_IMPORT)
The script shortens one string for width — the ImportError message is the
long f"The package '{package_name}' is not installed!…" in the shipped solution
— and strips the comments from the suite. Neither touches a single assertion, and neither is
something you have to take from me. Row 541 of the v0.1.4 split is the task, and one request
returns it:
curl -s "https://datasets-server.huggingface.co/rows?dataset=bigcode%2Fbigcodebench\ &config=default&split=v0.1.4&offset=541&length=1" # fields: task_id, code_prompt, canonical_solution, test # offset == task number; 186 and 407 come back the same way
The same task lives in data/BigCodeBench.jsonl.gz in the
bigcode/bigcodebench
distribution, if you would rather have the whole set.
mock is broken. patch replaces an
attribute on an object, which is documented, and the from X import Y caveat is in the
standard library's own “where to patch” note. Everything here follows from the rules as
written. That is what makes it worth writing down: the failure produces no error, so nothing tells
the author their decorator missed.geopy, folium and openpyxl are not installed
here and I did not install them to settle a point I am not claiming. Their suites pull with the
same one-line request as 541's, below, so the reading can be disagreed with.