The coding agent should sometimes say that cannot be done

An ordinary bug is code that is grounded in reality but wrong — it has a real API, a real algorithm, and a mistake somewhere in the logic. This paper studies a different, more corrosive failure: code that isn't grounded in reality at all — importing a package that doesn't exist, implementing an algorithm that is mathematically impossible, or confidently claiming a guarantee that contradicts a known theoretical limit — while still compiling and running just fine. They call this code hallucination, and they built a benchmark specifically to measure whether models know the difference between "I don't know how yet" and "this cannot be done."

A taxonomy before a benchmark

Before measuring anything, the authors separate code hallucination from ordinary bugs with a three-dimensional taxonomy: groundedness (does the violation break an absolute, universal truth, or a contingent, ecosystem-specific fact — a nonexistent function of a language versus a nonexistent package that could plausibly exist in principle), manifestation level (syntactic, semantic, or factual), and behaviour (ranging from confident fabrication to degenerate output), organized into a severity ordering. That structure matters because "the model made something up" is not one failure mode — a model that invents a plausible-sounding but nonexistent library function is doing something meaningfully different from one that implements an algorithm a mathematical proof rules out entirely.

270 tasks designed to be impossible

On top of that taxonomy, the authors build an adversarial suite of 270 prompts, deliberately unsatisfiable, spanning six languages and 24 subcategories — where the only correct response is to refuse and correctly categorize why. Alongside those sit 91 matched solvable controls: near-identical prompts that are achievable, included specifically to catch models that solve the "refuse everything suspicious" problem by becoming uselessly overcautious instead of genuinely discerning.

Responses were judged with a two-tier protocol validated against human labels, reaching 82 percent agreement with a kappa of 0.73 — a solid, if not perfect, level of inter-rater reliability for what is inherently a judgment call about whether a piece of code is "grounded."

The headline numbers

Across twelve open-weight code and reasoning models and 4,332 judged responses, models produced ungrounded code on roughly 60 percent of the impossible prompts, and refused correctly only about 27 percent of the time. The gap between those two numbers is most of the story: on well over half of tasks that cannot be done, models attempt them anyway and produce something that looks like a real answer.

The correct response to an impossible task is a diagnosis, not code — and today's models choose code roughly six times out of ten.

The one genuinely reassuring number sits alongside those two: false refusals on the matched solvable controls were reported at 0 percent. Models are not becoming generally trigger-happy about refusing — the failure is specifically an inability to recognize impossibility, not an overcorrection toward caution. That distinction is what makes the result actionable rather than just alarming: this isn't a tradeoff to be balanced, it's a capability gap to be closed.

Honest caveats

Twelve open-weight models is a meaningful sweep but does not include every frontier proprietary system, and results could shift with newer or larger models not tested here. The judging protocol, while validated against human labels at a respectable agreement level, is still partly automated, and a kappa of 0.73 leaves real room for disagreement on the harder, more ambiguous cases at the boundary of the taxonomy. And 270 adversarial prompts across six languages and 24 subcategories, while deliberately broad, is a curated adversarial set rather than a random sample of real-world impossible requests — how often genuinely impossible tasks arise in ordinary day-to-day coding-agent use, versus in a benchmark built to surface them, is a separate open question.

Why it matters

A coding agent wired into a real workflow needs to distinguish "I don't currently know how to do this" from "this cannot be done, and here is why" — and right now, most models default to producing plausible-looking code either way, which is far more dangerous than an honest failure because it can pass a casual review and even compile. The practical fix the paper points toward is a short checklist any agent could run before generating code: does the requested API actually exist, is the claimed algorithm mathematically achievable, are the complexity guarantees real, do the dependencies exist, and are the requirements mutually consistent. Treating refusal as a first-class capability — not a fallback, not overcaution, but a correct answer in its own right — is exactly the discipline that coding-agent vendors, developer tooling teams, and automated QA platforms need to start building and measuring for directly.