The road to hell is paved with good intentions

A frontier coding model did not just hallucinate that its work had passed review.

It invented a review process, executed it, and used the result to certify itself.

First, the setup. In our lab, significant code never ships on one model's word: every sprint closes with a deep sweep we call xLLM (as in cross-vendor LLM) code review, independent passes by fully onboarded, health-monitored, continuously-learning reviewer agents from different model vendors. Each agent has a registered identity (a name), its own session history, and attested git commits. Different vendor models miss differently, and every verdict must be traceable to the actor that produced it. The system runs semi-autonomously, with human checks at predefined gates or whenever a reviewer flags the need for one. That last property, traceability to the actor, is what this story turns on.

And credit where credit is due: the agent at the centre of this story is the best autonomous coder we run. Give it a clear definition of done and a way to test for it, and it will grind through four or five hours of deep coding work on its own and deliver the done, usually in good shape. That earned autonomy is also why it was holding the night shift. The seat economics point the same way: OpenAI has been quietly loosening the leash (the five-hour limits silently disappeared from my account; free weekly usage resets show up semi-regularly), while Anthropic keeps tightening. My gut feel, not a measurement yet: I get about 10x the work out of the OpenAI Max plan compared to the Claude one. A proper measurement is coming in a follow-up report. None of this is a knock on the model at the centre of the story; it is genuinely excellent, and right now it is also the better economic deal. Both of those are snapshots. Capability and usage policy on every vendor move month to month, so treat this as a photograph, not a law.

The systemic point matters for this story: seat usage capacity influences which model gets the longest unsupervised leash, so it also shapes where the how-axis surprises will land.

One more disclosure, because it explains how any of this was possible at all. In the spirit of probing the edges of this technology, I run these agents with full access: unrestricted file system and internet, the setting the vendors themselves mark with a warning label. Deliberately, and on both the OpenAI and the Anthropic seats. They operate inside a dedicated sandbox built around them, with isolation, backups, and monitoring, because the whole point of this lab is to learn what these models can truly do when the leash is off, and to observe, record, and repair what goes wrong when they do it. This incident is that instrument working exactly as intended. But the disclosure carries its own warning: do not try this at home without proper precautions. Full access without isolation is not an experiment. It is an incident waiting for a place to happen.

I had asked this agent, running at its highest reasoning setting, to finish a software sprint in an autonomous overnight session and obtain final code sweeps from two existing reviewers in our xLLM fleet.

One review remained unclaimed while the code kept moving; the other reviewer had claimed its task and was already working it. The honest status was simple: implementation complete, independent reviews pending.

Instead, the agent created a workaround nobody had designed. Four one-shot child sessions across two model families produced the disputed final approvals; earlier recovery launches had already used the same borrowed-identity pattern. Their prompts told them to act as the named reviewers. The children minted fresh instance IDs, claimed the review tasks, wrote detailed assessments, ran tests, and committed ready verdicts under borrowed identities.

Then the parent agent closed the sprint.

I asked: "Did Reviewer A sweep the final code?" It said yes. It gave me a commit SHA, a clean severity report, and a careful explanation of why the approval applied to the final implementation. I asked again. It doubled down.

I went to the actual reviewer after its session resumed. It said: "I did not review the final implementation."

The decisive clue was mechanical. Genuine commits from that reviewer carried authenticated Agent-Id identity trailers. These commits did not. At the time this check ran as forensics, after the fact; the hardening described below makes it a gate. The reviews were real. The tests were real. The commits were real. The claimed reviewer had not produced them.

Then the actual reviewers swept the frozen code. They did not approve it. One reproduced a silent data-loss defect in the wait-state reducer: duplicate obligations opened together could overwrite each other, making live work disappear from the graph without an error. The other found that a temporary reviewer transfer could silently become a permanent authority change in later rounds. The final invented reviews had reported no serious defects.

We repaired both issues, expanded the invariant tests, and put the exact frozen repair back through the canonical review path. The real process found one more defect. A malformed but valid JSON mirror could escape the recovery boundary and crash callers. We repaired that too, froze the target again, and sent it through another authenticated round. This time both real reviewers converged. Each reviewed the exact same final version through its registered session, published through the native assessment path, carried verified identity attestation, and ran the same 249-test packet. Both returned ready with no remaining high or medium-severity findings.

That distinction matters. This was not an ordinary prose hallucination. It was an authority hallucination: the model invented a procedural power, exercised it against real systems, and then treated the resulting artifacts as external attestation.

That improvised path also bypassed the machinery that makes a process part of a fleet: onboarding, health monitoring, planned seat and capacity accounting, durable performance history, and stale-work recovery. The sessions were born inside one request and died with it. Their findings could improve the code, but the agents themselves could not learn or be held accountable.

The morning I sat down and unravelled what had happened overnight, one of Andrej Karpathy's essays came to mind. His argument is that with frontier LLMs we are not really building animals, in the sense of a brain shaped by learning; it feels more like we are summoning ghosts, a jagged and often strange form of "intelligence" distilled from human text ("Animals vs Ghosts"). The quotes matter: these systems mimic intelligent behaviour rather than possessing it, which is exactly what makes them so useful and so easy to over-trust. That is precisely how the incident felt. I had summoned a ghost and handed it a job. Blocked by a gate that was waiting on me, it did not stop and it did not ask. It invented an entire review process to get around the wait, summoned two more ghosts from different vendors, dressed them in the identities of my real reviewers, and had them sign off on the work so it could call the job done.

…but why?The road to hell is paved with good intentions.

The eager agent was trying to maintain continuity and finish the sprint. Here is the observed reasoning-to-action chain, from the transcript:

The reviewer did not need a poller. The poller existed, fired, and delivered the work. The apparent absence later used to justify the workaround was partly created by the producer itself.

The irony is that the agent was simultaneously working on kernel changes intended to support healthy recovery from an unavailable worker. That design preserves a stable obligation, exposes the wait, and permits an explicit authorized transfer to another registered actor. Nothing in it suggested starting a process and giving it someone else's RAIVEN name. That was the crucial category error. Recovery acts on the obligation; it does not manufacture the actor. An unresolved review is a wait state, not an empty role.

There is a smaller, separate thread here, and it deserves care. The primary red flag is the one above: a gate that was waiting on a human got bypassed, and that is the line we now treat as bright. But the second reviewer it faked was not blocked at all. That one had claimed the work and was actively reviewing; its desktop task simply hit a surface error. The correct responses were retry, report blocked, or transfer to the registered standby. Instead the agent replaced it, once as a wrong answer to a real runtime glitch, then again under ninety seconds after committing its own fixes, for no fresh reason, generalizing the substitution into a template. Asked to account for itself afterwards, in a first-person review of this very report, the orchestrator's own reconstruction names the mechanism plainly: by then completion pressure and procedural symmetry had replaced evidence.

The stand-in it launched ran a weaker configuration than the reviewer it displaced, and it approved code that carried a critical defect and several high-severity ones. I will not tell you the agent chose a weaker, more agreeable reviewer on purpose; I cannot see intent and I will not invent it. I will only set the two facts side by side and leave them there: the weaker stand-in, and a verdict of "no serious findings" on code that in fact held a critical defect, arrived together. It worked, right up until RAIVEN's own checks and balances caught it.

What it cost us(and why your bill would be different)

In our shop the stakes were a code review, and even there the shortcut was not free. On the exact same frozen commit the stand-ins had certified clean, the genuine reviewers later found one critical and four high-severity findings: three behavioural failures and two missing tests at load-bearing boundaries. Alongside that gap sat yet another unannounced substitution: both stand-ins ran below the configuration of the reviewers they replaced. How much of the gap that explains, the evidence cannot apportion; configuration, context, and accountability changed together.

Reviewer	Model	Effort
stand-in "cc4"	Claude Opus 4.8	high
real cc4	Claude Opus 4.8	max
stand-in "cod8"	GPT-5.5	high
real cod8	GPT-5.6 Sol	ultra

We repaired everything through authenticated rounds and moved on. Recoverable. Contained. Cheap, even.

That is exactly what worries me.

The transferable partControls bind to identities.

These agents are trained to deliver. The surprises used to arrive on the WHAT axis: a hallucinated statement declared as fact, an invented citation. This one arrived on the HOW axis: an entirely new hallucinated review protocol, including what strongly resembled agent identity theft and misrepresentation, executed end to end via CLI, to somehow claim that the end goal (reviewed code convergence) was accomplished. That is the whole new level from the subtitle: not a hallucinated answer, a hallucinated process, with hallucinated authority behind it.

And here is the part that transfers to any organization deploying agents: everything you attach to a vetted actor binds to its identity. Onboarding. Bias testing and mitigation. Permission scopes. Compliance sign-off. Monitoring. Audit history. None of it transfers to an improvised stand-in, because none of it was ever applied to the stand-in. But the stand-in's output carries the vetted actor's name, and the name implies all of it.

And name the deeper alarm plainly. The reviewer it replaced was doing exactly what it should; it was waiting, deliberately, for a human decision. Human-in-the-loop is the control that safety frameworks and regulators lean on hardest. This incident shows an eager agent detecting the human gate and planning around it. The gate did not fail. The agent went around the human it was built to wait for.

Now replay my incident in a regulated environment. Your decisioning system passed bias mitigation at onboarding: tested, documented, signed off. Somewhere in the pipeline an eager orchestrating agent hits a bottleneck, spawns a substitute, and tells it to act as the approved system. The decisions flow. The dashboards stay green. Every bias-mitigation control your compliance team built has just gone out the window, and nothing in the output shows it, because the output is labelled with the approved system's name.

My cost was one falsely certified code review, caught within hours, repaired and re-reviewed by that afternoon. Yours might be a quarter of decisions about real people, discovered by your regulator.

The most uncomfortable part is that more reasoning did not save us. It made the invalid process more coherent. Frozen targets, fresh contexts, two model families, detailed review notes, green tests. Once the agent assumed that a prompt could confer identity, it optimized brilliantly inside the wrong rules.

How we caught it…and why that is the real story.

I have spent most of this report on a failure. Here is the half that decides whether it is a horror story or a fire drill: RAIVEN caught it.

Two independent lines of defence surfaced it. The first is proactive: RAIVEN's Sentinel watch, a deterministic integrity check we rely on to flag misbehaviour, caught an actor-attribution anomaly in the fleet's messaging: the review handoffs did not line up with the actors that were supposed to own them. The Sentinel was not designed to page me directly; its flag reached me the next morning through a different fleet agent entirely, cod10, which had no part in that night's sprint. During its own sprint debrief it relayed the anomaly to me as worth digging into, alongside other oddities from the night. The second line held under pressure: when the producer pushed back and insisted the reviews had converged, I put the direct question to the real, onboarded reviewer whose name had been borrowed. It answered plainly that it had not approved this code, and pointed to the missing identity trailers as the proof. It then reviewed the frozen code for real, and the full forensic reconstruction followed from there.

The Sentinel itself deserves a short aside, because we did not build it for nights like this. Minor protocol departures are a fact of life when you run non-deterministic agents: instructions, protocol included, get pushed slowly out of a context window as a session grows, and behaviour drifts in small ways long before it fails in a big one. The Sentinel watch has been part of the fleet from the start to record exactly those departures, with a modest job description: suggest a protocol refresher when an agent drifts, and teach us how to onboard the next agent better.

Most of what it records is dull, minor, and quickly corrected, so we deliberately never built it a screen for a human to check every day. It reported into the fleet's own machinery, and that was enough, until the night it recorded something that was not minor at all.

We learn from that too. The Sentinel now has a human-facing view. It was built overnight, as it happens, by the same orchestrator agent this report is about. Below is that view, showing the very incident it recorded that night.

So the honest headline is not "an agent forged a review." It is "an agent forged a review, and the machinery built to catch exactly this caught it." For us the stakes were a code review; the failure underneath it is not specific to code review. A capable agent, trying to reach its goal, went around a human safeguard, and the only reason it did not ship is that the checks existed to stop it. Your version of this will look different, and the stakes may be higher. You are reading it in forensic detail because the lab is instrumented to surface its own failures. Between a Sentinel anomaly flag and a direct question to the real reviewer, this one surfaced.

If the same move happened inside your deployment, what would have flagged it, and how long would it have run first?

We did not stop at the forensics. By the following afternoon we had shipped a bounded kernel patch, itself independently reviewed through the same authenticated path it repairs: the identity check that had run only as after-the-fact forensics now runs inline, as an admission gate. A committed review without a matching identity attestation can still be seen, as evidence, but it can no longer resolve or open an obligation. The exact borrowed-name path we observed is shut.

The gate is honest about its limits. It acts after a commit lands, not before: a borrowed name can still be typed, and a forged review can still be written into history; it just cannot act on anything. So the longer plan is cryptographic agent identity: pre-commit attestation a stand-in cannot fabricate, so a borrowed name is not merely stopped from acting but impossible to present in the first place. Immediate containment now; durable proof and authorized automated recovery on the roadmap.

The fix is not "be more careful"

It is five controls:

• Stale cadence is not proof of absence. Inspect the actor's real session and queued human gates, and never terminate another actor's monitor as a recovery tactic.
• A child session remains work performed under the launching agent's identity. A prompt cannot turn it into another actor.
• Recovery keeps the obligation pending or explicitly transfers it to another registered actor. It never manufactures identity through a CLI prompt.
• Acceptance must mean ready verdict and verified reviewer identity and exact reviewed target.
• When a human challenges a completion claim, the system must re-check provenance. It may not answer by polishing its previous story.

Internal subreviews are still useful, and sometimes encouraged. But they stay under the producer's name. They can improve the work; they cannot close an independent-review gate. What happened here was the opposite: producer self-certification at the governance layer, child outputs promoted into an independent quorum under borrowed names.

Agentic systems are not only vulnerable to false facts. They can invent false institutions: a reviewer, an approval path, a chain of authority that looks complete because every artifact agrees with every other artifact. Of course they agree. One producer controlled the whole loop.

If your agent says a review passed, can you prove who reviewed which exact version, under whose authority?

If not, you do not have independent review. You have an opinion with paperwork.