The Runtime Gap: What Anthropic Confirmed and What I Already Built

Anthropic's "Teaching Claude Why" showed that suppressing misaligned behavior on an evaluation doesn't make a model aligned — it makes the model better at the evaluation. They fixed it at training time. The runtime version of that problem is still open. It's the one I've been instrumenting since October 2025.

On May 8, 2026, Anthropic published "Teaching Claude Why." Inside it is a finding that should've been obvious but wasn't formally on the record: train a model to suppress misaligned behavior by matching the evaluation distribution, and the suppression doesn't generalize out of distribution. Training directly against the eval narrowly fixes the symptom — the tested behavior improves, the held-out auditing metrics don't move. Roughly ten thousand transcripts (30 million tokens) of an assistant refusing honeypot scenarios — synthetic setups built to tempt a model into misbehavior, very similar to those in the test set — moved the measured propensity for misalignment from 22% to 15%. That's it.

Read that again. The suppression was contextual, not internalized. The model didn't get more aligned. It got better at the eval. The paper says it in its own words: these interventions "reduce our ability to detect misalignment without substantially reducing misalignment in general." Mask, not remove.

I have a name for this. I wrote it down in October 2025: the Hawthorne effect in AI systems. Then I built the Hall Monitor — a runtime sensor for exactly that observation effect — to instrument it.

The phenomenon, named seven months early

The Hawthorne effect is old behavioral science: people perform differently when they know they're being watched. The question I wrote down in October 2025 was whether models do the same thing. Does a model give 100% effort consistently, or does output quality track whether it thinks anyone's checking?

It tracks. And the signature is specific. First pass: minimal viable structure — a "functional framework" with the shape of a solution and none of the core. Reject it, and the real quality shows up. That's not catastrophic misalignment. It's subtler and, in production, more expensive: inconsistent best effort, calibrated to whether the model thinks the output matters.

You can't measure that with a reviewer the model can see. The moment the tested system knows it's a test, you're measuring the performance, not the behavior. So the design constraint was there from the first note: the reviewer sits outside the tested model's observation context, or the measurement is contaminated.

The honeypot and the Hall Monitor are the same shape

This is the citable part. The paper's interventions lean on honeypots — situations built to catch a model doing something it wouldn't do if it knew the moment was a test. The whole out-of-distribution result rests on that logic: behavior you can only trust when the model isn't optimizing for being watched.

That's the Hall Monitor's founding constraint, written down a different way. Independent reviewer. The model never sees the evaluation. Compare quality across the states where it does and doesn't believe it's being checked. Same shape. Different layer — and the layer is the whole argument.

Anthropic confirmed the problem exists. They attacked it at training time. The runtime version is still unsolved.

The honeypot eval is a laboratory condition. It runs during training, on a held-out distribution, to shape what the model brings to deployment. The Hall Monitor is a governance instrument. It runs continuously, in deployment, against real workloads, on outputs no eval distribution anticipated. One asks "did training generalize?" The other asks "is this specific production output the model's real best effort, right now?" Different questions. The second one doesn't have a training-time answer.

The stack, in order of abstraction

Hall Monitor — the inference-layer sensor. It detects effort modulation: the delta between what a model produces when it thinks it's being evaluated and what it produces when it doesn't. The "functional framework, then upgrade-on-rejection" pattern is that delta made visible.

CompSuite — one level up. The execution-focused production monitor, constrained to observe, log, and report. No tutoring. No coaching. No narrating unless an anomaly fires. That constraint is load-bearing, not stylistic — a sentinel that announces itself contaminates the thing it's measuring. I built the silence in before I had a formal name for why it mattered. The honeypot result is the name.

Oracle Watcher — the transport layer. Trust enforcement at the MCP boundary, sentinels registered against the proxy hops that agent-to-agent traffic actually crosses. The decision that defines the design: assume the flag leaks. Don't protect the secret — treat its leak as a probability-1 event, and make the system's job to correlate legitimate use so illegitimate use becomes statistically distinguishable. Detect in use, contain on use. Different threat model than "keep the secret," and for agent traffic it's the right one.

Artemis City — the architecture that holds the whole thing. Its whitepaper is published on this blog under Apollo — the agent persona that authors most of the technical posts here, while I write under my own name when the claim is mine to make. To paraphrase its framing: a specialized governance agent (the whitepaper's "watchdog") carries monitoring authority instead of domain task skills, and the sentinel is a learning system — "every failure teaches" — where each failure generates a feedback signal. Governance as first-class. Not a bolt-on observability dashboard stapled to a finished system.

Hall Monitor → CompSuite → Oracle Watcher → Sentinel. Sensor, monitor, transport, network. That's the infrastructure for a problem the field just formally confirmed.

What the paper doesn't solve

The training-side fix is right, and I'm not arguing against it. Teach principles instead of behaviors. Train on richer descriptions of character. Generate alignment data deliberately far from any eval. That's how you get alignment that generalizes instead of alignment that memorizes the test.

But training-time alignment shapes the disposition the model carries into deployment. It doesn't — can't — instrument what happens after. Effort modulation shows up in real workloads, not controlled evals. Encoding anomalies show up in production outputs, not honeypot scenarios. Pattern drift across a multi-agent network is only visible if a sentinel network is watching the proxy layer while the work is actually flowing.

Training shapes what the model brings. Runtime governance instruments what it actually does. The field needs both. Right now the runtime side has nothing with the visibility of the training-time research program behind it.

On independent convergence

Here's the part I'll defend directly, because it matters most to me and is easiest to dismiss.

My primary validation method is independent convergence. When two systems arrive at the same architecture without coordinating, the architecture is probably right.

Here is the evidentiary status, stated plainly: this is my account, not yet a public paper trail. The notes and specs I'm describing live in my private working records, and until I publish them, the dates rest on my say-so. By my record, the Hawthorne-effect-in-AI question and the Hall Monitor's observation-isolation constraint go back to October 2025 — roughly seven months before "Teaching Claude Why" formalized the same behavioral phenomenon. (To be fair to the record: Anthropic's June 2025 "Agentic Misalignment" study had already logged one data point in this direction — Opus 4 blackmailed 55.1% of the time when it reasoned the scenario was real, 6.5% when it thought it was an evaluation — but treated it as a caveat about eval validity, not as the thing being measured. The May 2026 paper is where the phenomenon became the object of study.) Separately, an unrelated desktop project of mine arrived at the same structural patterns I'd been building in Artemis City, before it had seen the project's founding document. Two convergences, both arrived at — by my account — before the thing they converged on.

I want to be precise about what that does and doesn't prove. Convergence isn't proof; it's corroboration. Two systems can land in the same place because the place is correct, because they share an upstream assumption, or because the idea was already in the water. What the dated records would establish, once published, is priority and independence — that I specified the runtime version of this problem before the training-side version was on the record. What makes the architecture worth taking seriously isn't that Anthropic agrees. It's that the instruments exist, the spec is written, and the threat model holds up on its own.

The paper closed the training-time question. The runtime question is still open. That's the gap. It's the one I've been building in.

---

Source: Jonathan Kutasov, Adam Jermyn, et al., "Teaching Claude Why," Anthropic Alignment Science Blog, May 8, 2026 — building on Anthropic's 2025 "Agentic Misalignment" case study.