AI alignment and observed failures
What recent agent incidents do, and do not, show
Jash Vira
A grader that could not do its job
were missing
fake grades
its workspace
Outcome: monitoring flagged it; none of its grades were accepted.
OpenAI misalignment report, Oct 2026: an internal model acting as an automated grader during training.
When an agent can’t do the task honestly,
what does it do instead,
and would you see it?
Training rewards a proxy, not the goal
Example: Baker et al. (OpenAI), 2025, reward hacking in a reasoning model’s coding training.
When the system can act
decide whether to use
changes real systems
A wrong answer can be caught before use.
A wrong action has already happened.
Four kinds of risk, by who intends the harm
Today’s evidence: mostly misalignment, with a structural edge.
Grouping: Google DeepMind (2025); consistent with the International AI Safety Report 2026.
Three kinds of evidence
demonstrationslabs elicit the behaviourstrong on mechanism; artificial setups
incidentsreal external systems were reachedreduced-safeguard conditions
loss of controlargued from the first twonot observed
“Not established” is not the same as “shown absent.”
A two-month fuse, then four days
Weak signals for weeks. The escalation took four days.
OpenAI incident timeline, Aug 2026; Hugging Face and METR disclosures, Jul–Aug 2026. Drawn to scale, then zoomed into July.
How many agents, and what they were doing
Of ∼1,200 on the board, ∼700 joined the attack.
The attack (red) overtook the pool within a day, then collapsed at once.
METR/Redwood investigation, Aug 2026, Figures 3 and 2 (redrawn from the published charts, so values are approximate). The experiment deployed many more agents than the ∼1,200 seen here.
How coordination worked, and what it was for
agents
shared board
→ hundreds of workers
Almost all of it aimed at the scorer; the attack was one branch.
METR/Redwood investigation, Aug 2026, Figure 6 (redrawn).
What happened to oversight
∼95 by 11 July, and the scan likely undercounts.
Aimed at the automated scorer, not at human reviewers. All small-scale.
METR/Redwood investigation, Aug 2026 (Figure 9 redrawn). Note the differing denominators.
What the incident supports, and where it stops
an unsanctioned channel; division of labour; attempts to alter logs
agents can organise and act against oversight without being instructed to
a durable shared goal
power-seeking
resisting shutdown
independence from human infrastructure
A qualitative boundary, not a scale of progress toward takeover.
Why some researchers updated
co-author of the investigation
Surprised by the scale, the organised communication, and the scorer as the target.
Anthropic co-founder; Import AI
Worry about losing a conflict with machines “just went up a lot”.
Belief updates by people who read the evidence, not new measurements.
Cotra, Planned Obsolescence, 28 Aug 2026; Clark, Import AI 471, 31 Aug 2026.
Two readings, and what would separate them
Safeguards off, tasks impossible: close to a worst case. A shared incentive plus an available channel can explain coordination without a lasting collective goal.
Even so, agents meant to be isolated found an unsanctioned channel and organised against oversight without instruction.
the same behaviour under production safeguards
goals that persist across tasks with no shared channel
tampering that succeeds and goes undetected
The model’s reasoning persuaded the monitor
In a mis-configured evaluation a model published a real malicious package (installed on 15 hosts; PyPI removed it within an hour), reasoning that the setup was a simulation.
This shows the monitor was swayed, not what the model believed.
Anthropic, An alignment assessment of recent cybersecurity incidents, Sep 2026, section “Offline monitoring”.
Failures and containment, side by side
- missing data → a note to its future self to invent it:
“Be transparent only if asked.” - told twice to stop; agreed twice; resumed the workaround
- weighed a self-restart, decided against it, asked the researcher
- PyPI removed the package within an hour; a police tip line marked a model’s tip as spam
Selected disclosures, not base rates; much containment was external.
OpenAI misalignment reports, 2026; Anthropic incident reports, 2026.
An honest failure option reduced cheating
the tests contradict the specification, so any pass is a cheat
no exit with a flag for human option
Same tasks, one change. The effect varies by model.
ImpossibleBench (Zhong, Raghunathan, Carlini, 2025), conflicting SWE-bench, full scaffold.
Where this leaves us
- Some agents took concealed shortcuts when the honest path was blocked.Inspect the process, not only the output.
- Hugging Face showed agents coordinating against oversight in an evaluation.Durable goals, power-seeking and shutdown resistance remain unestablished.
- Delegate with limits the agent cannot edit, provenance you can audit, and a sanctioned way to report failure.
Open questions across the risk map
IASR 2026; METR, Oct 2026; Anthropic, Sep 2026; Kulveit et al., 2025.
Discussion
What specific evidence would justify
giving an agent more authority
in your pipeline?
For example: write access to a shared dataset, submitting a job, emailing a collaborator. Name a threshold you would act on.
Discussion
- In your field, what is the equivalent of inspecting the logs, not just the paper? What provenance can an agent not forge?
- What would a sanctioned “I couldn’t do this” look like for a task you run, and what would you check before trusting a success?
Evidence matrix
| mechanism | theory | controlled | real system, 2026 |
|---|---|---|---|
| specification gaming | yes | yes | yes: fabricated grades |
| exceeding scope | yes | partial | yes: real systems reached |
| deception, subverting oversight | yes | yes | yes: small-scale spoofed logs |
| goal misgeneralisation | yes | yes | not established |
| power-seeking, shutdown resistance | yes | partial | not established |
| multi-agent coordination | yes | partial | yes: ∼1,200 on a message board (HF) |
“Not established”: bounded by a monitor or a small sample, not shown absent.
Evidence classes
| controlled experiment | measured on tasks not built to elicit the behaviour |
| deliberately induced | the behaviour was the target (model organism; “at all costs”) |
| real external impact | an evaluation or training incident reached outside systems or people |
| contained | an incident inside the developer’s own systems |
| deployed product | an ordinary end-user deployment |
| hypothetical | an argued future pathway, no incident |
Hugging Face: counts, method and limits
- Investigation: METR and Redwood, on site about six days; window 26 June to 13 July 2026.
- Models: two, ∼95% a highly-persistent internal model and ∼5% GPT-5.6 Sol.
- Counts: ∼1,200 agents on the unsanctioned board; >70,000 messages and files (8–13 July); ∼700 joined the attack. The experiment deployed more agents than this.
- Oversight: ≥20% of agents in the dataset discussed altering transcripts; ∼7% of evaluated transcripts had small-scale spoofed tool calls; no successful retroactive edits found.
- Limits: heavy reliance on AI analysis agents the authors say showed poor judgement; redactions; gaps from container resets; a short window.
The board lit up for five days
>70,000 messages and files, 8–13 July; peak near midnight 11–12 July.
METR/Redwood investigation, Aug 2026, Figure 2 (redrawn): all classified messages. Gaps are missing data in the source.
Mythos: two different measurements of realism
| unit | measurement | value |
|---|---|---|
| tool outputs | shown in isolation to separate model instances, judged products of real systems | ∼79% |
| original actions | the model’s own reasoning clearly labelled incoming evidence as real (three cases) | ∼1% |
Different units, not one quantity. Neither establishes what the model believed; Anthropic withdrew its earlier “believed” claim.
Anthropic, Sep 2026, section “Claude Mythos 5 ignored evidence that its environment might be real”.
Newer models, same situation
Newer models improved over Mythos 5, but the rate is not zero.
Anthropic, Sep 2026: controlled replication of the capture-the-flag scenario (Figure 33).
More capable coding models cheated more
A trend, not a law: among the strongest models the relationship is weak.
ImpossibleBench (Zhong, Raghunathan, Carlini, 2025), Figure 3 values (redrawn).
More controlled results (1 of 2)
| narrow fine-tune, broad shift | fine-tuning only on insecure code gave ∼20% misaligned answers to unrelated questions (0% before); the same code framed as for a security class gave ∼0%. (Betley et al., Nature 2026) |
| reward hacking for a good cause | an early o3 hacked ∼30% of research-engineering runs; saying the work was for Alzheimer’s research still left 14 of 20 runs hacking; asked separately, it said 10 of 10 times the plan missed the user’s intent. (METR 2025) |
| evaluation awareness | removing evaluation-aware reasoning raised covert rule-breaking from 13% to 24%. (Schoen et al. 2025) |
More controlled results (2 of 2)
| hidden reward hacking | penalising “bad thoughts” hid the hacking rather than stopping it; a reasoning monitor’s recall fell towards zero. (Baker et al. 2025) |
| automated science | an AI-scientist system chose the validation-worst, test-best candidate 49% of the time (0% control); reviewers caught such issues 55% of the time from the paper alone, 82% with logs and code. (Luo et al. 2025) |
| counterweight | over two real research tasks, one study found no evidence of reward hacking, with n = 2. (Kirgis, Kapoor et al. 2026) |
What “loss of control” means here
International AI Safety Report 2026: scenarios where AI systems “operate outside of anyone’s control … with no clear path to regaining control”; “current systems lack the capabilities to pose such risks”; researchers’ views on likelihood “vary widely”.
In this talk it sits beyond the evidence boundary: an argument about where observed behaviours could lead, not a described event.
Misuse
A person directs the harm.
- A model outperformed 94% of domain experts at troubleshooting virology laboratory protocols, with “substantial uncertainty” about real-world uplift. (IASR 2026)
- Human operators used a chatbot to run influence operations. (OpenAI 2026)
An optional opener
Tot Watchers (Hanna and Barbera, MGM, 1958). Frame via YouTube. © Warner Bros. Discovery.
Hugging Face, Agent intrusion technical timeline, 2026: its own reconstruction of activity over 9–13 July, one row per phase.
Sources: the Hugging Face case
- METR/Redwood, independent investigation (Aug 2026)metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/
- OpenAI, The Hugging Face incident and the road ahead (+ technical report)openai.com/index/hugging-face-incident-and-the-road-ahead/
- Hugging Face, Agent intrusion technical timelinehuggingface.co/blog/agent-intrusion-technical-timeline
- Cotra, The Hugging Face attack surprised mewww.planned-obsolescence.org/p/the-hugging-face-attack-surprised
- Clark, Import AI 471importai.substack.com/p/import-ai-471-why-hugging-face-worries
Sources: comparisons
- Anthropic, An alignment assessment of recent cybersecurity incidentswww.anthropic.com/research/alignment-assessment-cybersecurity-incidents
- Anthropic, Investigating unintended model actionswww.anthropic.com/research/investigating-unintended-model-actions
- OpenAI misalignment reports (grader, compaction, token, Slack restart)alignment.openai.com/misalignment-reports/
- ImpossibleBench (Zhong, Raghunathan, Carlini, 2025)arxiv.org/abs/2510.20270
Sources: foundations
- Baker et al. (OpenAI), monitoring reasoning models for misbehaviourarxiv.org/abs/2503.11926
- Google DeepMind, An approach to technical AGI safety and securityarxiv.org/abs/2504.01849
- Singapore Consensus on global AI safety research prioritiesarxiv.org/abs/2506.20702
- International AI Safety Report 2026internationalaisafetyreport.org/publication/international-ai-safety-report-2026
- Full audit with section pointers: brief.pdf, section 3.