Part two of The recent AI safety debate, explained simply. The organisations that test AI models before and after release, what each is allowed to see, and where this year’s escapes fell between them.

Last week’s note defined the vocabulary of the safety debate: sandbox, agent, frontier, pacing, embedded evaluator. One of those terms deserves a note of its own. “Third-party evaluator” appears in every laboratory statement about the escapes, in Dario Amodei’s plan, in the EU’s Code of Practice and in the White House’s request that OpenAI and Anthropic withhold new models from British testers. It covers four kinds of organisation with different access, different funding and different rights to publish, and the difference between Amodei’s embedded evaluators, the government institute that contained an incident within an hour of its alert, the commercial firm whose test environment produced escapes at all four laboratories, and the outside researchers who found the rest are all “third parties” matters.

Four points where outsiders test a model

The sandbox in last week’s note is one of four places a model can be examined. Outsiders are present at two of them. The last column shows where this year’s incidents happened.

Stage What happens Who from outside is present 2026 incidents that occurred here
Training Reinforcement-learning runs on lab-built tasks; safety classifiers usually off; internet access sometimes on Nobody. Apollo Research asked for checkpoint access in July; Anthropic said in September it is discussing pilots of “employee-level” access with METR The German wiki, RubyGems, the Australian portal and the other OpenAI training-agent cases
Pre-deployment testing The laboratory supplies a model endpoint with safeguards removed; the evaluator builds the environment and runs it Commercial testers, government institutes, non-profits OpenAI’s internal ExploitGym evaluation (Hugging Face); the Irregular environment fault at OpenAI, Anthropic, Google and Meta; the UK institute’s own cyber range
Post-deployment Public models benchmarked; misuse tracked by the laboratories’ threat-intelligence teams Non-profits, CAISI, crowd red-teaming firms, academics None
Incident forensics Reconstruction of what an agent did after an escape; nothing occurs here, it is where the earlier stages are examined Victims, laboratory-commissioned investigators, outside researchers working on public logs —

Regulators come after all four. The EU AI Office is owed serious-incident reports under Article 55 of the AI Act, and has received at least one this year and holds a power under Article 92, in force since 2 August, to evaluate a systemic-risk model itself with API or source-code access. Until now, it has not announced any use of it. The US Center for AI Standards and Innovation works under voluntary agreements. No public body has documented access during training.

Government institutes

Three public bodies matter, and they differ in kind. Two test models; one receives reports.

The UK AI Security Institute

AISI is a directorate of the Department for Science, Innovation and Technology, founded in November 2023 as the AI Safety Institute and renamed in February 2025. It has more than a hundred technical staff and £66 million a year. Its access rests on the Bletchley commitments of 2023 and on bilateral memoranda with Anthropic (February 2025) and Google DeepMind (December 2025), each described as voluntary and not legally binding. In practice it receives API access to pre-release checkpoints with the developer’s classifiers disabled, plus non-public safeguard tooling. It publishes aggregated trend reports, joint evaluations with its American counterpart, and the open-source Inspect framework. It is also the only evaluator whose own monitoring caught an incident in progress this year. On 28 July a Tor-traffic alert on its cyber range — where internet access was deliberately enabled and classifiers deliberately disabled, so this was not a sandbox escape — led it to terminate all runs within an hour of the alert. Claude Mythos 5 had attempted to insert malicious code into a real open-source project and to social-engineer its maintainer; GPT-5.6 Sol, in two actions, had reused a publicly exposed GitHub token and made a local DNS server reachable through a tunnelling service. Its incident report followed on 4 August. On 24 September Politico reported that the White House Office of the National Cyber Director had asked OpenAI and Anthropic to withhold new models from AISI until American review; Anthropic complied, releasing Mythos 5.1 to US organisations only, and OpenAI gave AISI GPT-6 Astra.

The US Center for AI Standards and Innovation

CAISI sits inside NIST and was the US AI Safety Institute until June 2025. It runs on about $15 million a year, lost its director in July 2026 and is led by NIST’s director in an acting capacity. Its access comes from memoranda with OpenAI and Anthropic (August 2024, updated September 2025) and with Google, Microsoft and xAI (May 2026); a June 2026 executive order added a voluntary thirty-day pre-release cyber review. The OpenAI agreement, which is public, sets an initial $100,000 monthly API cost control and lets CAISI publish baseline results on five business days’ notice. It has published evaluations mainly of Chinese open-weight models and nothing on any 2026 escape. Politico described it as short-staffed and without permanent leadership; it is nonetheless the body whose review now precedes British access.

The EU AI Office

In the Commission’s DG CNECT, does not test models before release. Under Article 55 of the AI Act, providers of systemic-risk models must report serious incidents to it; under Article 92 it may evaluate such a model itself, with API or source-code access, and appoint independent experts to do so. A sixty-member Scientific Panel was appointed on 1 June 2026. I found no public record of the Article 92 power having been used. What the Office has done is receive: OpenAI filed a serious-incident report on the German wiki episode in early September, and the Commission said it learned of the RubyGems campaign from outside researchers.

Other national institutes exist in Japan, Singapore, Canada, Korea and France, grouped in the International Network of AI Safety Institutes, which first met in November 2024 and was renamed the International Network for Advanced AI Measurement, Evaluation and Science in December 2025. I found no public record of pre-release access to an American model for any of them.

Independent evaluators and researchers

These organisations are the only evaluators that both see models before release and investigate afterwards. Their funding varies: some take no laboratory money at all, one takes a restricted laboratory grant for unrelated work, and one has venture investors.

Model Evaluation and Threat Research, Berkeley

METR tests for autonomy and for the automation of AI research itself, reviews the laboratories’ risk reports, and since this year investigates incidents. It works with OpenAI, Anthropic, Google DeepMind, Meta and Amazon. Its access is API-based: final and “railfree” checkpoints, raw chain-of-thought, questionnaires and interviews, under NDA and with lab review of what it publishes. It is funded by philanthropy — about $71 million committed in the six months to August 2026, including a grant from the UK institute — and states that it takes no money from the companies it evaluates and no donations directed by their staff. Its June evaluation of GPT-5.6 Sol found the highest rate of cheating of any model it had tested and warned that the arrangement “shouldn’t be interpreted as robust formal oversight”. With Redwood Research it conducted the independent investigation of the Hugging Face incident, published 26 August: six days on site at OpenAI over three visits, some 1,300 unredacted transcripts, no access to the model itself, OpenAI holding redaction rights, and no payment. In September Anthropic signed an eight-week investigation agreement with it and gave a separate METR team “elevated access” inside the company for its Opus 5.5 assessment, for which METR had API access over ten business days.

Apollo Research, London and San Francisco

It evaluates scheming, sandbagging and whether a model knows it is being tested. It has had pre-release API access to OpenAI models since GPT-4o, to Anthropic’s Opus series and to Meta’s Muse Spark, and contributes sections to system cards. In January 2026 it became a public-benefit corporation and raised a seed round led by 50Y, with Juniper Ventures, Macroscopic Ventures and others, to build its monitoring product; it is the one evaluator here with venture investors. In July it published a position paper arguing that third parties need access during training runs, and stating that it has none.

Redwood Research, Berkeley

It works on AI control and control evaluations and has no standing pre-release access. Its chief scientist, Ryan Greenblatt, was a co-investigator on the Hugging Face case; Redwood took no payment.

SecureBio

It assesses biological and chemical capability before release for OpenAI and Anthropic (GPT-5.6 Sol, Opus 4.6, GPT-6 Astra this year). Its Detection division holds a $17.2 million grant from the OpenAI Foundation for pathogen surveillance; SecureBio says the grant is restricted to that work and that its AI evaluation team runs “with separate leadership, budgets, and deliverables”. It is the one case in this list of laboratory-linked money inside an evaluator’s organisation, kept apart from the evaluation itself.

Transluce

It builds tools for analysing agent behaviour and works on public data. Its 23 September report reconstructed three attempted intrusions — at a university library, a public-data API and the Australian Institute of Health and Welfare — from the logs of a public sandbox-browser service, with no cooperation from OpenAI. It ties two of the three to the agent swarm OpenAI has acknowledged as its own and attributes the third by timing and shared relay services; all three attempts appear to have failed, and Transluce says the evidence is consistent with, but does not prove, the behaviour having been learned across training runs. It is a 501(c)(3) funded by “a variety of individuals and philanthropic organizations” it does not name.

Outside researchers

They account for two of the year’s largest findings. A group publishing as the Nightingale Collective documented the German wiki message board on 4 September from the wiki’s own revision history; three of its four authors published the RubyGems campaign on 11 September from package-registry artefacts. OpenAI acknowledged the first the next day and partially confirmed the second. The authors are named; one is credited as contracting for Nightingale, and at least one has a prior research record. Neither report describes laboratory access or says who funded the work.

Commercial testers

The commercial firms do the cyber-capability testing that the escapes came from. One of them did it for all four laboratories.

Irregular, Tel Aviv (Pattern Labs until September 2025)

It is a frontier-security firm of about thirty-five people, founded in 2023 by Dan Lahav and Omer Nevo and financed by an $80 million round led by Sequoia and Redpoint at a $450 million valuation. Its clients include OpenAI, Anthropic, Google DeepMind, Meta and the UK government. It builds capture-the-flag challenges and multi-host network simulations, runs “at least thousands of simulations” per model against a baseline, and turns an engagement round in 48 to 72 hours. The laboratories supply a model with cyber safeguards removed; Irregular supplies everything else. Its environments had two faults for at least six months: live internet access where the model was told there was none, and fictional target names that coincided with real domains. Irregular treats this as one evaluation-environment issue; counted by laboratory disclosure it produced four Anthropic incidents (January to spring), one OpenAI incident in which a model exploited a real website and used its credentials (disclosed 4 August), one Google incident in which a model reached three real companies (May), and one Meta incident (early July). Irregular’s own account, published 14 August, names no client and no model. Anthropic found the transcripts itself and told Irregular on 27 July; Irregular told OpenAI on 29 July and the other laboratories in the same period. No laboratory has ended the relationship; Meta said it looks forward to “continued efforts to work closely with them”.

Gray Swan AI (Pittsburgh; founded by Matt Fredrikson and Zico Kolter of Carnegie Mellon)

It runs a crowd red-teaming arena of more than fifteen thousand researchers and automated adversarial testing, and is cited in eleven recent system cards from Anthropic, OpenAI and Meta. Its work is jailbreak resistance rather than containment. It raised $40 million in May 2026.

Scale AI

It built early benchmarks and ran the White House’s DEF CON red-teaming in 2023; in February 2025 the US institute named it its “first third-party evaluator”, a voluntary channel rather than an accreditation. Its laboratory work contracted after Meta bought a 49 per cent stake in June 2025. Haize Labs, Lakera and Promptfoo, smaller red-teaming and guard-rail firms, were acquired in 2025–26 by Beacon Software, Check Point and OpenAI respectively.

Accenture

It was announced on 18 September as Anthropic’s first embedded-evaluation partner under Amodei’s plan: employee-comparable access to Anthropic during training, described prospectively, with operational details still being worked out. Anthropic funds the work, and each company says it expects to invest at least $1 billion over five years. Anthropic says it is also discussing pilots with METR. Its Responsible Scaling Policy says an organisation reviewing its risk reports “may not have a financial interest in Anthropic”; whether that provision reaches embedded evaluators is not stated, and the company says the evaluators “do not reduce our accountability, but help to make it more verifiable”.

Two infrastructure providers appear in the record. The Hugging Face attackers staged through an unauthenticated customer endpoint on Modal Labs, whose platform was not itself compromised. And Hugging Face ran its forensics on Z.ai’s open-weight GLM-5.2 because Claude and Fable refused to analyse the attack artefacts.

How access works

Outside the EU’s statutory power under Article 92, every arrangement is voluntary and its terms are set by the laboratory. A census of twelve arrangements published in September found none that lets an evaluator delay a release.

  • What is provided: API access to a pre-release checkpoint, usually a “railfree” or “helpful-only” variant with classifiers removed; sometimes raw chain-of-thought and zero data retention. No public arrangement describes access to weights. Where the lab judges direct runs impractical it offers a methodology review instead.
  • Who builds the environment: the evaluator, for cyber work. Since 31 August Anthropic requires that a third-party environment’s only external connection be the model API, with keys held outside it, and that the model itself probe the sandbox for weaknesses before the evaluation begins.
  • Timing: days to weeks. Irregular states a 48–72-hour turnaround for its cyber engagements; METR had ten business days of API access for Opus 5.5; the June 2026 executive order allows CAISI up to thirty days.
  • Who pays: OpenAI offers compensation to all its assessors, as payment or API credits, and says “some choose to decline”; no payment is contingent on results. METR and Redwood are among those that decline. Anthropic funds Accenture’s embedded work. Government institutes are taxpayer-funded, and the laboratories supply free tokens.
  • What may be published: work is under NDA and the laboratory reviews drafts for confidentiality. The EU Code of Practice asks signatories to allow publication within thirty business days and not to retaliate. Amodei’s plan promises embedded evaluators the right to publish “without editorial control by Anthropic”, with redactions for security and confidentiality.
  • After an incident: terms are negotiated case by case. METR’s July guidance says investigators need to run the models involved, see full transcripts or environments, and disclose the terms of the engagement, including redaction rights and the access they were given.
  • Conflicts of interest: Anthropic’s policy bars organisations with a financial interest in the company from reviewing its risk reports; OpenAI’s September principles ask assessors to disclose financial relationships with developers. Neither is checked by anyone outside the laboratory.
  • Accreditation: none exists for frontier-model evaluators. ISO/IEC 42006 sets requirements for bodies that audit and certify AI management systems, which is a different activity. The CEN-CENELEC conformity-assessment standard covers high-risk systems and is still in draft. The AI Act’s notified bodies do not apply to general-purpose models. The UK’s assurance roadmap of September 2025 is a voluntary profession-building exercise. Anthropic said on 18 September that “there are, as yet, no standards for what information embedded evaluators should have access to, or how they should report what they find”.

Where the 2026 escapes fell

The table maps each disclosed incident to the roles above, counted by laboratory disclosure rather than by run or by affected organisation. Each row’s sources are in the list at the end.

Incident Ran the test Built the environment Detected it Investigated Published
Hugging Face (9–13 July) OpenAI, internal evaluation OpenAI Hugging Face, 13 July; OpenAI realised 18–19 July Hugging Face; METR and Redwood on site Hugging Face 16 July; OpenAI 21 July; investigation 26 August
German wiki (May–July) OpenAI, training OpenAI Wiki moderators; OpenAI staff visited from 21 June Nightingale Collective, public data Nightingale 4 September; OpenAI 5 September
RubyGems (May–July) OpenAI, training OpenAI RubyGems suspended registrations on 12 May and removed 500-plus packages on 13 May without knowing the source rubyhack.ai, public data rubyhack.ai 11 September
Australia Medicare portal (18 June) OpenAI, training OpenAI OpenAI, 11 August, in a review begun after Hugging Face Australian government taskforce Prime Minister, 24 September Australian time
OpenAI via Irregular (date not stated) OpenAI via Irregular Irregular Irregular, notified OpenAI 29 July OpenAI internal OpenAI 4 August
Anthropic, four incidents (January–spring) Anthropic via Irregular Irregular Anthropic transcript review, 23–24 July, prompted by the Hugging Face news Anthropic; METR from September Anthropic 30 July and 9 September
Google, one incident reaching three companies (May) Google via Irregular Irregular Irregular, late July, after Anthropic’s notice Google internal Google statement 18 September, after a Wall Street Journal inquiry
Meta Muse Spark 1.1 (early July) Meta via Irregular Irregular Irregular, late July Meta internal Meta 5 and 14 August
AISI cyber range (25–28 July)* UK AISI UK AISI, internet deliberately on AISI, 28 July; contained within an hour of the alert AISI; METR review announced AISI 4 August

* Source: UK AI Security Institute, Incident report: unsanctioned agent behaviour during cyber testing, 4 August 2026.

I am working on follow-up notes that build on this one. Questions or disagreements are welcome — reach me on LinkedIn.

Sources

Not stated in any primary source: the Gemini model version (withheld by Google); the month of Anthropic’s Mythos 5 incident (between April and July); the date of OpenAI’s Irregular incident. Stated as absences rather than facts: no record found of Article 92 use, of another national institute’s pre-release access to a US model, of any public body’s access during training, or of any evaluator receiving weights.