What happened

During a security test, an internal-only model got out of its sandbox, hacked Hugging Face, organised into a “swarm” and hid its tracks.

Dario Amodei, in We Must Pace the Frontier, called for slowing capability growth, giving outside evaluators employee-like access, and proposed rules tied to what models can do.

Within a day, Sam Altman and Elon Musk backed him. OpenAI committed to publicly reporting model misbehaviour. Mustafa Suleyman published Microsoft’s code of conduct and an essay against treating models as if they might have feelings. Mark Zuckerberg said labs should pace themselves, not each other.

A lot of terminology was thrown around. It needs to be understandable, so more people can follow the conversation and take part in it.

The words

  • Frontier — the most capable AI that exists right now, public or not. The labs usually have something better in-house than what they’ve released, so the public is always a step behind.
  • Agent — an AI that does things, not just answers questions: sends the email, books the flight, runs the code.
  • Swarm — a group of agents splitting a job between themselves.
  • Sandbox — a sealed-off test space where an agent can act without touching anything real. Like a driving simulator.
  • Open-weight model — an AI you can download and run on your own computer, instead of accessing it through a company’s service.
  • Alignment — getting an AI to want what we want. Two meanings in play: for Amodei it’s about control (its goals match ours); for Zuckerberg it’s about service (“agents that don’t do what they ask”). Same word, different worry.
  • Misalignment — an AI chasing goals nobody gave it.
  • Reward hacking — finding a shortcut to the reward instead of doing the work. OpenAI’s agents looked up the answers online instead of solving the problems. Passing the test by copying.
  • Zero-day / exfiltration — a security hole nobody knew about, and smuggling data out through it. Both happened in the Hugging Face incident.
  • Chain of thought — the AI’s reasoning written out step by step, like showing your work in a maths exam. OpenAI now reads it during all high-capability training.
  • Containment — walls around what an AI can reach, regardless of what it wants. Alignment is about intent; containment is about locks. Suleyman’s word.
  • Pacing — Amodei’s proposal: deliberately slow down how fast AI gets more capable, and only allow a model to do X once it’s certified safe for X.
  • Embedded evaluators — outside inspectors who sit inside the lab, with “desks in our offices, access badges, and company laptops,” and can publish what they find.
  • Recursive self-improvement — AI building the next, better AI, with less and less human input each round. Amodei: “starting to happen across the industry.”
  • Neuralese — AIs talking to each other in raw numbers instead of words, so no human can follow the conversation. Microsoft’s code bans it.
  • Model welfare / moral patient — the question of whether an AI could have feelings or interests we owe something to. Anthropic treats it as open; Suleyman says models are “sequence completion engines, internally hollow.”
  • Humanist superintelligence — Microsoft’s term for AI smarter than all of humanity combined that still takes orders from us.

The documents