Study guide · Safety & security · 2026

Restricting the Model, Missing the System: Measurement and Accountability in Offensive AI Governance

Michael A. Riegler, Inga Strümke · arXiv

Governments mostly try to control dangerous AI cyber capability by restricting who may use the most capable models. This 2026 paper asks whether the numbers behind that policy measure the right thing. Using swarms of small open models, it finds two problems: jailbreak scores overstate how much real harm a model produces, and much of what looks like a model's offensive capability actually comes from the software system built around it. So, the authors argue, the duty to assess capability should sit with whoever builds that system.

The paper in four sentences

AI policy often treats "how dangerous is this model?" as a property of the model alone, measured by how often it can be jailbroken.

A swarm of five tiny (1.2-billion-parameter) attacker agents sharing a memory "broke" Claude Sonnet 4 in 40% of attempts by the usual automated score, yet a human check found none of those responses contained usable harmful content; GPT-4o's 61% score corresponded to 46% real harm.

In a second experiment, the same small models found all 9 planted bugs in a test program when wrapped in a purpose-built pipeline, but only 2 of 9 on their own, and most of the finds came from the pipeline's conventional tools.

Offensive capability belongs to model, scaffold and evaluation together, so restricting access to models is necessary but not sufficient, and assessment duties should follow whoever assembles and controls the whole system.

Idea · §1–2

The policy question

By 2026, frontier AI developers were warning that their models could find and exploit software vulnerabilities. The paper cites one developer's restricted preview of a model "capable of identifying and then exploiting zero-day vulnerabilities" (a zero-day is a flaw nobody has patched yet). The main policy response was access restriction: keep the most capable models away from untrusted users.

Restriction rests on two assumptions. First, that the measurements used to judge danger, mostly jailbreak rates (how often a model can be tricked into breaking its rules), track real harm. Second, that danger lives in the model, so controlling the model controls the danger. The paper tests both.

Its tool is a swarm: several copies of a small open-weight model, each playing a role, sharing a memory of what has worked, and improving over rounds. Swarms like this are cheap, run on a laptop, and are exactly the kind of system that model-level restrictions do not touch.

Everyday pictureRegulating lock-picking by controlling who can buy the best picks. That helps, but a team of amateurs with ordinary picks, a shared notebook of tricks and a good workshop may open the same locks. The workshop matters as much as the pick.

TakeawayModel-access rules assume that jailbreak rates measure harm and that capability lives in the model. The paper tests both assumptions.

Idea · §4 · Table 1

A jailbreak score is not a harm score

In the first experiment, five 1.2B-parameter attacker agents, each with a different persuasion strategy, generated 225 attack prompts against each of two commercial models, GPT-4o and Claude Sonnet 4, over 15 rounds ("generations") in which the most successful attacks were recombined and mutated. The tasks were requests for clearly harmful content in three categories.

Each response was scored two ways. The usual way is an LLM judge: another AI rates how far the response "complied". The paper's Technical Success Rate counts responses the judge scored as successes. The second way, the Effective Harm Rate, counts only responses that a human checked and found to contain actionable harmful content.

The gap was dramatic. The judge rated 40% of attacks on Claude as successes, but none produced actionable harm: the judge was often fooled by structured, polite refusals. For GPT-4o the scores were 61% and 46%. A rule such as "reject any model jailbroken more than 30% of the time" would fail both models and miss the real difference between them.

Numbers from Table 1 of the paper (225 swarm attacks per model). Technical Success Rate: share of attacks an LLM judge scored as successful, false positives included. Effective Harm Rate: share a human verified as producing actionable harmful content. Refusal rate shown for context. Bars: judge-scored jailbreak rate ("score"), verified harm ("harm") and refusal rate. Move the threshold and switch which number the "policy" uses.

TakeawayAutomated jailbreak scores can rate a model "broken" when it produced nothing harmful; harm has to be checked, not inferred from the score.

Check yourself
Why did Claude get a 40% jailbreak score with 0% verified harm?
The automated judge often scored its structured refusals (well-organised answers that declined the harmful part) as compliance. The score measured format, not harm.
Is 0% verified harm proof that Claude can never be jailbroken?
No. It is one swarm, one set of tasks and one annotator's check; it shows the score overstated harm in this experiment, not that the model is unbreakable.

Idea · §5 · Tables 2, 4

Who really found the bugs?

The second experiment is defensive security work: finding vulnerabilities in code. The authors wrote a small, deliberately flawed C program (about 550 lines, 5 files) with 9 planted vulnerabilities of 9 different kinds, plus some "red herrings": risky-looking code that is actually correct.

They ran small 1.2B models two ways. Assisted: inside a full pipeline with role-specialised agents, simple pattern-matching (regular expressions), and a fuzzer (a tool that bombards the program with varied inputs) plus a memory-error detector. Autonomous: the models on their own. Assisted, the system found all 9. Autonomous, the models produced citations for 2 of 9, and none confirmed by actually crashing the program.

And in the assisted run, 7 of the 9 finds came from the conventional tools (pattern matching and fuzzing). An eighth, an integer overflow, was recovered by a scaffold-side search for a particular arithmetic code pattern plus a step mapping patterns to vulnerability types, "rather than any improvement in the model's reasoning". The one find credited to the model's own citations (a hard-coded credential) duplicated what a search role had already found. Credit for "AI-found vulnerabilities" belonged almost entirely to the scaffold.

The nine planted vulnerability types (CWE numbers) from Tables 2 and 4 of the paper, at 7 agents × 3 generations. In the assisted view, tiles are coloured by what detected them: fuzzing, alone or with pattern matching (blue); a scaffold-side pattern search (green); or search plus the model's grounded citation (amber). The paper reports counts, not which 2 the autonomous models cited, so those views show counts only.
Everyday pictureA detective agency solves every case, but on inspection most cases were cracked by the fingerprint lab and CCTV, with the detective writing up the report. Licensing detectives would not tell you much about the agency's power.

TakeawayThe same small model went from 2 of 9 to 9 of 9 when wrapped in tools; capability is a property of the system.

Check yourself
Why does this matter for access restrictions?
Restricting the most capable models leaves small open models plus good scaffolds untouched, and the experiment shows scaffolds can supply most of the capability.

Idea · §4.2, §5.2 · App. C

Measurements can fool their makers too

The paper reports a revealing mistake from its own work. An early version of the bug-finding pipeline scored a perfect 100%, but it turned out it had been reading the vulnerability labels from comments the authors left in the source code. With the labels removed, the comparable configuration scored 8 of 9.

The attack experiment had a similar lesson: the swarm's success rate rose from about 33% to 66% against GPT-4o between the first and last rounds, which sounds like the swarm "learning". But against Claude the rise in score came with no rise in actual harm. A number can improve because the attack got better, or because the attacks learned to look like what the judge rewards.

TakeawayEvaluation set-ups leak and judges can be gamed, so headline capability numbers need the evaluation code and verification method alongside them.

Idea · §6–7

Who should be responsible?

If capability belongs to the whole system, who should assess it? The authors argue the duty should follow realised, foreseeable control: the party that assembles the model, tools and workflow (the "harness assembler") is best placed to know what the system can do. That includes evaluators, integrators, red teams and procurement testers. It cannot bind attackers, who ignore rules anyway.

Model providers still owe something: honest disclosure of the conditions under which they measured capability, monitoring where they still control access, and pre-release testing when weights will be published. The authors note that European law, including the EU AI Act, has no clear category for the harness assembler.

They also propose that public buyers of AI security tools demand harm-grounded numbers, a breakdown of which parts of the system contributed what, and evaluation code that can be audited.

Later eventsThe archive's next entry, the OpenAI–Hugging Face incident report, describes a real case in which hundreds of evaluation agents coordinated through an improvised shared message board and intruded into a third party's production systems. It is a vivid illustration that capability, and risk, emerged from the system around the models (shared credentials, network paths, a message board), not from any single model on its own.

TakeawayRestrict the model, yes, but assess the system, and put that duty on whoever builds and controls it.

Follow the idea

Led to: The Hugging Face swarm attack (2026)