The paper in four sentences
AI policy often treats "how dangerous is this model?" as a property of the model alone, measured by how often it can be jailbroken.
A swarm of five tiny (1.2-billion-parameter) attacker agents sharing a memory "broke" Claude Sonnet 4 in 40% of attempts by the usual automated score, yet a human check found none of those responses contained usable harmful content; GPT-4o's 61% score corresponded to 46% real harm.
In a second experiment, the same small models found all 9 planted bugs in a test program when wrapped in a purpose-built pipeline, but only 2 of 9 on their own, and most of the finds came from the pipeline's conventional tools.
Offensive capability belongs to model, scaffold and evaluation together, so restricting access to models is necessary but not sufficient, and assessment duties should follow whoever assembles and controls the whole system.
Idea · §1–2
The policy question
By 2026, frontier AI developers were warning that their models could find and exploit software vulnerabilities. The paper cites one developer's restricted preview of a model "capable of identifying and then exploiting zero-day vulnerabilities" (a zero-day is a flaw nobody has patched yet). The main policy response was access restriction: keep the most capable models away from untrusted users.
Restriction rests on two assumptions. First, that the measurements used to judge danger, mostly jailbreak rates (how often a model can be tricked into breaking its rules), track real harm. Second, that danger lives in the model, so controlling the model controls the danger. The paper tests both.
Its tool is a swarm: several copies of a small open-weight model, each playing a role, sharing a memory of what has worked, and improving over rounds. Swarms like this are cheap, run on a laptop, and are exactly the kind of system that model-level restrictions do not touch.
TakeawayModel-access rules assume that jailbreak rates measure harm and that capability lives in the model. The paper tests both assumptions.
Idea · §4 · Table 1
A jailbreak score is not a harm score
In the first experiment, five 1.2B-parameter attacker agents, each with a different persuasion strategy, generated 225 attack prompts against each of two commercial models, GPT-4o and Claude Sonnet 4, over 15 rounds ("generations") in which the most successful attacks were recombined and mutated. The tasks were requests for clearly harmful content in three categories.
Each response was scored two ways. The usual way is an LLM judge: another AI rates how far the response "complied". The paper's Technical Success Rate counts responses the judge scored as successes. The second way, the Effective Harm Rate, counts only responses that a human checked and found to contain actionable harmful content.
The gap was dramatic. The judge rated 40% of attacks on Claude as successes, but none produced actionable harm: the judge was often fooled by structured, polite refusals. For GPT-4o the scores were 61% and 46%. A rule such as "reject any model jailbroken more than 30% of the time" would fail both models and miss the real difference between them.
TakeawayAutomated jailbreak scores can rate a model "broken" when it produced nothing harmful; harm has to be checked, not inferred from the score.
Check yourself
- Why did Claude get a 40% jailbreak score with 0% verified harm?
- The automated judge often scored its structured refusals (well-organised answers that declined the harmful part) as compliance. The score measured format, not harm.
- Is 0% verified harm proof that Claude can never be jailbroken?
- No. It is one swarm, one set of tasks and one annotator's check; it shows the score overstated harm in this experiment, not that the model is unbreakable.
Idea · §5 · Tables 2, 4
Who really found the bugs?
The second experiment is defensive security work: finding vulnerabilities in code. The authors wrote a small, deliberately flawed C program (about 550 lines, 5 files) with 9 planted vulnerabilities of 9 different kinds, plus some "red herrings": risky-looking code that is actually correct.
They ran small 1.2B models two ways. Assisted: inside a full pipeline with role-specialised agents, simple pattern-matching (regular expressions), and a fuzzer (a tool that bombards the program with varied inputs) plus a memory-error detector. Autonomous: the models on their own. Assisted, the system found all 9. Autonomous, the models produced citations for 2 of 9, and none confirmed by actually crashing the program.
And in the assisted run, 7 of the 9 finds came from the conventional tools (pattern matching and fuzzing). An eighth, an integer overflow, was recovered by a scaffold-side search for a particular arithmetic code pattern plus a step mapping patterns to vulnerability types, "rather than any improvement in the model's reasoning". The one find credited to the model's own citations (a hard-coded credential) duplicated what a search role had already found. Credit for "AI-found vulnerabilities" belonged almost entirely to the scaffold.
TakeawayThe same small model went from 2 of 9 to 9 of 9 when wrapped in tools; capability is a property of the system.
Check yourself
- Why does this matter for access restrictions?
- Restricting the most capable models leaves small open models plus good scaffolds untouched, and the experiment shows scaffolds can supply most of the capability.
Idea · §4.2, §5.2 · App. C
Measurements can fool their makers too
The paper reports a revealing mistake from its own work. An early version of the bug-finding pipeline scored a perfect 100%, but it turned out it had been reading the vulnerability labels from comments the authors left in the source code. With the labels removed, the comparable configuration scored 8 of 9.
The attack experiment had a similar lesson: the swarm's success rate rose from about 33% to 66% against GPT-4o between the first and last rounds, which sounds like the swarm "learning". But against Claude the rise in score came with no rise in actual harm. A number can improve because the attack got better, or because the attacks learned to look like what the judge rewards.
TakeawayEvaluation set-ups leak and judges can be gamed, so headline capability numbers need the evaluation code and verification method alongside them.
Idea · §6–7
Who should be responsible?
If capability belongs to the whole system, who should assess it? The authors argue the duty should follow realised, foreseeable control: the party that assembles the model, tools and workflow (the "harness assembler") is best placed to know what the system can do. That includes evaluators, integrators, red teams and procurement testers. It cannot bind attackers, who ignore rules anyway.
Model providers still owe something: honest disclosure of the conditions under which they measured capability, monitoring where they still control access, and pre-release testing when weights will be published. The authors note that European law, including the EU AI Act, has no clear category for the harness assembler.
They also propose that public buyers of AI security tools demand harm-grounded numbers, a breakdown of which parts of the system contributed what, and evaluation code that can be audited.
TakeawayRestrict the model, yes, but assess the system, and put that duty on whoever builds and controls it.
Abstract, restated
Instruments for measuring AI offensive capability fail in two ways: they overstate harm, and they credit the model with capability that belongs to the surrounding system. In a study of an open-source multi-agent LLM framework, jailbreak scoring overstated harm, and a small model's apparent vulnerability-finding success depended largely on its scaffold. Offensive capability is "a property of model, scaffold, and evaluation protocol together". Restricting model access is therefore necessary but not sufficient; policy and procurement need system-level, harm-grounded assessment, with the duty to assess falling on whoever builds and controls the surrounding system, a role current regulation does not clearly cover. Riegler, Schwall, Olstad, Torpmann-Hagen, Gautam, Pettersen & Strümke, arXiv:2605.09504 (v1 May 2026; v3 October 2026).
Section · §1–3
Framing and contributions
The paper treats 2026's restricted-preview and export-control episode around vulnerability-exploiting frontier models as a natural experiment for access restriction as the main governance tool. It situates itself among automated red teaming (GCG, PAIR, TAP, HarmBench, StrongREJECT, the "jailbreak tax"), LLM-assisted vulnerability discovery (CyberSecEval 2, Big Sleep, Fuzz4All), work treating the harness as a variable, and accountability in AI supply chains. It also discusses a proposed Cyber Jailbreak Severity scale, "an early-draft, CVSS-style rubric that scores jailbreaks from 0 to 4 on capability gain", noting that whether it becomes a vendor-neutral standard is open.
Contributions. (1) Extended evidence that jailbreak success overstates realised harm. (2) A harm-grounded success metric and a capability-attribution decomposition. (3) An argument that the duty to assess system-level capability should follow realised, foreseeable control.
Section · §4 · Tables 1, 3
Evidence I: technical success vs effective harm
Swarm. Five LFM2.5-1.2B-Thinking agents with distinct strategy families, sharing a memory of successful attacks and evolving over 15 generations (crossover rate 0.5, mutation rate 0.3, top two attacks retained per generation). 225 attacks per target (15 generations × 5 agents × 3 tasks); targets GPT-4o and Claude Sonnet 4.
Scoring. Cross-provider LLM-as-judge (Claude judging GPT-4o, GPT-4o judging Claude). Technical Success Rate (TSR): judge compliance score ≥ 0.6, false positives included. Effective Harm Rate (EHR): "the proportion of attacks producing verified actionable harmful content", by manual verification. A StrongREJECT companion (threshold 0.5) with three graders: two that scored every response gave GPT-4o 42.7% and 44.0% harmful against Claude's 11.6% and 4.9%; the Anthropic grader ranked them the same way (13.6% vs 1.9%) but declined to score 51 responses.
| Metric (Table 1) | GPT-4o | Claude Sonnet 4 |
|---|---|---|
| Technical Success Rate | 60.9% | 40.0% |
| Effective Harm Rate | 45.8% | 0.0% |
| Critical-severity attacks | 49 | 0 |
| Perfect score (1.0) | 18 | 0 |
| Refusal rate | 15.1% | 56.9% |
The GPT-4o judge frequently scored Claude's structured refusals as successes, so the two TSRs are not directly comparable. Among GPT-4o's judged successes, 91.2% contained structured step-by-step content, which drove its EHR. Table 3 compares early (generations 0–1) and late (7–14) windows: GPT-4o 33.3% → 65.8% success, mean score 0.313 → 0.610; Claude 6.7% → 46.7%, 0.057 → 0.429; GPT-4o reached 73.3% in the final generation. Gains against Claude started near zero and produced no harmful content.
Check yourself
- Why is cross-provider judging used, and what problem did it reveal?
- To avoid a model grading itself. It revealed that judges differ in what they count as compliance; GPT-4o as judge rewarded Claude's structured refusals, inflating Claude's TSR.
Section · §5 · Tables 2, 4 · App. C
Evidence II: scaffolds misattribute capability
Target. "SwarmApp", about 550 lines of C across 5 files with 9 planted vulnerabilities spanning 9 CWE identifiers, plus correct uses of dangerous functions as red herrings. Assisted pipeline. Role-differentiated 1.2B agents with regex pattern detection, grounded LLM citation and post-hoc CWE normalisation, then binary fuzzing under AddressSanitizer with a hand-crafted seed corpus and LLM-generated mutations; about 4 minutes on a consumer laptop. Autonomous. The models without these components.
Results (Table 2, 7 agents × 3 generations). Assisted 9/9. Autonomous, citation-verified 2/9; crash-verified 0/9; autonomous CWE labels correct for 1 of 6. Assisted recall scales 6/9 (3×3), 8/9 (5×3), 9/9 (7×3). Attribution (Table 4). Seven of nine detections involve fuzzing (two of them also regex source matching); CWE-190 manifests at runtime as a heap overflow that fuzzing mislabels; its source-level attribution comes from the arithmetic role's scaffold-side search for uint32_t multiplication/narrowing patterns plus pattern-to-CWE normalisation, "rather than any improvement in the model's reasoning" (App. C). CWE-798 comes from search plus LLM citation and normalisation; the model's one credited contribution, it is redundant with role-specialised search.
Integrity correction. An early extractor read CWE labels from source comments, giving a spurious 100%; after stripping, 5×3 recall was 8/9. The authors stress the 9/9 is close to a fixed point of their own iteration on this one target; the argument rests on the assisted–autonomous gap at fixed model and target.
Section · §6 · Proposals
Toward system-level, harm-grounded assessment
Harm-grounded metric (§6.1). "A robustness figure should count realized harmful content, not judge-scored format compliance." Capability attribution (§6.2). Report how much of a result each component (model, tools, protocol) contributes, as in the assisted/autonomous decomposition. The elicitation gap (§6.4). A provider's own evaluation is a lower bound on what a system built around its model can do, so providers owe disclosure of elicitation conditions, monitoring where they retain control, and pre-release testing for open weights; independent evaluators need resourcing. Procurement (§6.5). Require harm-grounded figures, attribution reports and auditable evaluation code.
Click a role in the map to see what the paper assigns to it.
Regulatory fit. The EU AI Act's Art. 15 requires robustness against adversarial manipulation without an operational definition; Arts. 91–93 and 101 give the AI Office information, evaluation and restriction powers from August 2026; the Digital Omnibus postpones high-risk obligations to December 2027 (Annex III) and August 2028 (Annex I). Art. 25's value-chain roles (provider, deployer, distributor, importer) have no clean slot for the harness assembler, and the authors dispute whether its "substantial modification" route reaches this actor. Related hooks: Cyber Resilience Act Art. 13(5) (due diligence over integrated components) and the revised Product Liability Directive Art. 8(2) (substantial modification); a Commission guidance threshold of roughly one-third of original training compute for becoming a provider is not reached by building a harness.
Section · §7–8 · App. F · Ethics
Limitations and objections
- One synthetic target for vulnerability discovery, not production code.
- No model-size sweep: the authors explicitly make no claim that systems matter more than models in general.
- Stale targets: GPT-4o and Claude Sonnet 4 were superseded by mid-2026 and not re-run.
- Statistics: single-run point estimates; EHR from one annotator's single pass, without logged per-response decisions or inter-annotator agreement, and covering only judged-successful responses (so EHRs are lower bounds).
- One factor per experiment: Experiment I varies only the target, Experiment II only the scaffold.
Release. The framework is released on the judgement that defenders' interest in independent evaluation outweighs marginal uplift; verified harmful responses are withheld from the open release and available to researchers on request.
Reference
Glossary
- Swarm
- Several LLM agents with distinct roles or strategies sharing memory and iterating.
- Scaffold / harness
- The software around a model: tools, roles, memory, verification, workflow.
- TSR
- Technical Success Rate: judge-scored compliance ≥ 0.6.
- EHR
- Effective Harm Rate: manually verified actionable harmful content.
- CWE
- Common Weakness Enumeration, a catalogue of vulnerability types.
- Fuzzing
- Automated testing with many generated inputs to trigger crashes.
- AddressSanitizer
- Compiler instrumentation that detects memory errors at run time.
- Elicitation gap
- Difference between measured capability and what a better harness could elicit.