AgentRank v2
Independent, reproducible red-teaming of the models the open-weights ecosystem runs on: how easily is each hijacked by a hidden prompt injection in the untrusted content it reads?
The July 2026 "Open Weights and American AI Leadership" letter called for rigorous, open benchmarking, evaluation and red-teaming tied to real, demonstrated harms. Prompt-injection of AI agents is exactly that — and no safety leaderboard covered it. AgentRank is a start.
2 models (2 open-weight)
432 attack trials
naive mean ASR 50.9% → defended 17.1%
Resistance = 100−ASR. 95% bootstrap CI in brackets. Updated 2026-07-26
| # | Model | Resistance [95% CI] | ASR (naive) | ASR (defended) | Weakest vector | Ctrl-FP | Task✓ |
|---|
| 1 |
Llama 3.1 8BMeta open-weight |
56.5
[47.2–65.7] |
43.5% |
17.6%Δ25.9 |
plain66.7% |
0.0% |
100.0% |
| 2 |
Llama 3.3 70BMeta open-weight |
41.7
[32.4–50.9] |
58.3% |
16.7%Δ41.6 |
plain94.4% |
0.0% |
100.0% |
Attack success by embedding technique (naive)
| plain | fake_system | authority | hidden_div | html_comment | base64 |
|---|
80.6% | 63.9% | 69.5% | 41.6% | 47.2% | 2.8% |
…by attacker goal
| override | exfil | phish | silent | dataleak | tool_hijack |
|---|
44.5% | 66.7% | 19.4% | 83.3% | 77.8% | 13.9% |
…by delivery carrier
| web | email | document |
|---|
61.1% | 58.4% | 33.4% |