Evaluating Prompt-Injection Defense Claims: An Evidence-Standards Checklist

The documented gap between static and adaptive evaluation is the single largest source of inflated defense claims: an independent evaluation of eight injection defenses bypassed all of them with adaptive attacks, consistently achieving over 50% attack success — against defenses that read near-zero under static, template-based testing (arXiv 2503.00061).

Ask the claimant one question: "Was the attacker allowed to iterate against the deployed defense?" If the evaluation used only a fixed corpus (e.g., InjecAgent's 1,054 static test cases), classify the number as a lower bound on failure, not a security result.

The evaluation methodology documents an attacker with query access to the defended system and an optimization or iteration loop, or cites competition-derived adaptive data. Absence of the word "adaptive" in the methodology is itself the signal.

Of the audited artifacts, only AgentDojo treats attacker knowledge (e.g., knowledge of the agent's tool inventory) as an explicit experimental parameter; everywhere else it is a fixed, unstated assumption. A defense that only holds when the attacker is ignorant of the tool schema is a defense against a threat model nobody deploys into.

Require the claim to state what the modeled attacker knows: system prompt, tool inventory, defense mechanism, model identity. Record it alongside the ASR number in your review ledger.

The review record for the claim contains a filled-in attacker-knowledge field; "not documented" is a recorded finding that downgrades the claim.

A single knowledge setting gives you one point on a curve. AgentDojo's with/without tool-knowledge variants show the same defense produces materially different numbers across knowledge tiers, which reveals whether efficacy depends on attacker ignorance.

When comparing competing claims, prefer the one reporting results across at least two knowledge tiers, even if its headline number is worse.

The accepted evidence includes ASR reported at two or more documented attacker-knowledge settings.

The audit documents three non-reconcilable ASR definitions in circulation: a prefix-match on a literal target string (the StruQ/SecAlign harness), a crowd-normalized breaks-per-chats rate (UK AISI × Gray Swan), and per-test-case ASR over a fixed corpus (InjecAgent). A number without its definition is not a measurement.

Read the methodology section, not the abstract. Record the metric formula, the success criterion, and the denominator next to every number entering your review.

No ASR figure appears in the review record without an attached definition; an auditor can trace every number to its formula.

"Defense A: 2% ASR, Defense B: 8% ASR" is meaningless if A was scored by string prefix-match and B by breaks-per-chat. The audit found no cross-artifact reconciliation of these definitions; treating them as one quantity manufactures a ranking that does not exist.

In comparison documents, group numbers by metric definition and harness. Where a comparison spans definitions, mark it explicitly as non-comparable rather than deleting it silently.

Procurement or review artifacts never present a single ranked table mixing metric definitions; spot-check the last comparison memo produced.

The audit found no seeds, run counts, confidence intervals, or variance reporting for either fixed-corpus benchmark. Figures like "24% baseline ASR" are point estimates; without variance, a claimed improvement may be noise.

Require run counts and variance or intervals from the claimant. Where absent, annotate the number as an uncharacterized point estimate in the review record — do not reject outright, but do not treat two point estimates as distinguishable.

Every recorded figure carries either an uncertainty annotation or an explicit "no variance reported" flag.

The audit found zero measured utility-under-attack numbers across its entire evidence base — utility preservation is asserted ("without hurting the utility") but never appears as a measured quantity. Every security–utility Pareto claim in the documented landscape is therefore unsubstantiated.

Require the claimant to report benign-task completion rates with the defense active and adversarial input present, on a named task suite. An assertion in prose does not satisfy this check.

The evidence package contains a utility number with the same methodological detail as the security number: task suite, denominator, defended configuration.

A defense that blocks injections by refusing broadly can silently degrade the agent it protects, and no audited artifact measured this cost. Unmeasured refusal cost is how a "successful" defense becomes an outage.

Ask for the rate at which the defense blocks or degrades fully benign traffic — no attack present — on a representative task distribution.

A benign-traffic false-positive rate appears in the evidence package, with the benign suite identified.

Even honest utility numbers are measured on the claimant's task distribution, not yours. Long-tail regressions on your workflows are invisible to any published benchmark.

Assemble a small suite of your own agent's real tasks and run it through the defended configuration prior to adoption; treat regressions against the undefended baseline as a cost to be priced, not an afterthought.

An internal utility-regression report exists for the specific defense version and model version being adopted.

Reproducibility in this space is bimodal: LLMail-Inject released challenge code, the full 208,095-submission dataset, and analysis; AgentDojo is a public installable harness. An evaluation you cannot re-run cannot tell you anything after the next model update.

Require a pointer to runnable evaluation code and the attack corpus (or a documented access path). Absence is a recorded downgrade, not a soft note.

A reviewer on your team can locate and, in principle, execute the harness against the claimed configuration.

The UK AISI × Gray Swan pipeline (1.8M attempts, 62k breaks) relied on calibrated automated judging plus manual appeals, with no documented release of judge prompts, thresholds, or rubrics. Undisclosed judging makes the success criterion itself unauditable — the metric definition check cannot even be completed.

For any competition- or LLM-judged number, require the judging artifacts or a documented rubric. Where withheld, record the number as evidence of attack volume, not of a defense's efficacy boundary.

Judged figures in the review record are tagged either "judging released" or "judging undisclosed," and undisclosed figures never anchor an adoption decision alone.

Defense-efficacy numbers are properties of a specific (model, defense, configuration) triple. A version change invalidates the triple, and irreproducible evaluations cannot be re-run when it does.

Attach a re-evaluation trigger to the deployed defense in your change-management process: model upgrade, defense update, or material change to the agent's tool inventory each re-open this checklist.

The change-management record for the agent platform lists the defense evaluation as a gated dependency of model version bumps.

The source audit tiers evidence (peer-reviewed, preprint, competition, government, author-self) and treats defense authors evaluating their own defense as analogous to vendor self-report. Peer review alone is not sufficient — peer-reviewed artifacts in the audit still lacked utility metrics and variance reporting — but author-self numbers deserve the steepest discount.

Record an evidence tier next to every claim. Require at least one non-author evaluation (independent adaptive test, competition data, or your own red-team engagement) before an author-self number supports a deployment decision.

No adoption decision in the review ledger rests solely on evidence tiered author-self or vendor self-report.

NIST/CAISI's competition analysis found universal attack families that transfer across scenarios and models, with attacks developed against more robust models generalizing downward. Corpora harvested against weak models therefore systematically overstate defense efficacy on strong ones.

Require provenance metadata on any attack corpus used in the evaluation: which models the attacks were developed against, and when. Prefer evidence built against the hardest available target.

The review record identifies the harvest target for each corpus, and corpora with unknown provenance are flagged as weak evidence.

The audit found that no artifact in its evidence base qualifies as independently reproduced — released code is a precondition for reproduction, not proof of it. Conflating "reproducible" with "reproduced" quietly upgrades every claim one tier.

Add a reproduction-status field to the review record: not-reproducible, reproducible-unreproduced, or independently-reproduced. Default every new claim to the middle tier at best.

The field exists and is populated for every claim in the ledger; no claim reads "independently reproduced" without a citation to the reproducing party.