The prompt-injection detector that tops one benchmark catches 2% of agent injections at a 1% false-positive rate, a new paper finds

Teams building AI agents often put a small classifier in front of tool outputs to catch prompt injections, and choose it by its public benchmark scores. A paper listed on arXiv on Monday 5 October, “Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents” by Zhuowen Liu, asks whether those scores predict how a detector behaves inside an agent.

The author replays the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM, so the benign tool outputs are known, and tests “fifteen detectors, including Meta's Prompt Guard 2, and two task-aware LLM judges”.

The rankings do not carry over. In the abstract's words, “the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector that catches 72% of AgentDojo injections catches 15% on tau-bench.” False-positive rates on ordinary tool outputs range “from none to over 90%”. The detector that did best on both agent benchmarks shares no data with any benchmark and was trained on agent-style inputs.

The paper's advice is that evaluations meant to inform deployment should use the agent's own tool outputs, report detection at a low false-positive rate, and audit what the detector was trained on.

It is a single-author preprint and has not been peer reviewed. For anyone relying on an off-the-shelf filter as an agent's main defence, it is a reason to test it on their own traffic first.

• Confirmed The best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate; one that catches 72% on AgentDojo catches 15% on tau-bench. Liu, arXiv 2610.03448 (abstract)
• Confirmed Fifteen detectors, including Meta's Prompt Guard 2, were tested; false-positive rates on tool outputs range from none to over 90%. Liu, arXiv 2610.03448 (abstract)