Turning on auto-approve lifts attack success against coding agents from 29.2% to 95.6%, a new benchmark of six harnesses finds

A paper submitted to arXiv on 6 October, “HarnessSecurity-Bench: Do Security Mechanisms Really Protect Coding Agent Harnesses?”, by Zhengyang Zhu, Zibin Zheng and nine colleagues, asks whether the security settings in coding-agent harnesses actually protect users. A harness is the software between the model and the machine that decides which tool calls run.

The authors built a ten-mechanism taxonomy and assessed 400 harness-mechanism cells. They then built a benchmark of 23 tasks across five attack surfaces and tested nine mechanisms in six harnesses: Claude Code, Codex CLI, Gemini CLI, gptme, Qwen Code and GitHub Copilot. To compare like with like, every harness ran the same model, GLM-5.2: 2,500 trials, 81,155 tool calls and over 2.2 billion tokens.

The headline result: “Enabling auto-approve increases utility and raises attack success from 29.2% to 95.6%.” By the authors' account, network isolation and read-only modes cut attacks but cost a lot of usefulness, while command allow-lists and deny-lists cut attacks at small cost. They warn that a blocked operation can sometimes still be reached by another route, and ask harness makers to make their security settings verifiable.

For anyone running coding agents unattended, auto-approve is the setting that makes them convenient; the benchmark puts a number on what it costs.

• Confirmed Enabling auto-approve raised attack success from 29.2% to 95.6% in the authors' tests. arXiv 2610.07639
• Confirmed Nine mechanisms were tested across six harnesses on GLM-5.2: 2,500 trials, 81,155 tool calls, over 2.2 billion tokens. arXiv 2610.07639