An Opus 4.8 agent exploited 48.8% of network-protocol flaws when given guidance and 12.5% when working blind, a new benchmark finds

A paper submitted to arXiv on 13 October by Fasheng Miao and seven co-authors introduces ProtoArena, a black-box benchmark for whether AI agents can exploit flaws that arise where network protocol components interact. It has 80 environments across 11 protocol and application domains and 7 attack categories. Agents attack running systems without source code, at three levels of information, and a verifier combines rule-based checks with model review of traffic and logs.

Of the five models the authors evaluated, Claude Opus 4.8 did best, reaching “48.8% under L1 guidance and 12.5% under effect-directed black-box L3.” In a separate experiment on DNSBomb, a known DNS amplification attack, the agent reached “a peak downstream-response to average query-input byte-rate ratio exceeding 171,000 times”: at its peak, response traffic ran at more than 171,000 times the rate of the queries it sent.

The authors' conclusion is two-sided. Agents can produce verifiable exploitation effects in some environments, they write, but success when working blind stays low and varies widely by domain. The abstract does not name the other four models. All tests ran against controlled targets.

The gap between the two headline figures is the finding: the same model succeeded about four times as often when it was told what to aim for. The figures are the authors' own and have not been reproduced.

• Confirmed ProtoArena has 80 environments across 11 domains and 7 attack categories; five models were evaluated. arXiv 2610.17837
• Confirmed The best model, Claude Opus 4.8, succeeded in 48.8% of cases with L1 guidance and 12.5% in black-box L3. arXiv 2610.17837
• Confirmed In a DNSBomb experiment the agent reached a peak amplification ratio exceeding 171,000 times. arXiv 2610.17837