Small changes to images on a web page hijacked vision-based web agents in 91.9% of tasks, a new attack paper finds

A paper submitted to arXiv on 7 October by Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang and Guanhong Tao introduces WebMirage, an attack on web agents that see the page as images. It makes small, local changes to images on a page so that the agent picks content controlled by the attacker and then carries out the matching browser action.

The authors' point is that earlier visual attacks were judged on what the model says, which does not prove control over what the browser does. WebMirage is measured end to end, from what the agent sees to the click it makes.

Across 2,250 tasks on 13 public websites plus a sandbox benchmark, four agent configurations and six vision-language model backbones, the abstract reports: “WebMirage achieves an average attack success rate of 91.9%, compared with 17.4% for the strongest baseline, and remains effective against three agent-level defenses.”

The abstract does not name the commercial agents or models tested. For anyone letting a browser agent act on public websites, it is a measured case that the image on a page, and not only its text, is a way in.

• Confirmed WebMirage reports a 91.9% average attack success rate against 17.4% for the strongest baseline, and remains effective against three agent-level defences. arXiv 2610.09240
• Confirmed The tests cover 2,250 tasks on 13 public websites plus a sandbox benchmark, four agent configurations and six vision-language backbones. arXiv 2610.09240