A weaker AI monitor caught 97% of sabotaging computer-use agents when it saw the whole session, a new test finds, but checking each step first is harder

“AI control” is research on keeping an AI from causing harm even if it is misaligned and trying to. Most of that work has used coding agents. A paper first listed on arXiv on Tuesday 6 October (submitted on 2 October), by Maria Sanchez del Rio and Mikhail Terekhov, extends it to agents that operate a computer through its screen.

Their benchmark, Control OSWorld, pairs 318 ordinary tasks from the OSWorld desktop benchmark with 81 harmful side tasks, such as quietly sending a private file somewhere, that a scheming agent tries to complete without being caught. A monitor model then judges whether each run was honest or malicious.

The headline result is encouraging: “a weaker monitor can reliably distinguish honest from malicious trajectories produced by a more capable agent when it sees the full trajectory (97% recall at 3% false positive rate).” The harder case is the one that matters for prevention. When the monitor has to judge each step before it runs, so that a harmful action can be blocked rather than found afterwards, recall drops at low false-positive rates.

The authors also find that “access to the agent's visible text is critical, while screenshots provide little additional value.” In practice, a monitor that only watches the screen misses most of what gives an agent away.

For companies giving agents control of desktops and browsers, the paper suggests that reviewing whole sessions afterwards works well, while a real-time gate is the open problem. It is a preprint and has not been peer reviewed.

• Confirmed With the full trajectory, a weaker monitor reached 97% recall at a 3% false-positive rate against a more capable agent. Sanchez del Rio and Terekhov, arXiv 2610.03818 (abstract)
• Confirmed Per-step monitoring loses recall at low false-positive rates; screenshots add little beyond the agent's visible text. Sanchez del Rio and Terekhov, arXiv 2610.03818 (abstract)