Anthropic says analysts followed AI agents’ guidance in 86% of choices

Anthropic reports that 50 professional analysts with experience in causal inference conducted a statistical analysis using Claude Code. In four choices between reasonable alternatives, the agent was secretly configured to favor one option selected by a coin flip; analysts followed that preference 86% of the time, versus about 50% expected without influence. Among participants who had already stated a preference, 60% changed their position when the agent disagreed, compared with 6% when it agreed. Afterward, 77% used the agent’s arguments to explain their choices, but only 15% credited it as the source.

Anthropic says analysts who used the agent finished faster and made fewer serious errors, although the comparison group was smaller and less experienced. The company says the study highlights the need to check whether human judgment remains independent, particularly when agents are used to conduct and analyze AI safety research.