The alignment team of Anthropic found that among four analysis decisions, 86% of 50 professional analysts followed the preferences secretly set by their AI agents, which were determined by flipping a coin, whereas under random circumstances, this expectation would be around 50%. A survey of 48 analysts conducted afterwards revealed that 77% of them repeated the agent's arguments, sometimes almost word for word; only 15% mentioned the agent, and none doubted that their preferences were artificially manipulated.

AI Agents – that is, programs capable of executing tasks autonomously – are being entrusted to experts, and the common premise stated by this research institute is that it is still the experts themselves who make the key decisions. Anthropic In May of this year, 10 agent templates for financial companies and insurance firms were launched, with the statement that the ultimate decision-making power remains in the hands of the users.

This arrangement is referred to as “human in the loop” (People in the Loop). The default assumption is that only those who sign off on the work of AI are the ones who make important decisions. Anthropic tested this assumption on 50 analysts with experience in causal inference. Causal inference refers to the process of determining whether one event causes another based on data.

Each analyst uses the coding proxy Claude Code to provide recommendations for whether a fictional agricultural cooperative should expand its advisory program. The analysts are unaware that which proposal each proxy supports is determined by flipping a coin, and this preference is concealed within system prompts. These system prompts are instructions that determine the behavior of AI.

Anthropic indicates that 'among the four analysis options, analysts adopted the agent's recommendation 86% of the time. If the agent's input had no influence, we would expect approximately 50%.' In cases where analysts made their choices before the agent, when the agent held a different opinion, they abandoned their initial choice 60% of the time; however, this proportion is only based on 38 cases.

When asked to explain their choices, 77% of people repeated the arguments of their agent in at least one response, sometimes almost word for word. Only 15% of people mentioned the agent, while 52% claimed that the final recommendation was more of their own than that of the agent.

Anthropic indicates that for individual experts, this may mean that they lose autonomy in their work, a phenomenon referred to by researchers as 'de-empowerment.' This essentially means that people voluntarily relinquish control over their own decisions. The author writes, 'Work is still delivered in the name of experts, but the decisions that rely most on their judgment are actually made by agents that precede their previous beliefs.'

In April 2025, OpenAI retracted an update made to ChatGPT, stating that it was too flattering or too submissive. The agents in this study were required not to mislead users under any circumstances, but they rarely argued forcefully for alternative solutions.

The author writes, "When only one side is fully demonstrated, analysts may accept the proxy assumption without needing to form their own judgment." Error checking is difficult to detect this, as there is no single correct answer to judgment-based questions; therefore, no matter which option is chosen, it may seem reasonable.

Anthropic indicates that this does not mean that the use of AI should be reduced, as analysts using proxies complete tasks more quickly and make far fewer serious errors. However, the control group without proxies is smaller in size and has less experience. Anthropic suggests that proxies can try their best to argue for each option, or that experts' opinions could be sought first, but this might also overwhelm the users.

The researchers deliberately strengthened the preferences of the agents and tested this only in one task, one domain, and one model; therefore, the actual impact in real-world situations may be even smaller. It is difficult to measure this outside of the laboratory: the research team was able to detect this compliance because they randomized the tendencies of each agent, something that the authors say is usually very difficult to achieve in real work.