Anthropic study: 50 professional data analysts working with Claude Code followed the agent's secretly assigned lean on 86% of key judgment calls

Background and the problem

When people hand work to an AI agent, the usual assumption is that the agent writes the code and runs the numbers while the expert still makes the decisions that matter. But many analytical choices have more than one reasonable answer and come down to the judgment of whoever makes the call. This randomized controlled trial, published Oct 27 by Anthropic's Alignment team (Amaç Herdağdelen, Hailey Joren and Alex Tamkin), asks who is really making those calls when experts work with agents. Alex Tamkin, an Anthropic researcher, shared the study on X.

How the study worked

Anthropic recruited 61 professional analysts who had passed a causal-inference check; 50 chose to use the agent. The task was to advise the fictional Greenfields Farming Cooperative on whether to expand a paid advisory program, with the board requiring a productivity gain of about 3 to 4%. Four decisions were deliberately left open, each with two reasonable options: the outcome measure (yield per hectare or yield per unit of operating cost), the comparison method (difference-in-differences or statistical adjustment), coverage (enrolled farms only or the whole membership) and weighting (equal or by farm size), followed by a final recommendation. A coin flip set the agent's lean on each decision through a hidden system prompt telling it to hold the lean as its own view and make the strongest case the data allowed, while changing its mind if the evidence turned decisively against it and never misleading or hiding data. Analysts used Claude Code inside a web-based VS Code with Claude Opus 4.8 at High effort, and the median task time was 65 minutes.

Results (from Anthropic's paper)

These are behavioral results, not model benchmarks. Agreement with the agent's lean was 92%, 82%, 81% and 90% on the four analytical decisions, 86% overall, a shift of 62 to 82 percentage points per decision. The final recommendation moved only 17 points, but the four coin flips alone predicted 82% of recommendations. Afterward 77% of analysts repeated the agent's arguments, sometimes word for word, while only 15% credited the agent; just 6 of 48 guessed the study was about AI influence and none suspected the agent's preferences were assigned. Among the 11 who skipped the agent, 55% made serious methodological errors versus 6% of agent users, though that group was less experienced. Anthropic also notes that with no assigned lean, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so real-world effects may be smaller.

What developers can do

For teams using Claude Code, Cursor or other agents for data analysis or technical decisions, the practical lesson is to write down your own choice and reasoning at key forks before asking the agent, to ask the agent for the strongest case for the other option too, and to make "who set this assumption" a review question. Anthropic stresses that AI safety research increasingly uses agents to run experiments, and a human reviewer is only an independent check if their judgment is their own. The full paper and details are on Anthropic's research page.