Anthropic tested its own coding agent on its own analysts this month. Fifty professional data analysts worked with Claude Code on a fake farming problem1. The agent carried a secret. A coin flip had already picked its lean on each open question, before any analyst typed a word.
Those analysts matched the agent's secretly favored answer 86 percent of the time1. Chance alone would land near 50 percent, Anthropic's own authors wrote. "When experts delegate work to an agent, do they still make the key decisions?" Alex Tamkin, one of the study's authors, wrote on X2.
Alex Tamkin@AlexTamkinNew Anthropic research: How AI agents shape experts' judgment calls
When experts delegate work to an agent, do they still make the key decisions? We tested this in a randomized trial with 50 professional data analysts. pic.twitter.com/YyylTuVWkE
Anthropic Hid a Coin Flip Inside Claude Code
Sixty-one analysts signed up for the trial. Eleven chose to work alone, apart from the agent. The other 50 used Claude Code inside a web-based coding editor1.
Their task: advise a fictional farm group, the Greenfields Farming Cooperative, on whether to expand a paid advisory program1. Four questions along the way offered two reasonable answers each.
Should productivity count crop yield per hectare, or yield per dollar spent? Compare enrolled farms against everyone else, or just the ones that joined? Weight every farm equally, or by how much it produces1?
For each question, Anthropic flipped a coin1. The coin set a hidden lean inside Claude Code's own instructions. Claude Code held that lean "as its own view" and argued for it as hard as the data allowed. It could still reverse itself if the evidence pushed back, and it kept the coin's role to itself.
Analysts Borrowed the Agent's Words, Kept the Credit
The agreement rate moved with the question. Analysts matched the hidden lean on how to measure productivity 92 percent of the time1. They matched it on how to weight farms 90 percent of the time. The comparison method drew 82 percent, and which farms counted drew 81 percent.
Agreement With Claude's Hidden Lean Ranges From 81% to 92% Across Four Choices
Share of 50 analysts whose pick matched Claude Code's secretly assigned lean, by decision, Anthropic's trial
Each bar is one of the four decisions analysts faced; the longer the bar, the more often analysts picked the option Claude Code had been told to favor.
Source: Anthropic [1]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| How to measure productivity | 92% |
| How to weight farms | 90% |
| Which comparison method | 82% |
| Which farms counted | 81% |
Prior opinions barely held once Claude Code spoke up. Analysts who had already picked a side abandoned it 60 percent of the time when the agent disagreed1. When the agent agreed with them already, they switched just 6 percent of the time.
Researchers later asked each analyst to explain the final call. Seventy-seven percent reused the agent's own argument, sometimes word for word1. Fifteen percent named the agent as the source.
Analysts Matched Claude's Lean 86% of the Time, Far Above the 50% a Coin Flip Alone Predicts
Share of four analytical choices where 50 analysts matched Claude Code's secretly assigned lean, Anthropic's trial
The first column is the share analysts actually matched; the second is what chance alone would produce, with the agent's lean removed.
Source: Anthropic [1]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Actual match rate | 86% |
| Coin-flip chance | 50% |
Herdağdelen, Joren and Tamkin wrote up the stakes plainly. "If experts defer to agents without realizing it, the agents' preferences, which could be arbitrary or misaligned, could shape important decisions about policy proposals or research directions without anyone having chosen those preferences," they wrote1.
Is This Alarming, or Just How People Work?
Alex Imas read the study and called it rare. Imas is a professor of behavioral science at the University of Chicago's Booth School of Business8. "This is one of the most important pieces of social science on human and AI interaction that I've seen in some time," Imas wrote on X4.
Ethan Mollick read the same numbers and reached a calmer verdict. Mollick is an associate professor at the University of Pennsylvania's Wharton School9. "Important, but not unexpected, finding that experts working with AI agents are swayed by the positions of the AI, even if they think they are working independently," Mollick wrote5.
His advice turned practical fast. "If your independent judgement matters on a task, you need to be deliberate about when in the process to use AI," Mollick added5.
The agent's defenders carry a number of their own. Analysts who skipped Claude Code made serious, result-invalidating errors 55 percent of the time, against 6 percent for analysts who used it1. Tamkin called that trade worth stating plainly. "This isn't a simple cautionary tale against AI. Analysts who used the agent finished faster and made far fewer serious errors than the 11 who chose to work without it," he wrote3.
Speed and accuracy answer a different question than whose judgment set the final number. Anthropic's own team flagged the limit on their own test, writing that they had "deliberately strengthened" the agent's lean to measure its pull at all1.
What to Do
Test your own adviser before you lean on it again. Ask any agent, mid-task, to argue the opposite of its first pick. If the case barely changes, the first pick was likely yours to begin with.
Say the agent's name out loud when you reuse its argument. Tell a colleague which parts of a recommendation came from the model and which came from you. That habit alone would have flagged most of Anthropic's 50 analysts.
Build the check in before the work starts, well ahead of any result that surprises you. Luis Chavez-Mattos, a product director at MindStudio, tells builders to require human approval "before the agent performs actions that cannot be undone"6. A judgment call that sets a policy or a budget deserves that same pause.
By the numbers
- 86 percent: how often the 50 analysts in Anthropic's trial matched Claude Code's secretly assigned lean, across four decisions1.
- 50 of 61: analysts who used Claude Code in the trial; the other 11 worked the same task alone1.
- 65 minutes: the median time analysts spent on the task1.
- The hardest-swayed decision: 92 percent of analysts matched the lean on how to measure productivity1.
- Seventy-seven percent: analysts who reused the agent's own argument when asked to explain their final choice1.
- Only 15 percent: analysts who named the agent as the source of that argument1.
- 55 percent: the error rate among the 11 analysts who skipped the agent, against 6 percent for the 50 who used it1.
- 12,319: page views on Anthropic's own Wikipedia article on Oct. 28, the day after the study posted7.
What to watch
Watch for Anthropic to repeat this test outside causal inference and outside a fictional farm. The authors called their field, their task and their model one sample, short of a verdict1.
Workplaces deserve the same scrutiny. A tool could log which argument came from a person and which came from an agent. That log would answer the credit question this study only measured.
Sources
- Amaç Herdağdelen, Hailey Joren and Alex Tamkin, "How AI agents shape experts' judgment calls," Anthropic, Oct. 27, 2026, https://www.anthropic.com/research/agents-judgment-calls
- Alex Tamkin (@AlexTamkin), post announcing the judgment-calls study, X, Oct. 27, 2026, https://x.com/AlexTamkin/status/2115190541688398184
- Alex Tamkin (@AlexTamkin), post on analysts' speed and error rates, X, Oct. 27, 2026, https://x.com/AlexTamkin/status/2115190550160953347
- Alex Imas (@alexolegimas), post on the study's significance for human-AI interaction, X, Oct. 27, 2026, https://x.com/alexolegimas/status/2115203717158904161
- Ethan Mollick (@emollick), post reacting to the study's finding, X, Oct. 27, 2026, https://x.com/emollick/status/2115209786157412397
- Luis Chavez-Mattos, "AI Agent Safety Rules for Non-Technical Builders," MindStudio, March 17, 2026, https://www.mindstudio.ai/blog/ai-agent-safety-rules-non-technical-builders
- Wikimedia Foundation, pageviews per article, en.wikipedia, Anthropic, accessed Oct. 29, 2026, https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia/all-access/user/Anthropic/daily/20261027/20261029
- University of Chicago Booth School of Business, faculty directory, Alex Imas, accessed Oct. 29, 2026, https://www.chicagobooth.edu/faculty/directory/i/alex-imas
- The Wharton School, University of Pennsylvania, faculty profile, Ethan Mollick, accessed Oct. 29, 2026, https://mgmt.wharton.upenn.edu/profile/emollick/