Stanford's Social and Language Technologies Lab opened a batch of 249,834 real Claude.ai chats on Aug. 26, 202612. Anthropic kept every chat on its own servers and sent the lab only the tallies. The sample ran from April through May 2026.
Most people assume a chatbot mostly drafts emails and sums up meetings. Stanford's count says otherwise. 56 percent of the chats it could sort carried work that hits someone else or is hard to reverse2.
Three readers feel this first. A hiring manager wonders if Claude only drafts email templates. Her recruiting partner screens candidates for AI fluency, Claude included. Their compliance officer just approved a chatbot for the firm's legal team.
What People Actually Bring to Claude
Stanford built a scale with four tiers2. Ephemeral work matters for an afternoon: a grocery list, a quick translation.
Operational work matters only inside one team: a meeting agenda, a status update. Consequential work touches someone outside the room, or it is hard to reverse. High-stakes work can carry legal, financial or personal consequences if it goes wrong.
The split surprised the lab's own team. Yijia Shao, the study's lead author, put it plainly the day the pilot went public: "We find that, yes, users sometimes bring trivial tasks to AI, like drafting emails, but much more often it is high-stakes, hard-to-reverse, and consequential work"3.
Count the tiers and the picture sharpens. Stanford's classifier skipped 34 percent of the sample, usually because the chat drifted across several jobs at once2.
Among the rest, 20 percent sat in the ephemeral tier and 24 percent in the operational one. Consequential work took 44 percent. High-stakes work took the remaining 12 percent.
56 percent of classifiable Claude chats carry consequential or high-stakes work
Share of 249,834 sampled Claude.ai conversations Stanford could sort into one task tier, April and May 2026
Each segment is one tier Stanford's classifier assigned; the two right-hand segments, consequential and high-stakes, add up to 56 percent of the bar.
34 percent of the full sample mixed several jobs at once and sits outside this bar entirely.
Sources: Stanford SALT Lab [2]; Stanford HAI [3]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Ephemeral | 20% |
| Operational | 24% |
| Consequential | 44% |
| High-stakes | 12% |
Add the last two tiers together and 56 percent of the classifiable chats carried real weight23. That is the number challenging the old assumption that chatbots mostly handle busywork.
Legal and Financial Guidance Tops the List
Stanford scored nine domains on a criticality index2. The index is a volume-weighted average running from 0 to 3. Legal and financial guidance ranked highest, at 2.14. Academic coursework ranked lowest, at 1.06.
Legal and financial guidance scores 2.14 on Stanford's criticality index, double academic coursework's 1.06
Task Criticality Index, a 0-to-3 volume-weighted tier average, by domain, Claude.ai conversations sampled April and May 2026
Each bar is one domain Stanford scored; a longer bar means more of that domain's chats landed in the consequential or high-stakes tiers. Lower is better for routine, low-risk work; higher marks real stakes.
Stanford scored nine domains in all. This chart shows the highest- and lowest-scoring domains it published; the other seven sit outside this view.
Source: Stanford SALT Lab [2]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Legal and financial guidance | 2.14 |
| Academic coursework | 1.06 |
A contract question or a tax question scores almost twice as high, on average, as a homework question2.
Lower Manhattan's towers hold thousands of the law and finance firms whose work shows up in that top score. A law firm's office two states away tells the same story on a smaller scale.
College work sits at the other end. A term paper rarely touches anyone outside the student and the professor. Most of it can be redone.
Humans Still Hold the Pen
Stanford also asked who stays in charge once the chat starts2. In nearly three-quarters of chats, people set the direction and Claude played the helper1.
People kept editing, too. Most users reworked Claude's answer instead of pasting it straight into their own work1.
Teaching showed up everywhere. Users taught Claude about their own case in 67 percent of chats, the study found2. Friction showed up in about half of all chats, and most of it looked useful. Almost 80 percent of the chats with friction showed the user working through it rather than giving up3.
The Case for Augmentation
Shao called the finding a surprise on X, the day the pilot went public6.
Yijia Shao@EchoShao8899It’s such a rare opportunity to study human–AI collaboration at scale, given the constraints around data access and privacy. Huge thanks to @AnthropicAI for making this possible through the Anthropic Insights system, and for their support throughout the project.
We started this… https://t.co/nL8bJ4N8eE pic.twitter.com/25NrjcedFc
Diyi Yang, the lab's senior author, reads the same numbers differently. She sees proof that people check Claude's work instead of accepting it outright. "Counter to many dire predictions, the vast majority of users of AI are not treating it as a 'free lunch' but rather as a tool that presents new ideas and ways of thinking, pushes back on the user's knowledge and reasoning, and collaborates in ways that humans find useful in improving their own work," she told Stanford HAI3.
Her worry sits one layer down. She and Shao both flag a split still forming. Some people treat AI as practice. Others treat it as a stand-in for their own judgment3.
The Case Against Trusting One Company's Grading
Every number above came from a tool called Anthropic Insights1. Researchers wrote a question in plain English. Claude itself read every sampled chat and sorted it into a category.
Davi Ottenheimer runs the security firm flyingpenguin. He raised that exact objection in April, about an earlier Anthropic paper built the same way. "A subject group talking to the product, about the product, scored by the product, reported by the product," he wrote4.
Stanford's own authors stand outside Anthropic's payroll, and Anthropic gave up its editorial say before the work began. Its contract limited review to privacy, policy violations, confidential information and research accuracy, "and the researchers are free to publish their results even if they are inconvenient for Anthropic"1.
The sample still carries real limits. It covers only Free, Pro and Max chat, so Claude Code and the agentic Claude Cowork product stayed outside the sample2. Every percentage above describes paying and free chat users in one seven-week window, a slice of the whole economy.
What to Do
If you manage people who use Claude for client work, ask what tier their last ten chats would fall into. A team spending most of its time in the consequential and high-stakes tiers needs a review step before anything ships. That is the same reason a junior associate's memo gets a second read.
Hiring for AI fluency calls for a different test. Weigh judgment over typing speed. Stanford's own numbers point the risk toward scrutiny.
Does the person check Claude's answer before using it? That matters more than whether they can write a prompt23.
By the Numbers
- 249,834 Claude.ai chats made up Stanford's sample, drawn from April and May 202612.
- Combined, 56 percent of the classifiable chats carried consequential or high-stakes work2.
- The consequential tier alone took 44 percent of the sample, and the high-stakes tier took 12 percent2.
- Legal and financial guidance scored 2.14 on Stanford's criticality index, the highest of nine domains measured2.
- Academic coursework scored lowest, at 1.062.
- Nearly three-quarters of chats had a person setting the direction while Claude helped1.
- Teaching showed up in 67 percent of chats, where a person actively taught Claude about their own situation2.
What to Watch
Watch whether Nature Human Behavior publishes Stanford's peer-reviewed version. That would put outside reviewers on every number here3. A second signal sits with Oxford's Human Information Processing Lab and METR. Publishing their own readings of the same pilot would answer Ottenheimer's complaint directly145.
Sources
- Anthropic, "Enabling Independent Research on How People Use Claude," Anthropic, Aug. 26, 2026, https://www.anthropic.com/research/enabling-independent-research
- Yijia Shao, Dora Zhao, Vishakh Padmakumar, Jennifer Wang and Diyi Yang, "Human-AI Collaboration at Scale: Task Criticality, Agency, and Friction Across 250,000 Conversations," Stanford SALT Lab, Aug. 26, 2026, https://www.alphaxiv.org/abs/2608.human-ai-collaboration-at-scale
- Andrew Myers, "Humans and AI Engage in Meaningful Collaborations," Stanford HAI, Oct. 12, 2026, https://hai.stanford.edu/news/humans-and-ai-engage-in-meaningful-collaborations
- Davi Ottenheimer, "Anthropic Says Claude Scores Itself Best Claude," flyingpenguin, April 23, 2026, https://www.flyingpenguin.com/anthropic-says-claude-scores-itself-best-claude/
- Emma Thompson, "Anthropic Opens Real Claude Usage Data to Outside Researchers for First Time," EdTech Innovation Hub, Sept. 7, 2026, https://www.edtechinnovationhub.com/news/anthropic-opens-real-claude-usage-data-to-outside-researchers-for-first-time
- Yijia Shao (@EchoShao8899), "It's such a rare opportunity to study human-AI collaboration at scale," X, Aug. 26, 2026, https://x.com/EchoShao8899/status/2092674746202923504