Mercor built a new accounting test this summer. Claude Opus 5 scored 100 percent on it2. Twelve licensed accountants took the same test. They averaged 37 percent.
A separate study asked 502 investment bankers to grade AI-built deal work. The bankers rated every output too rough to send a client, exactly as the agent wrote it7.
Four benchmarks built in 2026 now tell the same story. AI agents ace the steps inside a finance job. Finishing the whole job themselves is a different test. Most of them fail it.
A Perfect Score, on a Narrow Test
Mercor built the test with Ramp, an expense-software company2. Twenty simple, month-end closing tasks made up the whole exam. Claude Opus 5 solved every one correctly, each in under ten minutes.
Twelve licensed accountants took the identical test. Six had worked at a Big Four firm4. Their scores ranged from zero to roughly 90 percent. The group averaged 37 percent2.
A single task cost Claude Opus 5 about 21 cents to run. The same task cost a human accountant roughly $10.35, counting the median accountant's pay and the time the task took2.
Aden Barton, a researcher at Mercor, wrote up the results himself. "These data do suggest substantial changes, and productivity gains, in accounting in the coming years," he wrote2.
The Whole Job Looks Different
That 20-task exam covered only part of the job. Mercor and Ramp built a harder benchmark too: APEX-Accounting, 160 real accounting tasks spread across 10 simulated companies1.
Each task carries its own grading sheet, written by accountants1. A model earns partial credit for every line it gets right. That credit adds up fast: the top model, Claude-Fable-5, cleared 56.4 percent of its graded lines on average.
Finishing the whole task is a stricter test1. A model has to get every single criterion right, on all eight tries in a row. GPT-5.6-Sol topped that stricter test at 2.6 percent. Claude-Fable-5, despite its stronger line score, managed only 1.9 percent of tasks finished start to end.
Claude-Fable-5 Clears 56 Percent of Steps, Finishes 2 Percent of Whole Jobs
Average share of graded lines met, against correct finishes on all 8 tries, APEX-Accounting's 160 tasks; higher is better
The blue dot is how often a model satisfied its graded lines on average. The magenta dot is how often it finished the same task correctly on every one of 8 tries. The rod is the gap between passing a step and finishing the job.
Source: arXiv [1]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Average lines met | Passed all 8 tries |
|---|---|---|
| Claude-Fable-5 (Max) | 56.4% | 1.9% |
| GPT-5.6-Sol (Max+Pro) | 51.5% | 2.6% |
Mercor@mercorIt's month close. AI models can pass the CPA exam, but do you trust them to close the books?
Introducing APEX-Accounting, built in collaboration with @tryramp and @RampLabs, to test whether frontier AI can do real accounting work.
It’s a new benchmark comprising 160 tasks… pic.twitter.com/yfES7dEpkU
Every model Mercor tested left 93 of the 160 tasks unsolved1. By October, a newer model, Claude Opus 5, raised that score to 54.5 percent, CellCog's Nitish Garg found5. That is a real gain. The gap to its perfect score on the narrower test stays wide.
The gap between a passed step and a finished job is this story's real subject.
Zero Percent Client-Ready
Banking tells a sharper version of the same story6. Handshake AI built BankerToolBench with help from 502 investment bankers. Of them, 172 designed the hundred tasks themselves, logging more than 5,700 work hours.
Nine frontier models took the test. GPT-5.4 led the field, passing 58.1 percent of its roughly 150 graded criteria per task, on average6. Only 16 percent of its answers counted as an acceptable starting point. Just 13 percent held up on three separate tries in a row.
Two percent of its work cleared every critical criterion at once. Every one of the nine models' outputs missed a "client-ready" grade from the bankers who reviewed it6.
Banking's Benchmark Scores 58 on Steps, 0 on the Finished Deliverable
Best model's pass rate on six assignment types, from one step to a finished deliverable, 2026
Each bar is the best score any tested model reached on that kind of assignment. The bars shrink as the task moves from one simplified step toward a whole deliverable a banker or accountant could send untouched.
The accounting and banking bars come from two different benchmarks, built by two different companies; read each bar's own drop, since the two benchmarks measure different kinds of work.
Sources: arXiv [1]; Mercor [2]; CellCog [5]; arXiv [6]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Value | Note |
|---|---|---|
| One simplified accounting task | 100% | Claude Opus 5, Mercor's 20-task exam |
| Full accounting test, average | 54.5% | Claude Opus 5, APEX-Accounting's 160 tasks, average criteria met |
| Full accounting test, every try | 2.6% | GPT-5.6-Sol, correct on all 8 tries |
| Banking rubric, average | 58.1% | GPT-5.4, BankerToolBench's roughly 150 criteria per task |
| Banking draft, usable start | 16% | GPT-5.4, Pass at 1 try |
| Banking deliverable, client-ready | 0% | Best of nine models, graded by 502 bankers |
Claude Opus 4.6 built polished-looking spreadsheets with a hidden flaw6. It typed fixed numbers into cells where a working formula belonged. The sheet broke the moment anyone tried to update it.
Why Passing a Step Differs From Finishing a Job
Three separate teams of researchers reached the same limit.
A second, different study put AI agents through realistic banking support chats. Each chat required a search through 700 linked policy files before the agent could act8. Top models using their deepest reasoning mode passed only about 25.5 percent of those chats. The pass rate fell even further the more times the same agent tried.
Mercor built a third test too, scoring real Wall Street deal work with help from former Morgan Stanley and Citigroup bankers10. Across its three job types, every agent missed the bar on at least 40 percent of its runs. In the banking slice alone, the top score was 27.3 percent.
The APEX-Accounting researchers named the actual failure point themselves1. Models find the right documents and the right numbers almost every time. They slip applying judgment across each linked step in a row. A real close, or a real deal, always asks for exactly that chain of steps.
Both Sides Read the Same Numbers Differently
Gabriel Stengel reads these numbers as a market already here9. He runs Rogo, a finance AI startup. It raised $160 million in April, near a $2 billion value. Rogo's agent, Felix, already does research and drafts for more than 35,000 bankers at over 250 firms, Rogo says.
"Over the last few decades, it's also become an industry where some of the best people spend their time assembling decks and rebuilding models instead of talking to clients," Stengel said9. "A lot of the analytical work is done by a 21-year-old in tools from 40 years ago at 2 a.m.," he added. He wants Felix to take back that time.
Nitish Garg runs CellCog, an accounting software firm. He read the same study with more doubt5. Garg pointed at the gap between the narrow test's perfect score and the full test's far lower one as the real finding.
"For a small business, the practical version is simpler: the reconciliations, variance tables and journal entries in these four tasks are the work an AI can already do fast and carefully, and the parts that need context about your business are the parts you still own," Garg wrote5. The steps and the job, in his reading, stay two different things.
What to do
Treat a "passes the benchmark" headline as one fact, checked against the rest of the record. Ask which score the headline means: the partial-credit average, or the full-job pass rate16. The two numbers in this piece differ by a factor of ten or more.
Pilot an agent on narrow steps first: matched accounts, drafted entries, document checks, the work Garg says AI already does well5. Keep a person checking the point where separate steps link into one job, the point every test here still gets wrong.
By the numbers
- Claude Opus 5 scored 100 percent on Mercor's 20-task simplified accounting exam, each task solved in under 10 minutes2.
- Licensed CPAs averaged 37 percent on that same exam, across 12 accountants tested2.
- The best score any model reached finishing an APEX-Accounting task correctly on all eight tries was 2.6 percent1.
- Ninety-three accounting tasks, out of 160, went unsolved by every model in Mercor's full benchmark1.
- GPT-5.6-Sol averaged 51.5 percent on APEX-Accounting's own grading lines, before the stricter whole-job test1.
- 0 percent: the share of nine AI models' banking deliverables that 502 bankers called client-ready6.
- BankerToolBench's top model averaged 58.1 percent on its graded lines, before bankers reviewed the finished work6.
- Top reasoning models passed about 25.5 percent of a banking support test built from 700 linked policy files8.
- APEX-Agents' banking tasks, built with former Morgan Stanley and Citigroup bankers, topped out at 27.3 percent10.
What to watch
Watch Mercor's next APEX-Accounting update, expected as newer models reach its leaderboard. A climbing Pass^8 score, beyond a climbing partial-credit score, would mark real progress on whole jobs1. Watch, too, whether Handshake AI publishes a second round of BankerToolBench with any client-ready score to report6. Rogo's own usage numbers, if the company publishes them again, will show whether banks trust Felix with more than research and drafts9.
Sources
- Julien Benchek et al., "APEX-Accounting," arXiv, July 30, 2026, https://arxiv.org/abs/2607.27189
- Aden Barton, "AI vs. Accountants: Frontier Models Outscore CPAs," Mercor, Oct. 1, 2026, https://www.mercor.com/blog/human-baselines-for-benchmarks-ai-now-outperforms-junior-accountants/
- Manuel Uth, "AI beats licensed accountants on speed and accuracy, but still can't close the books without supervision," The Decoder, Oct. 2, 2026, https://the-decoder.com/ai-beats-licensed-accountants-on-speed-and-accuracy-but-still-cant-close-the-books-without-supervision/
- Courtney Vien, "AI beats out CPAs at month-end close tasks," CFO Brew, Oct. 9, 2026, https://www.cfobrew.com/stories/ai-beats-out-cpas-at-month-end-close-tasks
- Nitish Garg, "AI vs Accountants: Mercor's CPA Study, Read Closely," CellCog, Oct. 3, 2026, https://cellcog.ai/blog/ai-vs-accountants-mercor-study/
- Elaine Lau et al., "BankerToolBench: Evaluating AI Agents in End-to-End Investment Banking Workflows," arXiv, April 13, 2026, https://arxiv.org/abs/2604.11304
- Jonathan Kemper, "500 investment bankers review AI outputs and find none ready for client delivery," The Decoder, April 26, 2026, https://the-decoder.com/500-investment-bankers-review-ai-outputs-and-find-none-ready-for-client-delivery/
- Quan Shi et al., "τ-Knowledge: Evaluating Conversational Agents over Unstructured Knowledge," arXiv, March 4, 2026, https://arxiv.org/abs/2603.04370
- PYMNTS Staff, "Rogo Raises $160 Million to Lessen Wall Street Workloads," PYMNTS, April 29, 2026, https://www.pymnts.com/news/investment-tracker/2026/rogo-raises-160-million-to-lessen-wall-street-workloads/
- Bertie Vidgen et al., "APEX-Agents," arXiv, Feb. 23, 2026, https://arxiv.org/abs/2601.14242
