Artificial Analysis grades AI models for a living. On Sept. 7, it swapped two of its ten tests. It also raised the share of secret questions, from 40 percent to 45 percent1. The company calls the new version v4.3.
Claude Fable 5.1 scored 66 on Sept. 1. That was the highest mark Artificial Analysis had ever recorded2. Six days later, under the new version, the same model scored 531. The model itself stayed exactly the same in between.
What the Intelligence Index Actually Grades
Artificial Analysis runs ten separate tests on each model, then rolls the results into one number4. Agents tasks count for 30 percent of that number, coding for 20 percent, general knowledge for 30 percent, and scientific reasoning for 20 percent. A model that writes flawless code but forgets facts can still land in the middle of the board.
Some of those ten tests use a private test set. A vendor like Zapier keeps the questions and the correct answers locked away, so every lab meets them cold, only during grading1. Artificial Analysis grades every model against the same hidden questions, then publishes only the score.
What Changed on Sept. 7
Three things moved in the update. Terminal-Bench checks whether an agent can finish a job from a command line, and it jumped to a harder set of 66 tasks1.
AutomationBench-AA replaced an older banking simulation at the same 5 percent weight. Artificial Analysis built it with Zapier, on 657 real business workflows across finance, sales, and five other departments16. Because that new benchmark uses a held-out task set, the private share of the whole index climbed to 45 percent.
Zapier sees over two billion automated tasks run through its platform every month6. Anna Marie Clifton leads Zapier's agents and AI team. She built the underlying test for a plainer reason than ranking models. "Workflow success is the missing yardstick for enterprise AI, and building a credible benchmark requires two things: real workflow patterns and real tool complexity," she wrote when Zapier first released it.
Why a September Score and an October Score Disagree
Four models sat on both the old board and the new one. Every one of them fell by double digits. Claude Fable 5.1 dropped from 66 to 53, and Opus 5 slid from 63 to 51. Its smaller sibling, Claude Fable 5, fell from 62 to 50, while OpenAI's GPT-5.6 Sol dropped from 61 to 4721.
Every drop lands between 12 and 14 points, a tight band for four unrelated models.
Four Models Lose 12 to 14 Points When Artificial Analysis Moves to Index v4.3
Intelligence Index scores for the same models at max effort, Sept. 1 versus v4.3; higher is better
Each row is one model: the blue dot is its score reported Sept. 1, the magenta dot is its score under v4.3, and the rod measures the points the new test cost, apart from any change in the model itself.
The Sept. 1 scores predate both the Sept. 4 and the Sept. 7 methodology updates, so each rod reflects two rounds of test changes together.
Sources: Artificial Analysis [1]; Artificial Analysis [2]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Scored Sept. 1 | Scored on v4.3 |
|---|---|---|
| Claude Fable 5.1 (max) | 66 | 53 |
| Claude Opus 5 (max) | 63 | 51 |
| Claude Fable 5 (max) | 62 | 50 |
| GPT-5.6 Sol (max) | 61 | 47 |
That pattern describes the test changing, while the model held still. Anthropic ran Claude Fable 5.1 with a safety fallback that quietly reroutes a small share of replies to an older Claude model. Artificial Analysis tested it the same way both times2. Its exam just got harder.
Artificial Analysis has since patched the index twice more, refreshing its judge panel once and re-anchoring two score scales a second time4. Its own methodology page offers a reassurance worth testing against the data: "Whilst Elo scores shift, rank ordering is largely preserved". The four-model drop above backs that claim. Every model lost roughly the same amount, so the order barely moved even as the scale did.
A Harder Test, or a Confusing One
The rebuild followed a dispute over one model. GPT-6 Astra launched Sept. 3 to strong marks elsewhere. Epoch AI ranked it first among 267 models, and a separate reasoning test showed a sizable jump too8. Artificial Analysis's own index, by contrast, "scored Astra just on par with its predecessor," Matthias Bastian reported for The Decoder.
A smaller update followed within days. The version behind this story followed that within three more.
Clifton's case for the rebuild rests on what the new test actually measures. Zapier "initially built AutomationBench for internal use, to help evaluate which models to deploy across Zapier, and existing benchmarks didn't measure the element we cared about," she wrote, meaning whether an agent finished the job6. On that measure, GPT-6 Astra completed 68.5 percent of AutomationBench-AA's objectives. Grok 4.6 followed at 66.7 percent, and GLM-5.3 at 62.2 percent1.
Camila Duarte writes on AI infrastructure for the routing vendor Nexforce. She makes the opposite case, from the buyer's chair. "A score without the index version beside it is a number without a ruler, and that is how it circulates in sales proposals, in procurement spreadsheets and in committee slides," she wrote days after the new version shipped5. A procurement memo built on a bare Intelligence Index score ages the moment the index changes again.
A third voice complicates both sides. Ward Ed writes a leaderboard-auditing series for DEV Community. He flags noise even inside one version of Terminal-Bench. "A private set of 40 tasks is a good instrument for detecting leakage and a blunt one for ranking models that differ by a few points," he wrote of the index's private sections generally7.
GPT-6 Astra's 19.2-point lead over GPT-5.6 Sol survives that noise easily. A gap of two or three points folds into that same noise. That is roughly the gap separating half of today's top ten.
Today's Board, Read on Its Own Terms
Claude Opus 5.5 leads the live board at 583. Anthropic shipped it three weeks after the rebuild, and Artificial Analysis scored it only on the new test. Three others tie for second, at 56: Claude Sonnet 5.5, Claude Opus 5.5 at a lower effort setting, and Meta's Muse Wonder Full. A notch down, Claude Opus 5.5 at a third effort level scores 54.
Gemini 4 Argon, GPT-6 Astra, and Claude Fable 5.1 then tie at 533. Meta's smaller Muse Wonder rounds out the top ten at 52.
Claude Opus 5.5 Leads Today's Index at 58, With Four Labs in the Top Ten
Artificial Analysis Intelligence Index v4.3.2, every model and reasoning effort scoring 52 or higher; higher is better
Each bar is one model and reasoning effort; a taller bar means a smarter score on this index, apart from its price.
Source: Artificial Analysis [3]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Claude Opus 5.5 (max) | 58 |
| Claude Sonnet 5.5 (max) | 56 |
| Claude Opus 5.5 (xhigh) | 56 |
| Muse Wonder Full (max) | 56 |
| Claude Opus 5.5 (high) | 54 |
| Claude Fable 5.1 (max) | 53 |
| Claude Fable 5.1 (xhigh) | 53 |
| GPT-6 Astra (max) | 53 |
| Gemini 4 Argon (high) | 53 |
| Muse Wonder (max) | 52 |
Notice who is missing from that top tier. The Sept. 1 version of Claude Opus 5 and Claude Fable 5 anchored the dumbbell chart above. They still sit in the high 40s and low 50s today. Anthropic shipped its newer siblings only after the test had already changed once, which makes this a matter of timing.
What to do
Write the index version next to every score you repeat. Do it the way you would write the date beside a currency figure. A score paired with its version tells a colleague more than a bare number ever could.
Before you act on a close call, rerun both models on examples from your own workload. A close call means a model that sits two or three points above another. Ward Ed's point about sample noise applies to your decision as much as it applies to the leaderboard7.
Treat a gap of ten points or more as real. That is roughly what four unrelated models lost, purely from the test changing. Call anything smaller a coin flip, until you check it yourself.
By the numbers
- 45 percent: private, held-out weight in the new index, up from 40 percent before it1.
- 657: simulated business workflows inside AutomationBench-AA, built with Zapier across six departments1.
- 66: terminal tasks in Terminal-Bench 4.0, each run three times per model1.
- 13: average points lost by four carryover models between Sept. 1 and the Sept. 7 rebuild21.
- 58: the top score on today's board, held by Claude Opus 5.5 at its highest effort setting3.
- 19.2: percentage-point gap between GPT-6 Astra and GPT-5.6 Sol on the new Terminal-Bench1.
What to watch
Artificial Analysis says it is still rolling out a larger future index, in stages1. Watch for the next version number, because the next reset will likely repeat this one.
Watch AutomationBench-AA's second number, Tasks Completed, alongside its headline Score. The Score hands out partial credit; Tasks Completed only counts a workflow a model finished clean, every rule intact1. That second figure is closer to what a manager actually wants from an agent.
Sources
- Artificial Analysis, "Announcing the Artificial Analysis Intelligence Index v4.3," Sept. 7, 2026, https://artificialanalysis.ai/articles/artificial-analysis-intelligence-index-v4-3
- Artificial Analysis, "Claude Fable 5.1 tops the Artificial Analysis Intelligence Index but costs 20% more per task than Fable 5 despite a 75% cache read price cut," Sept. 1, 2026, https://artificialanalysis.ai/articles/claude-fable-5-1
- Artificial Analysis, LLM Leaderboard, accessed Oct. 21, 2026, https://artificialanalysis.ai/leaderboards/models
- Artificial Analysis, "Intelligence Benchmarking Methodology," accessed Oct. 20, 2026, https://artificialanalysis.ai/methodology/intelligence-benchmarking
- Camila Duarte, "Artificial Analysis v4.3: the new AI index test arrives," Nexforce, Sept. 14, 2026, https://nexforce.ai/en/blog/artificial-analysis-intelligence-index-v4-3-automationbench
- Anna Marie Clifton, "Introducing AutomationBench," Zapier, Apr. 20, 2026, https://zapier.com/blog/introducing-automationbench/
- Ward Ed, "Do 66 Terminal Tasks and a 45% Private Weight Settle AI Rankings? Reading Artificial Analysis Index v4.3," DEV Community, Oct. 20, 2026, https://dev.to/ward_ed_6b5e6aa8ded94a987/do-66-terminal-tasks-and-a-45-private-weight-settle-ai-rankings-reading-artificial-analysis-index-1872
- Matthias Bastian, "Artificial Analysis overhauls its Intelligence Index after GPT-6 Astra scoring drew skepticism," The Decoder, Sept. 5, 2026, https://the-decoder.com/artificial-analysis-overhauls-its-intelligence-index-after-gpt-6-astra-scoring-drew-skepticism/