Skip to content
Technology
ResearchAnalysis

AI's Hardest Tests Crack Within a Year

Five hard AI tests climb from single digits to near the ceiling within about a year of launch, replacing each other faster than ever.

6 min read · 1,342 words · 22 sources
A finish line banner stretches across a trail during the Routeburn Classic race
“The hardest math test built for machines lasted fourteen months. Does a benchmark that breaks within a single product cycle still measure anything?”

FrontierMath Tier 4 was built to defeat machines for years. It lasted fourteen months.

Epoch AI, the research group that built it, announced on Sept. 10, 2026 that AI had solved every problem in the tier1. OpenAI's GPT-6 Astra solved the last one, a problem written by the mathematician Jay Pantone. The tier had launched just 14 months earlier, in July 2025. Back then, the field's best models solved only three of its 50 problems between them2.

That pattern keeps repeating. Ai Lately tracked five hard AI benchmarks from their first scores to the point their builders called them full. Four of the five reached that point within about a year of launch.

One test, built the same way, still has room to climb. The gap between the four and the one is the real story here.

Five Tests, One Pattern

Start with the oldest. MMLU asked AI models to answer college-level quiz questions across 57 subjects, a test of broad knowledge rather than deep reasoning. Dan Hendrycks and his co-authors released it in September 202010.

GPT-3, then the best model around, barely beat guessing at random. Within a few years, that gap had nearly closed.

By May 2024, frontier models had closed most of the MMLU gap, and Wenhu Chen built a tougher successor. "The current models already hit the ceiling? It's time to upgrade MMLU!" he wrote on X11.

His new test, MMLU-Pro, showed the gap clearly. GPT-4o beat GPT-4-turbo by nine points on the harder test, but by only two on the original MMLU11.

The Newest AI Tests Cut Their Lifespan From 4 Years to 1

Launch and ceiling dates for five AI benchmarks, 2020 to 2026

Each pin marks one benchmark's launch or the date its builders called it full. The gap between the two pins shrinks from years, for the oldest tests, to about a year for the two newest.

Data through Nov. 17, 2026.

Sources: Epoch AI on X [1]; Epoch AI on X [3]; ARC Prize on X [8]; ARC Prize [9]; arXiv [10]; Wenhu Chen on X [11]; Epoch AI on X [13]; arXiv [14]; Vals AI [16]. Chart by Kelly Dennis.

The numbers behind this chart
DateEvent
Sept. 2020MMLU launches
March 2021MATH launches
Nov. 2023GPQA Diamond launches
May 2024MMLU-Pro replaces it. Built because MMLU had hit its ceiling
March 2025ARC-AGI-2 launches
April 2025MATH Level 5 saturates
July 2025FrontierMath Tier 4. Launches with a single-digit top score
April 22, 2026ARC-AGI-2 clears 85%
Sept. 2026GPQA Diamond retires
Sept. 10, 2026FrontierMath Tier 4 full

The MATH benchmark's hardest tier followed the same arc. Hendrycks and his colleagues published the 12,500-problem set in March 202112. By April 2025, Epoch AI posted that Level 5, the hardest slice, had become "quite saturated, with both o3 and o4-mini getting 98% accuracy"13.

GPQA Diamond moved faster still. Its questions came from working scientists rather than puzzle writers, and few outsiders could answer them. Researchers led by David Rein published it in November 2023, when GPT-4 answered 39 percent of its graduate-level science questions correctly14.

OpenAI's o1 beat human PhDs on the same test by September 2024, scoring 77.3 percent15. By September 2026, Vals AI's leaderboard put one model at 95.45 percent and declared the test saturated, retiring it from new write-ups16.

A graduate chemistry laboratory with precise, complex equipment
GPQA Diamond's questions are written at this level: graduate-level science, like the work done in labs such as this one.JohnRH4 / Wikimedia Commons, CC BY-SA 2.0

FrontierMath Tier 4 compressed the timeline again. Epoch AI built it with research mathematicians to push past the limits of the first three tiers3. At launch, the field's best systems solved three of the fifty problems, about 6 percent2.

After a correction that June, the top score on the tier reached 76 percent, and Epoch called it "approaching saturation"45. Ten weeks later, every one of its problems had fallen1.

Epoch AI@EpochAIResearch

Every FrontierMath Tier 4 problem has now been solved by AI, with GPT-6 Astra solving the last problem standing. Mathematicians often commented that AI found unintended shortcuts when solving their Tier 4 problems. Not so for this last one, which was created by Jay Pantone. pic.twitter.com/ieEZaomhaE

Epoch AI's chart showing every FrontierMath Tier 4 problem solved
Open on X (opens a new tab)

Epoch AI posted this chart on Sept. 10, 2026, the day it said AI had cleared the last of Tier 4's 50 research-level math problems.Post by Epoch AI (@EpochAIResearch) on X, Sept. 10, 2026

ARC-AGI-2 ran on the same clock. The ARC Prize Foundation announced the test on X in March 2025 as "an unsaturated frontier AGI benchmark that challenges AI reasoning systems." Reasoning models then scored under 4 percent8.

OpenAI's GPT-5.5 crossed the benchmark's 85 percent grand-prize bar 13 months later, on April 22, 20269. By Nov. 17, 2026, OpenAI's GPT-6.2 Astra sat well clear of that bar on the public leaderboard.

Three Hard AI Tests Climb From Single Digits to 97 Percent

Best public score by date for ARC-AGI-2, FrontierMath Tier 4 and GPQA Diamond, 2023 to 2026

Each line follows one test's best known score over time, higher is better. All three climb toward the top right as newer models post stronger results, and two of the three cross 95 percent within about a year and a half of launch.

Each test launched on its own date, so the three lines start at different points on the calendar. The climb itself is the comparison, apart from the calendar gap.

Data through Nov. 17, 2026.

Sources: Import AI [2]; Epoch AI on X [5]; OpenAI [6]; ARC Prize on X [8]; ARC Prize [9]; arXiv [14]; OpenAI [15]; Vals AI [16]. Chart by Kelly Dennis.

The numbers behind this chart
PeriodGPQA DiamondARC-AGI-2FrontierMath Tier 4
Nov. 202339%
Sept. 202477.3%
Mar. 20254%
Jul. 20256%
Jun. 202676%
Apr. 202685%
Sept. 202695.45%
Sept. 202697.6%
Nov. 202695.8%
Portrait of Francois Chollet, the Google researcher who created ARC-AGI
Francois Chollet, the Google researcher who built the original ARC-AGI test, co-founded the ARC Prize Foundation behind ARC-AGI-2.Kaicarver / Wikimedia Commons, CC BY-SA 4.0

Why the Climb Compressed

Credit the labs' own math homework. OpenAI said GPT-6 Astra's internal score on FrontierMath Tier 4 started at 17 percent and ended at 97.6 percent before shipping6. That climb happened behind closed doors, far from any leaderboard. Labs now treat reasoning and math as a named training goal, and they test against the hardest public benchmarks while they do it.

Mathematicians feel the pull too. Terence Tao, the UCLA mathematician, met OpenAI's James Donovan and Mark Chen at a math-and-AI workshop on March 4, 202617. That kind of meeting barely existed three years earlier. Research mathematicians now help write the very tests meant to outlast their students' models.

Terence Tao speaks with OpenAI researchers at a math and AI workshop
Terence Tao, the UCLA mathematician, met OpenAI's James Donovan and Mark Chen at a math-and-AI workshop on March 4, 2026, in Los Angeles.Institute for Pure & Applied Mathematics / Wikimedia Commons, CC BY 4.0

The Case for Climbing Higher

Epoch AI, the researchers behind FrontierMath, see this as proof the work matters. It is also a reason to build the next test. The group said so the day it called Tier 4 nearly solved. "We believe the future of math benchmarking lies in open problems drawn from real research, like those we've collected in FrontierMath: Open Problems"5.

The group makes the same case more broadly. A test that stays easy stops teaching anyone anything. "Most AI benchmarks share a common flaw: they saturate too quickly to study long-run trends," Epoch AI wrote in December 202518. The group was pitching a project that stitches many tests into one long scale.

That logic shaped ARC-AGI-2 from day one. The ARC Prize Foundation priced its grand prize to a bar that only genuine reasoning could clear78.

Skeptics Push Back

Some researchers reject the whole chase. Angela Aristidou is a professor at University College London and a Stanford fellow who studies how companies actually use AI. She argues the scores measure the wrong thing.

Aristidou studied radiology tools with strong benchmark scores that slowed doctors down once hospitals installed them. Her argument turns on where a model works, apart from how hard its test reads. She laid out the finding in MIT Technology Review in March 202619. "What appeared as a productivity-enhancing AI tool when tested in a vacuum introduced delays in practice."

Her verdict is blunt. "AI is almost never used in the way it is benchmarked," she wrote in the same piece.

Humanity's Last Exam backs a softer version of her point. Dan Hendrycks, whose Center for AI Safety built the test with Scale AI, announced it with a warning. "State-of-the-art AIs get <10% accuracy and are highly overconfident," he wrote20. Almost two years later, the best public score is still 38.3 percent21.

A 2026 study of 60 widely cited benchmarks, led by Mubashara Akhtar, found that nearly half had already saturated. Expert-built tests resisted the fast climb longer than crowdsourced ones22.

The speed-up is real. One carefully built test still resists it.

What to Do

Before trusting a vendor's benchmark claim, ask two questions this piece's own tracking answers. First, how old is the test? A score on a benchmark past its second birthday may already sit near a ceiling every rival has also reached.

That tells a buyer little about who actually leads. Age alone can hide the real gap between rivals.

Second, who built it, and how? Expert-curated tests such as Humanity's Last Exam held their difficulty far longer than quizzes built by crowdsourcing questions online22. Check an independent tracker such as Epoch AI or Vals AI before repeating a lab's own number.

By the Numbers

  • GPQA Diamond opened at 39 percent in 2023 and crossed 95 percent by 20261416.
  • FrontierMath Tier 4 went from launch to every problem solved in 14 months, July 2025 to Sept. 10, 202613.
  • ARC-AGI-2's 85 percent grand-prize bar fell to GPT-5.5 in 13 months, by April 22, 202689.
  • GPT-3 scored 43.9 percent on MMLU in 2020, the test that needed a harder successor within four years10.
  • Humanity's Last Exam's best public score sits at 38.3 percent, the one test here still far from its ceiling21.
  • Nearly half of 60 widely used benchmarks had saturated, a 2026 study found, with the share rising as a test ages22.
  • OpenAI's GPT-6.2 Astra topped ARC-AGI-2's public leaderboard at 95.8 percent as of Nov. 17, 2026, well clear of the grand-prize bar9.

What to Watch

Watch Dec. 4, 2026, when the ARC Prize Foundation posts its 2026 results. That update will confirm whether a model cleared the 85 percent bar on ARC-AGI-2's private evaluation set, beyond its public leaderboard9.

FrontierMath: Open Problems, the harder successor Epoch AI is already building, will show whether its difficulty holds against the same labs5. Humanity's Last Exam has held near the 38 percent mark for most of 2026. A climb past it would be the clearest sign yet that expert curation buys a test more time2122.

Sources

  1. Epoch AI, "Every FrontierMath Tier 4 problem has now been solved by AI," X, Sept. 10, 2026, https://x.com/EpochAIResearch/status/2098103831502708864
  2. Jack Clark, "Import AI 420: Prisoner's Dilemma AI; FrontierMath Tier 4; and how to regulate AI companies," Import AI, July 14, 2025, https://jack-clark.net/2025/07/14/import-ai-420-prisoner-dilemma-ai-frontiermath-tier-4-and-how-to-regulate-ai-companies/
  3. Epoch AI, "Introducing FrontierMath Tier 4," X, July 11, 2025, https://x.com/EpochAIResearch/status/1943744462972215305
  4. Epoch AI, "FrontierMath: Tiers 1-4," Epoch AI, June 12, 2026, https://epoch.ai/frontiermath/tiers-1-4
  5. Epoch AI, "FrontierMath: Tiers 1-4 is now approaching saturation," X, June 12, 2026, https://x.com/EpochAIResearch/status/2065488238543036528
  6. OpenAI, "GPT-6 Astra: A new generation of intelligence," OpenAI, Sept. 8, 2026, https://openai.com/index/gpt-6-astra/
  7. ARC Prize Foundation, "Announcing ARC-AGI-2 and ARC Prize 2025," ARC Prize, March 24, 2025, https://arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025
  8. ARC Prize, "Today we are announcing ARC-AGI-2, an unsaturated frontier AGI benchmark," X, March 24, 2025, https://x.com/arcprize/status/1904269307284230593
  9. ARC Prize Foundation, "ARC Prize Leaderboard," ARC Prize, Nov. 17, 2026, https://arcprize.org/leaderboard
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, Jacob Steinhardt, "Measuring Massive Multitask Language Understanding," arXiv, Sept. 7, 2020, https://arxiv.org/abs/2009.03300
  11. Wenhu Chen, "Tired of MMLU? The current models already hit the ceiling?," X, May 15, 2024, https://x.com/WenhuChen/status/1790597967319007564
  12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, Jacob Steinhardt, "Measuring Mathematical Problem Solving With the MATH Dataset," arXiv, March 5, 2021, https://arxiv.org/abs/2103.03874
  13. Epoch AI, "MATH Level 5 ... now appears to be quite saturated," X, April 18, 2025, https://x.com/EpochAIResearch/status/1913379488286601533
  14. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, Samuel R. Bowman, "GPQA: A Graduate-Level Google-Proof Q&A Benchmark," arXiv, Nov. 20, 2023, https://arxiv.org/abs/2311.12022
  15. OpenAI, "Learning to reason with LLMs," OpenAI, Sept. 12, 2024, https://openai.com/index/learning-to-reason-with-llms/
  16. Vals AI, "GPQA Diamond Leaderboard and Methodology," Vals AI, Sept. 1, 2026, https://www.vals.ai/benchmarks/gpqa
  17. Institute for Pure & Applied Mathematics, "Terence Tao at IPAM's Fireside Chat 2026," Wikimedia Commons, March 4, 2026, https://commons.wikimedia.org/wiki/File:Terence_Tao_at_IPAM%27s_Fireside_Chat_2026_(50m16s).jpg
  18. Epoch AI, "Most AI benchmarks share a common flaw: they saturate too quickly to study long-run trends," X, Dec. 3, 2025, https://x.com/EpochAIResearch/status/1996248575400132794
  19. Angela Aristidou, "AI benchmarks are broken. Here's what we need instead.," MIT Technology Review, March 31, 2026, https://www.technologyreview.com/2026/03/31/1134833/ai-benchmarks-are-broken-heres-what-we-need-instead/
  20. Dan Hendrycks, "We're releasing Humanity's Last Exam," X, Jan. 23, 2025, https://x.com/hendrycks/status/1882433928407241155
  21. Center for AI Safety, Scale AI, "Humanity's Last Exam Leaderboard," Center for AI Safety, Jan. 28, 2026, https://agi.safe.ai/
  22. Mubashara Akhtar, Anka Reuel, Prajna Soni, et al., "When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation," ICML / arXiv, Aug. 6, 2026, https://arxiv.org/abs/2602.16763

Header photo: Grapeman4 / Wikimedia Commons, CC BY-SA 3.0

Cite this piece

Kelly Dennis, "AI's Hardest Tests Crack Within a Year," Ai Lately, Nov 21, 2026, https://ailately.com/articles/ai-benchmarks-crack-within-a-year

Tags: AI benchmarks · FrontierMath · ARC-AGI-2 · GPQA Diamond · benchmark saturation

Related, lately

More

Keyboard

j k
Move through a list
Enter
Open the selected piece
/
Search, and browse the columns
g then r
Right Now
g then a
Articles
g then s
The Signal
g then o
Opinion
g then b
Analysis
g then p
People
?
This sheet
Esc
Close

AI Right Now is free with an account.The Citation Audit is free with an account.Skills are free with an account.Tutorials are free with an account.You have read your 3 free articles.

Every article and comic, AI Right Now, the Citation Audit, Skills and Tutorials.

Join Ai Lately for free

Join with Google or your email. Already a member? Sign in

Press anywhere else for the home page.