OpenAI spent two hundred million dollars teaching GPT-4.5 to predict the next word. It spent two million more drilling the model afterward1. That split, a hundred to one, held for years.
Pretraining teaches a model to predict text from writing. Post-training comes next.
Engineers give the model practice problems. They score its tries and nudge it toward better scores. Reinforcement learning, RL for short, drives that practice.
That hundred-to-one habit is breaking. Epoch AI, which tracks computing trends, finds post-training compute scaling roughly ten times every four months1. Pretraining still grows four to five times a year, at best.
Where the Research Budget Actually Goes
OpenAI's own numbers show why. Its compute budget ran about five billion dollars in 2024, climbing toward nine billion in 20251. Only a slice of that pays for one finished model.
Epoch AI dug into three labs' spending, and compared one final run against the full budget2.
GPT-4.5's Training Bill Split $200 Million to $2 Million
OpenAI's estimated pretraining and post-training cost for GPT-4.5, in millions of dollars, per Epoch AI
Each column is one phase of training GPT-4.5; the pretraining column stands far taller because it cost about a hundred times what post-training cost.
Source: Epoch AI [1]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Pretraining | $200M |
| Post-training | $2M |
OpenAI's final run took just 9.6 percent of its 2024 budget. That is a small slice. The rest paid for experiments and other research2.
Two Chinese labs showed the same pattern, at smaller scale. MiniMax's final run took 22.6 percent of its research compute. Z.ai's took 12.3 percent2.
Final Training Runs Take 9.6 to 22.6 Percent of Three Labs' Research Compute
Share of each company's research and development compute budget spent on its one final training run, per Epoch AI
Each bar is one company's final training run as a share of its total research compute; a taller bar means more of the budget went to that single run.
OpenAI's figure covers 2024. MiniMax's runs Q4 2024 through Q3 2025, and Z.ai's runs H2 2024 through H1 2025, so each bar reflects a different stretch of spending.
Source: Epoch AI [2]. Chart by Kelly Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| MiniMax | 22.6% |
| Z.ai | 12.3% |
| OpenAI | 9.6% |
Reinforcement learning sits inside that larger share. It is the fastest-growing piece of it.
OpenAI Says It Is Flipping the Ratio on Purpose
Dan Roberts leads OpenAI's work on the foundations of reinforcement learning. He posted about it in May 2025.
Dan Roberts@danintheoryMy short talk at @sequoia AI Ascent on how we at @OpenAI are attempting to flip the old "cherry-on-top" meme about "Scaling RL" pic.twitter.com/nVmpGZ8yan
"My short talk at @sequoia AI Ascent on how we at @OpenAI are attempting to flip the old 'cherry-on-top' meme about 'Scaling RL,'" Roberts wrote3.
That old meme treated reinforcement learning as an afterthought. It was a small polish on finished pretraining. Roberts says his team wants the opposite now. He wants a smaller pretrained model, built up mostly through practice.
New reasoning techniques make that trade possible. A lab can match GPT-4.5 using roughly a tenth of the pretraining compute. It then pours the savings into post-training instead1.
Why Labs Keep Buying Practice Problems
Practice beating lessons is an old idea in AI research. Google DeepMind proved an extreme version of it in 2017. Its program, called AlphaGo Zero, played the board game Go.
Earlier versions of AlphaGo studied thousands of human games first, but AlphaGo Zero skipped that step. It only played against itself.
"This technique is more powerful than previous versions of AlphaGo because it is no longer constrained by the limits of human knowledge," DeepMind's David Silver and Demis Hassabis wrote6. After three days, it won 100 games to 0.
Investors are backing a modern version of that bet. Anthropic discussed spending more than one billion dollars on reinforcement learning environments over one year, The Information reported5.
Surge builds those environments for labs. It made $1.2 billion from the work last year.
Will Brown, a Prime Intellect researcher, expects a crowded market. "RL environments are going to be too large for any one company to dominate. Part of what we're doing is just trying to build good open source infrastructure around it," Brown told TechCrunch5.
Brendan Foody, Mercor's chief executive, made a similar case. "Few understand how large the opportunity around RL environments truly is," he said5.
Noam Brown, an OpenAI researcher, described the pattern on the Dwarkesh podcast in September. "When you plot the performance of these reasoning models with test-time compute on the x-axis and performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern," Brown said4.
The Case Against the Trade
Toby Ord doubts the trade is worth it. He is a senior researcher at Oxford's AI Governance Initiative.
Ord measured how much came from reinforcement learning training itself. The rest came from letting a model think longer per answer. On a mock AIME exam, training alone added four points. Extra thinking time added forty-five8.
"In general, to get the same benefit from RL-scaling as from inference-scaling required twice as many orders of magnitude," Ord wrote. "That's not good"7.
Ord put the limit more bluntly elsewhere. "We have reached the point where it is too expensive to go much further," Ord wrote of scaling reinforcement learning training on its own7.
His own numbers leave the strategy unresolved. Labs are betting on far more post-training compute. That spending compounds ten times every four months. They are betting it beats the poor return Ord measured in one run.
What to do
Read a vendor's "bigger model" pitch with new skepticism this quarter. Epoch's own numbers show the money moving into reasoning and practice runs, ahead of larger pretrained bases12.
Ask any AI vendor how it measures a reasoning gain, separate from its parameter count, before you pay for an upgrade. The answer now says more about where its engineering budget actually went.
By the numbers
- $200 million to $2 million: OpenAI's estimated pretraining and post-training cost for GPT-4.5, a hundred-to-one split, per Epoch AI1.
- 10 times every four months: Epoch AI's estimate for how fast post-training compute can scale1.
- 9.6 percent: the share of OpenAI's roughly $5 billion 2024 research compute budget spent on its one final training run2.
- MiniMax and Z.ai's own shares: 22.6 percent and 12.3 percent, both bigger than OpenAI's2.
- More than $1 billion: what Anthropic discussed spending on reinforcement learning environments over one year, per The Information5.
- $1.2 billion: Surge's revenue last year from building those environments for AI labs5.
- 4 points against 45: the AIME score gain from reinforcement learning training, against extra inference time, per Toby Ord8.
- Under 5 percent: the share of pretraining compute Google spent on post-training for Gemini 2.5 Pro at launch, per SemiAnalysis9.
- Also under 5 percent: roughly what Alibaba's Qwen team spent on post-training for its own models, per SemiAnalysis9.
What to watch
Watch OpenAI's next pretraining run. A bigger one would show the industry pulling back from pure reinforcement learning scaling. Staying flat, paired with a bigger post-training bill, would confirm the shift is holding.
Anthropic's discussed billion-dollar environment budget deserves a second look too. A plan and a signed budget are different facts. Check whether that number turns into a filed figure next earnings season.
Sources
- Yafah Edelman, Jean-Stanislas Denain, Jaime Sevilla and Anson Ho, "Why GPT-5 used less training compute than GPT-4.5 (but GPT-6 probably won't)," Epoch AI, Sept. 26, 2025, https://epoch.ai/gradient-updates/why-gpt5-used-less-training-compute-than-gpt45-but-gpt6-probably-wont
- Jean-Stanislas Denain and Cheryl Wu, "Final training runs account for a minority of R&D compute spending," Epoch AI, March 23, 2026, https://epoch.ai/gradient-updates/r-and-d-vs-training-compute
- Dan Roberts (@danintheory), post on his AI Ascent talk about OpenAI inverting the "cherry-on-top" meme for scaling RL, X, May 9, 2025, https://x.com/danintheory/status/1920898285142716782
- Dwarkesh Patel, "Noam Brown," Dwarkesh Podcast, Sept. 17, 2026, https://www.dwarkesh.com/p/noam-brown
- Maxwell Zeff, "Silicon Valley bets big on 'environments' to train AI agents," TechCrunch, Sept. 21, 2025, https://techcrunch.com/2025/09/21/silicon-valley-bets-big-on-environments-to-train-ai-agents/
- David Silver and Demis Hassabis, "AlphaGo Zero: Starting from scratch," Google DeepMind, Oct. 18, 2017, https://deepmind.google/blog/alphago-zero-starting-from-scratch/
- Toby Ord, "How Well Does RL Scale?," tobyord.com, Oct. 20, 2025, https://www.tobyord.com/writing/how-well-does-rl-scale
- Toby Ord, "Evidence that Recent AI Gains are Mostly from Inference-Scaling," tobyord.com, Oct. 3, 2025, https://www.tobyord.com/writing/mostly-inference-scaling
- Dylan Patel and AJ Kourabi, "RL Environments and RL for Science: Data Foundries and Multi-Agent Architectures," SemiAnalysis, Jan. 6, 2026, https://newsletter.semianalysis.com/p/rl-environments-and-rl-for-science