Skip to content
Talent

Amazon Calls Token Metrics Flawed, Keeps Them

Amazon's new memo tells managers to judge AI by results, though it still tracks how much each team uses.

6 min read · 1,272 words · 5 sources
The glass domes of the Amazon Spheres rise over Amazon's Seattle headquarters campus
The glass domes of the Amazon Spheres rise over Amazon's Seattle headquarters campus. Photo: Andy Li / Wikimedia Commons, CC0
“Amazon's own engineering chief called AI token counts 'deeply flawed,' then told managers to keep sorting engineering teams by the 10th and 90th percentile of how much AI they use. Does a company's own warning change how its managers use the number anyway?”

Amazon told its software engineers this week to stop chasing AI token counts. A new internal memo calls token counts, AI adoption rates, and lines of code "deeply flawed" ways to judge good work1. The same memo still tells managers to track those same three numbers, team by team.

Terk Terkowitz, an Amazon vice president and distinguished engineer, signed the memo. He calls this moment the era of the "Renaissance Developer"1.

The Renaissance Developer, Defined by Amazon

Terkowitz leads engineering guidance for Worldwide Amazon Stores. His memo argues that judgment, more than any single use of AI, now marks a good engineer.

"Our most prolific engineers have demonstrated that neither the developer role nor their output is defined by the use of AI," Terkowitz wrote1.

That line retires Amazon's old scoreboard. Engineers once climbed an internal leaderboard by burning through AI tokens. The industry calls that habit tokenmaxxing: running a coding assistant hard just to post a bigger number. Terkowitz wants engineers blending AI with their own skill instead, building things "neither humans nor AI tools could produce alone"1.

He frames the shift as continuity, pointing to a 2012 line from Jeff Bezos: "What's going to change in the next five to ten years? But I very rarely get asked, 'What's not going to change?'"1.

A Flawed Number Amazon Still Watches

Here is the twist Terkowitz's own memo prints. Token counts, AI adoption, and code-change volume stay "deeply flawed" measures of a team's worth, he writes. They drift from what actually matters to customers1.

Yet the memo tells managers to keep reading them. It recommends watching the 10th percentile of teams, the slowest to adopt AI. The 90th percentile, the heaviest AI users, gets the same scrutiny. Either way, the memo asks whether a team's output backs up its number1.

Terkowitz even credits the flawed metrics with a point in their favor: "these input metrics correlate well with the outputs that matter," he writes, "and we expect they will continue to do so for some time."

Amazon's Day 1 Tower stands in downtown Seattle
Amazon's Day 1 Tower rises in downtown Seattle, photographed in January 2017. Terk Terkowitz's new memo to engineers quotes Jeff Bezos's own writing on what stays fixed while AI changes everything else.Adamajreynolds / Wikimedia Commons, CC BY-SA 4.0

So Amazon kept its token counts, instead of dropping them. It reclassified them, from a target engineers chase to a measure managers weigh "with care and judgement"1. A team or an engineer who finds little use for AI gets cover from the memo, rather than blame.

"If your work does not benefit from current AI tooling, that is a finding, not a failing," Terkowitz wrote1. "An engineer who can explain where AI does not yet help their work is demonstrating the judgement we are asking for."

Why Outcomes Are Harder to Fake

A token count rewards volume. An outcome, in theory, rewards a shipped feature or a fixed bug, the kind of result a manager can check against a customer's experience. Amazon's cloud chief has been making that case since summer.

The Amazon Invictus Building rises over South Lake Union in Seattle
The Amazon Invictus Building anchors the company's South Lake Union campus in Seattle, home to the software engineers Terkowitz's memo now judges by outcome instead of token count.Joe Mabel / Wikimedia Commons, CC BY-SA 4.0

"I was talking to a room full of CIOs just a couple of months ago," Matt Garman, Amazon Web Services' chief executive, said. "I asked, 'How many of you are either seeing materially positive ROI today or have a path in the next couple of months to really high ROI?' 90% of hands went up, which is totally different than a year before"2.

Garman ties the shift to picking the tool that fits the job, over the flashiest one. "One of the things that's driven up a bunch of cost of AI is that people were trying to use the best model for every single thing," he said2.

Developers Use AI Daily, Trust It 25 Points Less

Share of developers who use AI coding tools daily versus who trust AI output, Stack Overflow's 2026 Developer Survey

Each column is a separate share of the same group of developers: the first use AI coding assistants or agents every day, the second trust what AI gives them only once they can check it themselves.

Data through Oct. 6, 2026. Sample: 17,464 developers who answered the AI questions in Stack Overflow's 2026 Developer Survey.

Source: Gergely Orosz on X [3]. Chart by Ryan Elliott Dennis.

The numbers behind this chart
ItemValue
Use AI daily73%
Trust once checked48%

Outside Amazon, the data backs his instinct. Stack Overflow's 2026 Developer Survey found 73 percent of developers use AI coding assistants or agents daily. Only 48 percent trust the answers once they can check the work themselves4. Usage and confidence are two different numbers, and a token count only ever measures the first one.

69 Percent of Developers Point AI at Code They Know

Share of developers who use AI for each task, Stack Overflow's 2026 Developer Survey

Each bar is the share of developers who turn to AI for that task. The longest bar is also the task where a developer already knows what the right answer looks like.

Data through Oct. 6, 2026. Sample: 17,464 developers who answered the AI questions in Stack Overflow's 2026 Developer Survey.

Source: Gergely Orosz on X [3]. Chart by Ryan Elliott Dennis.

The numbers behind this chart
ItemValue
Code they already know69%
Debugging and refactoring64%
Writing tests58%

That same survey shows where developers actually point their AI tools. Sixty-nine percent use them on code they already know. Debugging and refactoring draws 64 percent, and writing tests draws 58 percent4.

Developers lean on AI hardest in the places they can still check its work by hand.

The Grand Plaza sits between Amazon's office buildings on its South Lake Union campus
Amazon's Grand Plaza sits between the Cricket and Invictus buildings on the South Lake Union campus, where Amazon Web Services chief Matt Garman has pushed engineers toward cheaper, task-fit models.Joe Mabel / Wikimedia Commons, CC BY-SA 4.0

Trillions of dollars sit behind this question, well past Amazon's own org chart. Gartner expects worldwide IT spending to reach $7.35 trillion in 2027, up 15 percent from this year. Data center and AI software spending drive most of the new money5.

"Collectively, the market forces are reinforcing spend around AI infrastructure, AI-enabled cloud platforms and advanced software ecosystems," said John-David Lovelock, a distinguished vice president analyst at Gartner. Real money now rides on whether any company can tell a busy AI dashboard from a useful one.

The Divide Gergely Orosz Sees Coming

Gergely Orosz edits The Pragmatic Engineer, a newsletter read widely inside the industry Amazon's memo addresses. He read the memo the morning it reached the public and posted his own verdict within hours.

"Amazon's leadership is very good at writing, but you still need to read between the lines," Orosz wrote. "What I read between these lines: Engineers who are not 'product engineers' (aka eng + PM interest) and don't like to be customer-facing will not have a future inside of Amazon"3.

Gergely Orosz@GergelyOrosz

Amazon's leadership is very good at writing, but you still need to read between the lines.

What I read between these lines:

Engineers who are not "product engineers" (aka eng + PM interest) and don't like to be customer-facing will not have a future inside of Amazon

Full:… pic.twitter.com/exf4g6z99U

A screenshot of Amazon's engineering memo, posted alongside Gergely Orosz's reading of it
Open on X (opens a new tab)

Gergely Orosz, editor of The Pragmatic Engineer, read Amazon's memo on Oct. 28, 2026, the day it reached the public, and posted his own warning about who it leaves out.Post by Gergely Orosz (@GergelyOrosz) on X, Oct. 28, 2026

Orosz is reading the same document Terkowitz signed, and finding a narrower message inside the warm one. Amazon's memo answers part of that reading directly. A team or an engineer who gets little lift from AI tooling earns a finding, the memo says, rather than a failing1.

It stays silent on an engineer who simply prefers writing code alone, away from customers. That is the exact profile Orosz says the memo is quietly sorting out.

Both readings can sit inside the same document. A company can mean its warning against tokenmaxxing honestly. It can still be building, sentence by sentence, a picture of the engineer it wants to keep.

What to do

If you manage people who use AI tools, read your own usage dashboard the way Amazon just read its own. A high count of prompts or tokens shows someone stayed busy. The quality of their work remains a separate question.

Ask instead for the outcome. A closed ticket counts. So does a fixed bug, or a customer who got a real answer.

Then ask the engineer to explain, in their own words, where AI fell short. Amazon now treats that answer as a sign of judgment instead of a confession.

By the numbers

  • Nine in 10 CIOs told Matt Garman they see positive ROI from AI, up sharply from a year earlier2.
  • 73 percent of developers use AI coding assistants or agents every day, Stack Overflow's 2026 survey found4.
  • Fewer than half, 48 percent, trust AI's answers only once they can check the work themselves4.
  • Most AI use stays close to home: 69 percent of developers point it at code they already know how to write4.
  • Two percentiles, the 10th and the 90th, are what Amazon uses to sort teams by AI use, the metric its memo calls "deeply flawed"1.
  • $7.35 trillion: worldwide IT spending Gartner forecasts for 2027, up 15 percent from 20265.

What to watch

Watch whether Amazon's performance reviews actually change, beyond the memo's own words. A manager who still ranks a team by its percentile of AI use, quietly, would answer Orosz's reading before Amazon's own words do.

Other large employers may publish their own version of this memo next. Amazon rarely moves alone on how it measures engineers. A second company's language will show whether "outcomes over tokens" is a lasting standard, or one memo's phrase.

Sources

  1. Terk Terkowitz, "An update from Amazon engineering leadership on the role of the developer in the age of AI," Amazon, Oct. 27, 2026, https://www.aboutamazon.com/news/company-news/amazon-software-engineer-role-guideline-ai
  2. Amazon Staff, "AWS CEO Matt Garman on why enterprise AI is finally delivering real returns," Amazon, Aug. 1, 2026, https://www.aboutamazon.com/news/aws/aws/aws-matt-garman-enterprise-ai-roi-interview-2026
  3. Gergely Orosz, "Amazon's leadership is very good at writing, but you still need to read between the lines," X, Oct. 28, 2026, https://x.com/GergelyOrosz/status/2115331737714622800
  4. Stack Overflow, "2026 Developer Survey: AI," Stack Overflow, Oct. 6, 2026, https://survey.stackoverflow.co/2026/ai
  5. Gartner Staff, "Gartner Forecasts Worldwide IT Spending to Grow 15% in 2027, Totaling $7.35 Trillion," Gartner, Oct. 21, 2026, https://www.gartner.com/en/newsroom/press-releases/2026-10-21-gartner-forecasts-worldwide-it-spending-to-grow-15-percent-in-2027-totaling-7-point-35-trillion

In the market

AmazonAMZN

Market value $2.75 trillion

$254.60 +0.59% since the last close

Prices from Yahoo Finance, read with yfinance, and delayed. This is a record of price, not investment advice.

Cite this piece

Ryan Elliott Dennis, "Amazon Calls Token Metrics Flawed, Keeps Them," AI Lately, Oct 29, 2026, https://ailately.com/articles/amazon-calls-tokens-flawed-keeps-them

Tags: Amazon · tokenmaxxing · AI metrics · software engineering · developer productivity

Related, lately

More

Keyboard

j k
Move through a list
Enter
Open the selected piece
/
Search, and browse the columns
g then r
Right Now
g then a
Articles
g then s
The Signal
g then o
Opinion
g then b
Analysis
g then p
People
?
This sheet
Esc
Close

AI Lately Pro

Be the one who already knew.

With Pro
You already know what ships next week and who just moved, with the source.
Without it
You hear about it days later, from a screenshot in someone's Slack.

$19 a month. Cancel any time.

See everything Pro opensAsk the editor

Secure checkout through Stripe. Member? Sign in.