Skip to content
Technology

Phone Chips Now Run AI Models Alone

Qualcomm's newest chip and Google's smallest Gemma model let a flagship phone answer and translate offline, while older and cheaper phones fall behind on memory.

Edited by Ryan Elliott Dennis5 min read · 1,249 words · 12 sources
A Qualcomm Snapdragon processor soldered to a smartphone circuit board
“A phone's own chip now decodes AI text seven times faster than a Raspberry Pi 5, a computer that skips the AI chip entirely. Does a phone finally carry enough silicon to answer on its own?”

Qualcomm's newest phone chip decodes AI text at 220 tokens a second3. That's fast enough to read a sentence back as the chip writes it. The phone can sit in airplane mode the whole time.

Google's smallest new model needs under 1.5 gigabytes of memory to run offline1. Two years ago, a model that size needed a data center behind it. That gap is closing fast.

What changed in the chip

Every modern phone chip now carries a neural processing unit, or NPU. An NPU is a small, separate chip. It runs AI math fast and light on battery, next to the regular processor that handles everything else.

A token is a small fragment of a word, the unit a model reads and writes one at a time. Qualcomm's newest Hexagon NPU handles them fast enough to feel instant, the company says3.

Skip the NPU, and a chip struggles by comparison. Google tested its own Gemma 4 E2B model on a Raspberry Pi 5. The Raspberry Pi skips a dedicated AI chip entirely.

Decoding text on that bare computer crawled to 7.6 tokens a second, a rate Qualcomm's Dragonwing IQ8 beats by four times1. That chip, built with an NPU for connected devices, reaches 31 tokens a second decoding the same model.

A Raspberry Pi 5 single-board computer, built around a general-purpose CPU alone
A Raspberry Pi 5, the bare computer Google used to show how slowly an AI model runs on a chip built around a CPU alone, in a test it published in April 2026.SimonWaldherr / Wikimedia Commons, CC BY-SA 4.0

A separate, independent test on an actual phone beat both. Daisuke Majima ran the identical Gemma 4 E2B model on an iPhone 17 Pro. Its A19 Pro chip measured 55.4 tokens a second with Google's own LiteRT-LM software. LiteRT-LM was the fastest of four runtimes he tried12.

An iPhone Decodes the Same AI Model 7 Times Faster Than a Bare Computer

Decode speed for Google's Gemma 4 E2B model, tokens per second, by chip and runtime

Each column is one chip running the identical Gemma 4 E2B model; the magenta column is the only one built into a phone.

Each chip ran a different software runtime tuned for its own hardware, so the gap reflects the software as well as the chip.

Data through June 2, 2026.

Sources: Google Developers Blog [1]; DEV Community [12]. Chart by Kelly Dennis.

The numbers behind this chart
ItemValueNote
Raspberry Pi 57.6 tok/sCPU alone
Dragonwing IQ831 tok/sNPU, connected devices
iPhone 17 Pro55.4 tok/sA19 Pro chip

Apple frames its own chip the same way. The company built a vapor chamber into the iPhone 17 Pro. That let the A19 Pro sustain its speed. Apple calls the pairing "ideal for gaming, video editing, and running large local language models"4.

MLCommons runs the industry's standard AI benchmarks, and it added tests built for exactly this shift in June. The new suite scores Llama models running locally on Qualcomm, MediaTek and Samsung chips11.

What changed in the model

The chip is half the story. Models had to shrink too. They still had to keep most of what makes them useful.

Google's Gemma 4 ships in five sizes. The smallest, E2B, holds 2.3 billion parameters. It scores 60 percent on a hard test called MMLU Pro2.

The biggest size built for phones and laptops, at 12 billion parameters, scores 77.2 percent on that same test2. E4B, the middle size, holds 4.5 billion parameters and lands almost exactly between the two, at 69.4 percent. A phone still needs far more memory to hold it.

A Bigger Gemma 4 Scores 17 Points Higher on One Benchmark

Google's own MMLU Pro score for three Gemma 4 sizes built to run on phones and laptops; higher is better

Each dot is one Gemma 4 size; further right means more parameters, and higher means a better score on Google's own test.

All three scores come from Google's own MMLU Pro run; a phone's memory still limits which size it can actually hold.

Data through April 2, 2026.

Source: Google AI for Developers [2]. Chart by Kelly Dennis.

The numbers behind this chart
NameParameters (billions)MMLU Pro score (%)
Gemma 4 E2B2.360
Gemma 4 E4B4.569.4
Gemma 4 12B1277.2

Meta and Microsoft ship the same tradeoff. Their smallest models land in between. Meta's Llama 3.2 scores 49.3 and 63.4 percent at its two sizes5. Microsoft's Phi-4-mini scores 67.3 percent on that same kind of test6.

Shrinking the file matters as much as shrinking the model. Google stores Gemma 4's smallest weights in 2 bits instead of the usual 16. That trick alone lets the model run in under 1.5 gigabytes of memory1.

The case for going offline

Speed and privacy carry the case for running AI on the chip. Both claims hold up once the numbers sit in front of you.

The speed case is the simple one. A phone answers in milliseconds because the question stays on the device. Cristiano Amon, Qualcomm's chief executive, made that case to The Register in June.

An AI agent should travel with its user, he said, rather than stay fixed to one piece of hardware. "The agent isn't tied to the device, it actually moves with the user," Amon said7.

Cristiano Amon, Qualcomm's chief executive, at a press conference
Cristiano Amon, Qualcomm's chief executive, at a Web Summit press conference. He told The Register in June 2026 that an AI agent should travel with its user rather than stay fixed to one piece of hardware.Web Summit / Wikimedia Commons, CC BY 4.0

Privacy is the harder case to prove. It is also the easier one to promise. Ziad Asghar, Qualcomm's senior vice president for XR, wearables and personal AI, ties both arguments to one engineering trick.

He means shrinking how a model stores its numbers. "For small devices which are really very limited from a memory perspective, as you know, if you do a model in 8-bit versus if you do it in 1-bit, it's one-eighth the size," Asghar told Business Today in October8.

Smaller weights are why Qualcomm's chip can answer a question that stays on the phone. The Register's own report on Amon's interview added a line Qualcomm rarely volunteers itself. On-device privacy holds only as far as the software vendor's willingness to respect it7.

Google AI@GoogleAI

Today, we’re launching Gemma 4, our most intelligent open models to date. Built with the same breakthrough technology as Gemini 3, Gemma 4 brings advanced reasoning to your personal hardware and devices.

Here’s what Gemma 4 unlocks for developers:

— Intelligence-per-parameter:… pic.twitter.com/JgwRZvQHgF

Open on X (opens a new tab)

Google posted this thread the day it released Gemma 4, pointing developers to the E2B and E4B sizes built for phones and other small devices.Post by Google AI (@GoogleAI) on X, Apr. 2, 2026

The case against

Storage is the complaint showing up first. It's already real. Google's AICore service runs Gemini Nano on Android, and it can occupy more than 10 gigabytes of a phone's storage at once9.

Ben Khalesi, who covers Android for Android Police, tested what that costs on a 128-gigabyte phone, the size millions of people still buy. "If a base phone cannot absorb model updates, media capture, and normal app growth without collapsing, it should not be the base phone anymore," Khalesi wrote in May9.

Reviewers raise quality next, and they keep finding it in the easiest features to test. Mat Smith reviewed Samsung's Galaxy S26 for Engadget in March and tried Now Brief, the phone's on-device daily summary. "Now Brief is capable of pulling data from more apps to generate more comprehensive daily summaries, but I mostly saw the usual suspects: weather, calendar reminders and not much else," Smith wrote10.

An iPhone charging on a wireless charging plate
A phone charges on a wireless pad. Reviewers count battery drain from sustained on-device AI work among the tradeoffs a faster chip has yet to erase.Seawhelan / Wikimedia Commons, CC BY-SA 4.0

Battery drain worries show up last. They are the hardest of the three to measure. Apple's own fix for it, that vapor chamber in the iPhone 17 Pro, exists for a reason. Sustained AI work gets hot enough to slow a chip down on its own4.

What to do

Check your phone's chip and its memory before trusting an on-device claim. A phone built before 2024 likely sends more AI work to a server than its marketing suggests. So does a phone with 8 gigabytes of memory or less.

Clear storage before an update if your phone sits near 128 gigabytes. AICore and its rivals can hold two model versions at once for a few days. That briefly doubles their footprint9.

Test a feature before trusting it with something that matters. Ask the same question twice, once offline and once online. Then compare the two answers.

By the numbers

  • 220 tokens a second: Qualcomm's claimed decode speed for its Hexagon NPU on the Snapdragon 8 Elite Gen 53.
  • 2.3 billion parameters: the size of Gemma 4 E2B, Google's smallest edge model2.
  • 60 percent: Gemma 4 E2B's score on the MMLU Pro benchmark, against 77.2 percent for the 12-billion-parameter version2.
  • Under 1.5 gigabytes: the memory Gemma 4 E2B needs to run on a phone1.
  • 10 gigabytes: storage Google's AICore service can occupy on an Android phone at once9.
  • 55.4 tokens a second: the fastest offline decode speed Daisuke Majima measured for the same model on an iPhone 17 Pro12.
  • 128 gigabytes: the base storage size Android Police says AI features have outgrown9.

What to watch

Watch for MLCommons to publish its first MLPerf Mobile v6.0 scores across real phones. Right now it only lists which chips the suite supports11. A public, independent number for Apple, Qualcomm and MediaTek silicon would settle the speed argument faster than any single company's claim.

Storage specs are worth watching too. A flagship phone that ships with more base storage would be a telling move. Doing that would mean the industry agrees with Khalesi.

Sources

  1. Google AI Edge Team, "Bring state-of-the-art agentic skills to the edge with Gemma 4," Google Developers Blog, April 2, 2026, https://developers.googleblog.com/bring-state-of-the-art-agentic-skills-to-the-edge-with-gemma-4/
  2. Google DeepMind, "Gemma 4 model card," Google AI for Developers, April 2, 2026, https://ai.google.dev/gemma/docs/core/model_card_4
  3. Abubakar Mohammed, "Qualcomm Snapdragon 8 Elite Gen 5 Announced with 3rd Gen Oryon Cores," Beebom, Sept. 25, 2025, https://gadgets.beebom.com/news/qualcomm-snapdragon-8-elite-gen-5-announced
  4. Apple, "Apple unveils iPhone 17 Pro and iPhone 17 Pro Max," Apple Newsroom, Sept. 9, 2025, https://www.apple.com/newsroom/2025/09/apple-unveils-iphone-17-pro-and-iphone-17-pro-max/
  5. Meta, "Llama 3.2: Revolutionizing edge AI and vision with open, customizable models," Meta AI Blog, Sept. 25, 2024, https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/
  6. Microsoft, "microsoft/Phi-4-mini-instruct," Hugging Face, Feb. 26, 2025, https://huggingface.co/microsoft/Phi-4-mini-instruct
  7. Tobias Mann, "'Resistance is futile,' says Qualcomm CEO. AI agents will become invisible, inescapable, follow you across devices," The Register, June 2, 2026, https://www.theregister.com/personal-tech/2026/06/02/qualcomm-ai-agents-will-be-as-transparent-as-they-will-be-inescapable/5249894
  8. Business Today, "Beyond the Smartphone: Qualcomm's Ziad Asghar on the rise of multimodal wearables and 1-Bit AI," Business Today, Oct. 29, 2026, https://www.businesstoday.in/technology/news/story/beyond-the-smartphone-qualcomms-ziad-asghar-on-the-rise-of-multimodal-wearables-and-1-bit-ai-564541-2026-10-29
  9. Ben Khalesi, "128GB had a long run. For modern Android flagships built around AI, it's done.", Android Police, May 16, 2026, https://www.androidpolice.com/128gb-phone-must-retire/
  10. Mat Smith, "Samsung Galaxy S26 review: The smartphone status quo," Engadget, March 13, 2026, https://www.engadget.com/mobile/smartphones/samsung-galaxy-s26-review-the-smartphone-status-quo-143033391.html
  11. MLCommons, "MLPerf Mobile v6.0: New GenAI Benchmarks for On-Device LLMs," MLCommons, June 15, 2026, https://mlcommons.org/2026/06/mlperf-mobile-v6/
  12. Daisuke Majima, "On-device LLM on iPhone: which runtime is fastest? MLX vs llama.cpp vs LiteRT-LM vs CoreML," DEV Community, June 2, 2026, https://dev.to/john-rocky/on-device-llm-on-iphone-which-runtime-is-fastest-mlx-vs-llamacpp-vs-litert-lm-vs-coreml-1b42

Header photo: Köf3 / Wikimedia Commons, CC BY-SA 3.0

In the market

QualcommQCOM

Market value $184 billion

$171.52 −0.55% since the last close

AppleAAPL

Market value $5.16 trillion

$354.90 −0.39% since the last close

Prices from Yahoo Finance, read with yfinance, and delayed. This is a record of price, not investment advice.

Cite this piece

Kelly Dennis, "Phone Chips Now Run AI Models Alone," Ai Lately, Nov 16, 2026, https://ailately.com/articles/phone-chips-now-run-ai-models-alone

Tags: Qualcomm · Google · Apple · Gemma 4 · Snapdragon 8 Elite Gen 5 · on-device AI

Related, lately

More

Keyboard

j k
Move through a list
Enter
Open the selected piece
/
Search, and browse the columns
g then r
Right Now
g then a
Articles
g then s
The Signal
g then o
Opinion
g then b
Analysis
g then p
People
?
This sheet
Esc
Close

AI Right Now is free with an account.The Citation Audit is free with an account.Skills are free with an account.Tutorials are free with an account.You have read your 3 free articles.

Every article and comic, AI Right Now, the Citation Audit, Skills and Tutorials.

Join Ai Lately for free

Join with Google or your email. Already a member? Sign in

Press anywhere else for the home page.