The Top Score on the X-Ray Test Skipped the X-Ray

A text-only program, kept away from every image in the test, posted a higher score than the radiologists it faced. It beat them by more than 10 percent on the largest public chest X-ray benchmark ever built. Stanford researchers reported the result in a paper posted in March and revised in April¹. The model carried three billion parameters, a fraction of the size of the frontier systems it beat. It read the question and skipped the scan.

Read only the top line and the reading room looks automated already. Executives are drawing exactly that line. Mitchell Katz, president and chief executive of NYC Health and Hospitals, the country's largest public hospital system, told a Crain's New York Business panel in March, "We could replace a great deal of radiologists with AI at this moment, if we are ready to do the regulatory challenge"². Katz picked the present tense, "at this moment," which frames the swap as ready today and stalled only by paperwork.

Katz's logic runs on a simple equation: a high score means a working eye. The Stanford paper measured the opposite. Strip the image from the test and the models kept 70 to 80 percent of their original accuracy, with some medical questions holding over 90 percent¹. A system rewarded only for the right answer can match a radiologist's score after skipping a radiologist's task.

Mohammed Suhail, a radiologist at North Coast Imaging in San Diego, read Katz's comment as an equation of a different kind. "Any attempt to implement AI-only reads would immediately result in patient harm and death," Suhail said, calling the plan "undeniable proof that confidently uninformed hospital administrators are a danger to patients"². Suhail's second phrase does the real work: confidence, in his telling, is the problem, and the benchmark manufactured it from a guess.

The workforce Katz wants to shrink is already thin in the wrong places. Radiology job postings analyzed this spring, 20,775 of them between March and May, left 1,470 positions open for over two months, about one in five of all unique listings, a RadBoard job market study found³⁴. Kirill Lopatin, the RadBoard analyst who ran the numbers, located the shortage precisely: "The shortage isn't national, it's concentrated in places radiologists won't move to"³.

Nebraska postings sat unfilled 68 percent of the time; Texas postings closed in weeks³. Radiology postings in the slowest markets, the same analysis found, pay up to $175,000 below postings that fill within weeks⁶. Money moves radiologists. A benchmark score moves paperwork.

Long-run projections deal Katz's shortcut the same verdict. Radiologist supply is set to grow 26 percent by 2055, the Neiman Health Policy Institute found in research published last year⁵. Imaging demand is set to grow at almost the same pace over the same span⁵. Eric Christensen, the institute's research director, put the conclusion in a sentence: "Given the comparable projected levels of growth in supply and demand, the present radiologist shortage is projected to persist unless steps are taken to grow the workforce and/or decrease per person imaging utilization"⁵.

Ask who profits from a shortage this precisely mapped. Hospitals that adopt AI reading tools skip the radiologists they struggled to recruit in Nebraska. Patients in those same stuck markets get a machine tuned on a benchmark that rewards a plausible answer over a correct one. Radiologists who already live where the jobs cluster keep their premium, and the model gets credited for solving a shortage that was always a matter of geography. Christensen's own remedy, more residency seats or lighter scan volume, asks for people and years; Katz's remedy asks only for a settled rule.

The verdict sits inside the paper Katz cited to make his case. A benchmark that scores an imageless answer as sight measures the wrong thing entirely. Hospital systems carry a real staffing crisis, clustered in towns radiologists decline to move to. Pay and a moving truck fix that faster than a model graded on a broken exam. Readers who want the next hospital story before the executives spin it can find it every morning on The Signal, AI Lately's daily email brief.

Sources

1. Asadi, Mohammad, et al. "MIRAGE: The Illusion of Visual Understanding." arXiv, Stanford University, March 23, 2026. https://arxiv.org/abs/2603.21687
2. Futurism Staff. "America's Largest City Hospital System Ready to Start Replacing Radiologists With AI, Its CEO Says." Futurism, April 4, 2026. https://futurism.com/artificial-intelligence/hospital-ceo-ai-radiology
3. Carey, Liz. "Radiology job market report casts doubt on U.S. radiologist shortage." AuntMinnie, June 8, 2026. https://www.auntminnie.com/practice-management/administration/article/15827084/radiology-job-market-report-casts-doubt-on-us-radiologist-shortage
4. RadBoard. "US Radiology Job Market Dynamics Report — May 2026." RadBoard, May 26, 2026. https://radboard.io/reports/2026-radiology-dynamics-report.html
5. Neiman Health Policy Institute. "New Studies Shed Light on the Future Radiologist Workforce Shortage by Projecting Future Radiologist Supply and Demand for Imaging." Neiman Health Policy Institute, Feb. 12, 2025. https://www.neimanhpi.org/press-releases/new-studies-shed-light-on-the-future-radiologist-workforce-shortage-by-projecting-future-radiologist-supply-and-demand-for-imaging/