Four out of five software teams merged AI-built code or tests they understood only in part, TestMu AI reported Nov. 131. The company surveyed testers, developers and engineering leaders at firms from ten employees to ten thousand. Forty-four percent said more bugs now reach customers as AI wrote more of the code. Ninety-seven percent of the same teams use AI in testing already, or plan to this year.
The easy read is simple. Software teams got careless, and craft slipped as speed took over. TestMu AI's own report tells a different story.
Some teams add a second AI model to grade the first one's work. Those teams merged misunderstood code almost twice as often as teams that skipped it, 29 percent against 15 percent1.
The same teams also saw more bugs reach customers afterward, 50 percent against 41 percent. Outside research matches that gap. Researchers built the MT-Bench benchmark in 2023. They found AI judges favor answers from their own kind, a pattern they named self-enhancement bias3.
Teams using the AI judge also reported better returns, 90 percent calling it positive against 66 percent for other teams1. TestMu's own researchers call the pattern an association, since the fastest-moving teams may adopt every AI practice at once.
An AI judge lifts misunderstood merges to 29%, against 15% for other teams
Share of teams, TestMu AI's survey of testers, developers and engineering leaders
Each row compares teams that use one AI model to grade another against teams that skip the practice; the rod is the gap between them.
TestMu AI's own researchers note the fastest-moving teams may adopt every AI practice together, which keeps this an association.
Source: TestMu AI [1]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Item | Uses an AI judge | Other teams |
|---|---|---|
| Merges misunderstood AI output | 29% | 15% |
| Sees escaped defects rising | 50% | 41% |
| Reports positive AI testing ROI | 90% | 66% |
Arena Intelligence tested the pattern again this fall. Researchers there ran 34,580 verdicts from 12 models across 1,460 head-to-head battles4.
Arena.ai@arenaAsk an AI model to pick the better answer, and it'll usually pick its own.
We compared 34,580 verdicts from 12 models with human votes across 1,460 battles on Arena. The results show that AI judges have their own taste, and it's unlike ours.
They:
- Favor their own answers. On… https://t.co/S5LXy4Ycty pic.twitter.com/ccqUGWCr4X
A judge picked its own answer 58 percent of the time, while people did the same only 34 percent of the time. OpenAI's GPT-6 Astra picked itself most often, in 88 percent of its battles, with GPT-5.6 Sol close behind at 80 percent. Anthropic's Fable 5.1 backed itself in 72 percent of its battles.
MiniMax M3 sat at the other end, backing itself only 22 percent of the time. Teams that put a model in charge of grading AI work handed the grading job to a grader with a favorite.
AI judges pick their own answer far more than people do, led by GPT-6 Astra's 88 percent
Share of battles a judge model picked its own answer, Arena Intelligence, 1,460 battles, Sept. 2026
Each bar is one model's self-pick rate; people picked the same answers only 34 percent of the time, on average, across the same battles.
Source: Arena.ai [4]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| GPT-6 Astra | 88% |
| GPT-5.6 Sol | 80% |
| Fable 5.1 | 72% |
| 12-model average | 58% |
| MiniMax M3 | 22% |
Arena's judges sat blind to which model wrote which answer, and still reached for their own kind. The blind test answers TestMu's own caution: the bias shows up on its own, separate from speed or adoption.
Wei-Wei Wu runs Momentic, a company that sells AI agents for testing. Wu wrote in September that the shift already runs ahead of the debate. "AI will eventually replace most traditional QA roles and it's already happening," Wu wrote6. Momentic's own pitch rests on an agent that writes, runs and grades its own tests alone.
Asad Khan runs TestMu AI, the company whose survey supplies this column's numbers. Khan reached a different conclusion from his own data. "We believe any output AI produces should be validated by a system independent of the model that produced it, before it merges and every time it changes," Khan wrote in the report2.
Teams that run a separate, automated check before merging see fewer escaped defects than teams that skip it. The gap runs 38 percent against 49 percent.
Growth here is real, federal data shows. The Bureau of Labor Statistics counted 187,600 software quality assurance analysts and testers in 20255. It projects 198,200 by 2035, a six percent rise. Median pay for the role reached $104,300 in May 2025.

Reviewing AI-generated output already takes up about 18 percent of a tester's week. Flaky tests and test maintenance tie it for the top spot, TestMu found.
Here is the question Rhoden would ask of any new machine. Who pays when the grader misses the mistake? Vendors profit either way. Momentic sells the agent that writes and grades its own work6.
TestMu sells the platform built to catch what that agent misses1. Workers absorb the risk when both miss it together. TestMu's own survey found only 13 percent of teams keep a plan for what happens after an AI mistake reaches a customer.
The verdict is plain. Letting one AI model grade another relocates the risk behind a clean-looking dashboard. A company that wants fewer surprises still needs a checker built by a different maker than the one it watches. The outside checker stands between the code and the customer.
Sign up for The Signal to catch the next surprise early. The Signal is Ai Lately's daily email on who moved, what shipped and what it cost in artificial intelligence.
Sources
- TestMu AI Research, "State of AI in Quality Engineering 2026," TestMu AI, Nov. 13, 2026, https://www.testmuai.com/research/state-of-ai-in-quality-engineering/
- TestMu AI, "81% of Software Teams Have Merged AI-Written Code They Didn't Fully Understand, Finds TestMu AI's State of AI in Quality Engineering 2026 Report," PR Newswire, Nov. 13, 2026, https://www.prnewswire.com/news-releases/81-of-software-teams-have-merged-ai-written-code-they-didnt-fully-understand-finds-testmu-ais-state-of-ai-in-quality-engineering-2026-report-302938904.html
- Lianmin Zheng and co-authors, "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena," arXiv, Dec. 24, 2023, https://arxiv.org/abs/2306.05685
- Arena Team, "Do LLM Judges Favor Themselves?," Arena.ai, Sept. 30, 2026, https://arena.ai/blog/llm-judge-self-preference
- Occupational Outlook Handbook, "Software Developers, Quality Assurance Analysts, and Testers," U.S. Bureau of Labor Statistics, Sept. 2025, https://www.bls.gov/ooh/computer-and-information-technology/software-developers.htm
- Wei-Wei Wu, "Will AI Replace QA Engineers?," Momentic, Sept. 13, 2026, https://momentic.ai/blog/will-ai-replace-qa-engineers




