Beth Israel Deaconess Medical Center tested a chatbot named AMIE on real patients before primary care visits in 20251. Ninety-eight patients finished both the chat and the doctor visit that followed. Those 98 chats ended with 0 needing a doctor to step in for safety.
Read quickly, the number says a chatbot can handle a doctor's visit alone. Seventeen percent of American adults went the year unassigned to a primary care doctor4. A certified doctor watched every one of the 98 chats happen in real time1. Rodman and his co-authors tested whether the chatbot could run safely, apart from whether it improves care.
BIDMC's 98 AI patient chats closed with 0 safety stops
Real-world AMIE chats with primary care patients, Beth Israel Deaconess Medical Center, April to November 2025
Each cube is one of the 98 finished chats, and every one closed with 0 safety stops.
Source: Beth Israel Deaconess Medical Center [1]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Needed a safety stop | 0 |
| Closed clean | 98 |
| Whole | 98 |
AMIE's possible diagnoses included the one doctors settled on 90 percent of the time, Google reported2. A list of guesses is a softer bar than one answer a patient could use alone.
Adam Rodman led the study as director of AI programs at BIDMC's Shapiro Institute1. "Before we begin evaluating them as workload interventions, AI tools must first be shown to operate safely and acceptably in real care workflows," Rodman said. Primary care doctors reviewed an AI-written summary before 44 of the 98 visits. Seventy-five percent said the summary helped them prepare, and 57 percent said it changed their approach.
AMIE's pre-visit summary changed 57 percent of doctors' approach
Physician-reported effects of reading AMIE's summary before 44 primary care visits, Beth Israel Deaconess Medical Center, 2025
Each column is the share of physicians reporting that effect after reading the summary before the visit.
Source: Beth Israel Deaconess Medical Center [1]. Chart by Ryan Elliott Dennis.
The numbers behind this chart
| Item | Value |
|---|---|
| Helped prepare | 75% |
| Changed approach | 57% |
Jacob Koshy, a hospital doctor on the research team, framed the open question before the study began3. "One of the most important questions was simply how patients would engage with a conversational AI system in a real clinical setting," Koshy said. Peter Brodeur, a heart fellow on the team, said the numbers now set a mark. "This study helps establish the baseline characteristics of such real-world conversations," Brodeur said.
Primary care doctors already chart after dinner most nights6. Reviewing a chatbot's transcript before every visit adds a new task to the doctor's morning. The share of doctors logging eight-plus hours a week on records reached 22.5 percent in 2024. That rate sat at 20.9 percent the year before.
Marc Cohen directs primary care at BIDMC and helped lead the study3. "We believe the future involves continued involvement of human clinicians in patient care but see a future where the triad of patient, human clinician, and AI help to provide our patients with the very best evidence-based care possible," Cohen said. Cohen's own sentence keeps the doctor in the room and still welcomes the machine beside the patient.
Steven Lin runs primary care at Stanford's medical school and studies these same tools4. "In its current state, the safest use of this tool is for more urgent care issues," Lin said. He named upper respiratory and urinary tract infections, visits a chatbot can carry alone. Lin draws the line at diabetes and heart disease, the cases primary care actually carries.
Tammy MacDonald felt that gap firsthand this year4. Her doctor died in August, and the nearest human appointment ran nearly two years out. Mass General Brigham's Care Connect, a chatbot paired with 12 remote doctors, saw her within 48 hours. "It was just such a difference from being told I had to wait two years," she said.
Michael Barnett sees Mass General Brigham patients and helps organize his colleagues' union4. "That sounds like a Band-Aid for a broken system to me," he said.
Madhuri Rao practices primary care at an MGB clinic in Chelsea and organizes alongside him4. "They don't make any effort to ease the shortage. They put their money into specialties," Rao said.
Google profits from the paper alone, its first in The Lancet's main journal2. Mass General Brigham profits from Care Connect too, since its $400 million pledge funds the chatbot first4.
Rodman and his BIDMC colleagues carry the new risk, the hours spent reading a transcript before meeting the patient1. One patient saw lymphoma on AMIE's list before a doctor explained it. The anxiety that followed was the only harm anyone reported in 98 tries.
The real primary care shortage is a pay problem before it is a software problem. Primary care pay runs 30 to 50 percent under what heart doctors and anesthesia doctors earn4. A perfect score, watched the whole time, answers a narrower question than this industry is asking.
Readers who want primary care's AI news checked against the pay numbers can find it in The Signal. Sign up for AI Lately's daily email, and get the staffing story before the next chatbot headline arrives.
Sources
- Staff, "BIDMC Researchers Conduct First Real-World Study of Safety and Quality of Patient-Facing AI in Primary Care," Beth Israel Deaconess Medical Center, Oct. 9, 2026, https://bidmc.org/news-stories/all-news-stories/news/2026/10/bidmc-research-shows-safety-quality-of-ai-in-primary-care
- Mike Schaekermann, "Study in The Lancet suggests AI could improve patient-physician relationships," Google, Oct. 8, 2026, https://blog.google/innovation-and-ai/technology/health/amie-clinical-study-lancet/
- Jacqueline Mitchell, "A Real-World Study of Patient-Facing AI in Primary Care," Harvard Medical School, Oct. 13, 2026, https://hms.harvard.edu/news/real-world-study-patient-facing-ai-primary-care
- Martha Bebinger, "Your next primary care doctor could be online only with an AI partner," WBUR, Dec. 22, 2025, https://www.wbur.org/news/2025/12/22/massachusetts-hospitals-pcp-shortage-artificial-intelligence-care-connect
- Staff, "AMA: AI usage among doctors doubles as confidence in technology grows," American Medical Association, Mar. 12, 2026, https://www.ama-assn.org/press-center/ama-press-releases/ama-ai-usage-among-doctors-doubles-confidence-technology-grows
- Staff, "Doctors work fewer hours, but the EHR still follows them home," American Medical Association, Aug. 19, 2025, https://www.ama-assn.org/practice-management/physician-health/doctors-work-fewer-hours-ehr-still-follows-them-home