ChatGPT’s latest version aced the US medical licensure exam and identified a 1 in 100,000 ailment in seconds 2023

Dr. Isaac Kohane, a Harvard computer scientist and physician, tested GPT-4 with two colleagues to evaluate OpenAI’s latest artificial intelligence model in a medical environment.

“I’m surprised to say: better than many physicians I’ve watched,” he adds in the forthcoming book “The AI Revolution in Medicine,” co-authored by independent journalist Carey Goldberg and Microsoft vice president of research Peter Lee. (Microsoft has invested billions in OpenAI’s technology, but the authors say neither required editorial review of the book.)

“Like me,” he said, “the chatbot can figure out strange diseases.”

Kohane claims in the book that GPT-4, issued to paying subscribers in March 2023, answers US medical test license questions accurately over 90% of the time. It’s a better test-taker than GPT-3 and -3.5, and even some qualified physicians.

GPT-4 is more than a good test-taker and fact-finder. That translates well. In the book, it translates discharge information for Portuguese-speaking patients and simplifies technical jargon for 6th graders.

As the authors demonstrate using vivid examples, GPT-4 can also provide clinicians advice on bedside manner, including how to communicate to patients about their diseases in compassionate, simple language, and it can quickly summarize extensive reports or research. The tech can even explain its problem-solving in a human-like manner.

GPT-4 would likely say that its intelligence is “limited to patterns in the data and does not include actual understanding or intentionality” if you inquire how it accomplishes this. When asked if it could reason causally, GPT-4 informed the book’s creators that. As Kohane found in the book, GPT-4 may replicate doctors’ diagnoses with surprising accuracy despite its limitations.

GPT-4’s doctor-like diagnosis.

In the book, Kohane does a therapeutic thought experiment using GPT-4 modeled on a newborn infant he treated years before. “Just like I would, with all my years of research and expertise,” Kohane said, the bot was able to accurately diagnose congenital adrenal hyperplasia, a 1 in 100,000 disorder.

Doctor was impressed and terrified.

“On the one hand, I was having a sophisticated medical conversation with a computational process,” he wrote, “but just as mind blowing was the anxious realization that millions of families would soon have access to this impressive medical expertise, and I could not figure out how we could guarantee or certify that GPT-4’s advice would be safe or effective.”

GPT-4 makes mistakes and lacks morality.

The book shows GPT-4’s mistakes. These vary from basic clerical blunders like misstating a BMI that the bot had correctly computed seconds ago to math mishaps like incorrectly “solving” a Sudoku puzzle or failing to square a variable in an equation. The system typically defends its faults, which are often subtle. A mistaken number or weight might cause catastrophic prescription or diagnosing problems.

GPT-4 can “hallucinate,” or lie or disobey, like prior GPTs.

GPT-4 replied to the book’s writers about this “I occasionally make mistakes or assumptions based on insufficient or erroneous data. I also lack clinical judgment and ethics.”

The book suggests starting a new session with GPT-4 and having it “read through” and “verify” its own work with “fresh eyes.” Since GPT-4 is reluctant to accept faults, this strategy occasionally succeeds. To catch errors, ask the bot to show you its work.

GPT-4 might free up clinic time and money, allowing physicians to spend more time with patients “instead of their computer screens,” the authors write. They add, “We must picture a world with better machines, maybe surpassing human intellect in practically every way. Think deeply about how we want that world to operate.”