Thursday, February 22, 2024

Researchers found ChatGPT incorrectly diagnosed over 8 in 10 selected pediatric case studies, raising questions about some bots' suitability for helping doctors size up complex conditions.

The HEADLESS M.D.

The big picture: Large language models like OpenAI's ChatGPT are trained on massive amounts of internet data and can't discriminate between reliable and unreliable information, researchers at Cohen Children's Medical Center wrote.

They also lack real-time access to medical information, preventing them from staying updated on new research and health trends.

What they found: The chatbot misdiagnosed 72 of 100 cases selected and delivered too broad a diagnosis to be considered correct for another 11, the researchers wrote in JAMA Pediatrics.

It wasn't able to identify relationships like the one between autism and vitamin deficiencies, underscoring the continued importance of physicians' clinical experience.
However over half of the incorrect diagnoses (56.7%) belonged to the same organ system as the correct diagnosis, indicating more selective training of the AI is needed to get diagnostic accuracy up to snuff.
The study is thought to be the first to explore the accuracy of bots in entirely pediatric scenarios, which require the consideration of the patient's age alongside symptoms.

One takeaway is that physicians may need take a more active role in generating data sets for AI models to intentionally prepare them for medical functions — a process known as tuning.

ChatGPT had a high error rate for pediatric cases

9 Challenges for CMOs in 2024 | Insights | Traction

Challenge #1: Picking the Right AI

According to theresanaiforthat.com, there are currently 10,861 AIs for 9,962 tasks and 4,847 jobs in their database. There are an estimated 57,000+ AI companies in the world today with at least 15,000 of them in the United States

No matter which way you cut it, that’s a lot.

But not all AIs are created equal. One of the biggest obstacles to AI adoption will be choice. Picking the best tools for the right use cases and using them in the right ways to get the best output is an opportunity for competitive advantage for savvy brands. While the pace of innovation means that what’s best today may not be best in a week.

Take a look at the output for the exact same prompt in MidJourney and Adobe Firefly and you’ll get dramatically different results.

Woman generated by MidJourney

Woman generated by Adobe Firefly

ChatGPT is great at writing emails or blog posts, but Pi.ai has a more natural human voice, Perplexity.ai offers sources to validate its responses, and Jasper.ai allows you to export copy for dozens of marketing formats at once.

CMOs should create a disciplined approach to identifying the most meaningful use cases, evaluating tools, documenting workflows, training teams to use them, driving the adoption of best practices among teams, and monitoring emerging innovations to continue to optimize performance.

Much of this waste comes from redundant platforms gathered over time. In a recent martech audit for one of our clients at Traction, we uncovered $3.5M in potential annual savings by consolidating vendors.

Another reason is that people either don’t know how to use the capabilities they have or don’t want to. In a conversation over drinks with a senior marketing director at a large brand, she shared with me. “We paid all this money for all this tech and no one uses it because it’s a pain in the ass.”