[ad_1]
A recent study published in the open access journal PLOS Digital Health showed that ChatGPT demonstrated the ability to perform at or around the 60% passing level on the United States Medical Licensing Examination (USMLE). The study, conducted by Tiffany Kung, Victor Teng and colleagues at Ansible Health, found that ChatGPT responses were consistent, intuitive and often contained insightful information. The results of this study indicate that ChatGPT has the potential to make a significant impact in the field of medicine and health care.
The AI software was able to achieve passing scores for the exam, which usually requires years of medical training.
OpenAI’s ChatGPT for the United States Medical Licensing Exam (USMLE) can achieve responses that contain consistent, intuitive, and repeatable insights, at or around the 60 percent passing threshold. This is according to a study by Tiffany Kung, Victor Teng and colleagues at Ancil Health, published in the open access journal February 9, 2023. PLOS Digital Health.
ChatGPT is a new artificial intelligence (AI) system known as a large-scale language model (LLM) designed to generate human-like text by predicting future word sequences. Unlike most chatbots, ChatGPT cannot search the Internet. Instead, it generates text using word relationships predicted by its internal processes.
Kung and colleagues tested the CHAT GPT’s performance on the USMLE, the most standardized and controlled series of three exams (Levels 1, 2CK, and 3) for medical licensure in the United States. Taken by medical students and physicians-in-training, the USMLE assesses knowledge of many areas of medicine, from biochemistry to diagnostic reasoning to bioethics.
After filtering to remove image-based questions, the authors tested the software on 350 of the 376 public questions available in the June 2022 USMLE release.
After indeterminate responses were removed, ChatGPT scored between 52.4% and 75.0% on the three USMLE tests. The annual pass limit is approximately 60%. ChatGPT showed 94.6% concordance across all responses and provided at least one significant insight (new, ambiguous, and clinically valid) for 88.9% of responses. Specifically, ChatGPT outperformed PubMedGPT, trained only on the biomedical domain literature, scoring 50.8% on an older USMLE-style question set.
Although the relatively small sample size limited the depth and breadth of the analyses, the authors note that their findings provide insight into the potential of ChatGPT to enhance medical education and ultimately clinical practice. For example, clinicians at AnsibleHealth already use ChatGPT to rewrite jargon-heavy reports for easier patient understanding.
“Achieving a passing score for this notoriously difficult expert test and doing so without any human reinforcement is a significant milestone in the maturation of clinical AI,” the authors said.
Author Dr. Tiffany Kung added that ChatGPT’s role in this study goes beyond being a research subject, “ChatGPT has made significant contributions to the literature.” [our] Manuscript … We met with ChatGPT as a colleague, asking him to complete, simplify, and provide counterpoints to drafts in progress … All co-authors value ChatGPT’s input.
Reference: “ChatGPT’s Performance on the USMLE: Potential for AI-Assisted Medical Education Using Large Language Models” by Tiffany H. Kung, Morgan Chatham, Ariel Medinilla, Zarina Silos, Lori De Leon, Camille Elepano, Maria Madriaga, Rimel Agabao, Giselle Diaz- Candido, James Maningo and Victor Teng, 9 February 2023; PLOS Digital Health.
DOI: 10.1371/journal.pdig.0000198
As a test, we asked ChatGPT to summarize this article. Here are the unedited results:
Research by Tiffany Kung, Victor Teng, and colleagues at Ancil Health showed that OpenAI’s ChatGPT can perform at a 60% passing rate on the United States Medical Licensing Examination (USMLE). ChatGPT is an artificial intelligence language model that generates text by predicting word sequences, but does not have the ability to search the Internet. The researchers tested ChatGPT’s performance on 350 of 376 public questions from the June 2022 release of the USMLE and found that it scored 52.4% and 75.0% on the three tests. The study showed that ChatGPT yielded at least one significant insight for 88.9% of the responses, and PubMedGPT outperformed the model trained only on the biomedical literature. The authors believe that ChatGPT has the potential to improve medical education and ultimately clinical practice.
[ad_2]
Source link
