ChatGPT, the artificial intelligence (AI) chatbot developed by OpenAI and launched in November 2022, is now proving to be quite adept at responding to healthcare related queries by people. Studies indicate that the responses provided by ChatGPT are nearly indistinguishable from those provided by human healthcare providers.
A new study by researchers at New York University’s (NYU) Tandon School of Engineering and Grossman School of Medicine reveals that artificial intelligence powered chatbots have the potential to become effective partners for healthcare providers to communicate with their patients.
For their study, the researchers provided around 400 people aged 18 and above with responses to health related questions from patients. Half of the responses were generated by a human healthcare provider and the other half by ChatGPT.
Participants in the study were asked to identify the source of each response, and to rate their trust in the ChatGPT responses using a 5-point scale from completely untrustworthy to completely trustworthy.
The study found people have limited ability to distinguish between chatbot and human-generated responses. On average, participants correctly identified chatbot responses 65.5 percent of the time and provider responses 65.1 percent of the time, with ranges of 49 percent to 86 percent surety for different questions. Results remained consistent no matter the demographic categories of the respondents.
The study found participants mildly trust responses by chatbots overall (3.4 average score), with lower trust when the health-related complexity of the task in question was higher. Logistical questions (e.g. scheduling appointments, insurance questions) had the highest trust rating (3.94 average score), followed by preventative care (e.g. vaccines, cancer screenings, 3.52 average score). Diagnostic and treatment advice had the lowest trust ratings (scores 2.90 and 2.89, respectively).
According to the researchers, the study highlights the possibility that chatbots can assist in patient-provider communication particularly related to administrative tasks and common chronic disease management. Further research is needed, however, around the ability of chatbots to take on more clinical roles.
However, researchers behind the study warned that both patients and healthcare providers should remain cautious and exercise critical judgment when curating chatbot-generated advice due to the limitations and potential biases of current AI models.
On a related note, researchers behind a separate, international multi-disciplinary collaborative study on artificial intelligence (AI) noted that it would soon be possible to predict our risk of developing serious health conditions later in life, with the aid of AI powered software.
Studies, led by researchers at Edith Cowan University’s (ECU) School of Science and School of Medical and Health Sciences in the United States, along with their counterparts in other universities and research centers in the US and Canada, led to the development of an AI software that could detect abdominal aortic calcification (AAC) much more faster than current methods.
Abdominal aortic calcification is a medical condition caused by calcium buildup, or calcification, on the inner walls of the abdominal aorta, the main artery from the heart to the abdomen. Calcification in this blood vessel is a marker used by doctors to evaluate the risks associated with cardiovascular disease events such as heart attacks and stroke, as well as risks of falls, fractures and late-life dementia.
Currently, bone density machine scans that help detect osteoporosis are also used to detect AAC. But analysis of these scans to detect AAC requires highly trained experts, who could take anywhere up to 15 minutes to scan each image. Even the best of experts are able to analyze less than 30 images in an eight-hour work day. In contrast, researchers behind the new software say that their AI-enabled system could analyze up to 60,000 images in a single day.
The significant boost in efficiency from the use of the new software is expected to speed up further research into AAC that could lead to new approaches in early detection of cardiovascular diseases, and in monitoring for the disease during routine clinical practice. Although the new software is not the first algorithm developed to assess AAC from scans, this study is the biggest of its kind and is the first to be tested in a real-world setting using images taken as part of routine bone density testing.
More than 5,000 images were analyzed by experts and the team’s software. After comparing the results, the expert and software arrived at the same conclusion for the extent of AAC (low, moderate or high) 80 percent of the time — an impressive figure given it was the first version of the software. More importantly, only 3 percent of people deemed to have high AAC levels were incorrectly diagnosed to have low levels by the software. The small error level is notable considering that these are the individuals with the greatest extent of disease and highest risk of fatal and nonfatal cardiovascular events and all-cause mortality.
The researchers admitted that there was more work to be done on fine tuning their software to improve its accuracy compared to evaluations by trained practitioners.
Automated assessment of the presence and extent of AAC with similar accuracy to imaging specialists provides the possibility of large-scale screening for cardiovascular disease and other conditions, even before someone has any symptoms. This will allow people at risk to make the necessary lifestyle changes far earlier and put them in a better place to be healthier in their later years.