Alexa, Siri, Cortana and Google Assistant have a long way to go before they get their MDs, according to research by Grace Hong, a social science researcher in the Stanford Healthcare AI Applied Research Team.
The study, which was published in the Annals Family Medicine last Fall, showed that, for now, we can't rely on their health advice. That's unfortunate, Hong has explained in an article published by Stanford University's Human-Centered Artificial Intelligence group. "As voice assistants become more ubiquitous, we need to know that they are reliable sources of information -- especially when it comes to important public health matters."
To evaluate the accuracy of voice assistants, five scientists asked Siri, Microsoft's Cortana, Amazon's Alexa, and Google Assistant whether they should get screened for 11 different types of cancers.
First, the team noted whether the answers they received were verbal or if the voice assistant provided only a list of web pages, as verbal answers are more accessible to people who are blind or have difficulty reading. Then, they compared the advice that the voice assistants gave -- regardless of whether the suggestions came verbally or via the websites -- to the U.S. Preventive Services Task Force's guidelines for cancer screening.
Hey Siri ...
Alexa frequently struggled, often responding, "Hm, I don't know that." And Siri was regularly unable to verbalize the answer, but did provide web search results, while Google Assistant and Microsoft Cortana provided both verbal and web-based responses. The websites were more likely to match the federal guidelines than the assistants' verbal responses were, providing an accurate age for cancer screening in response to about 60-70% of the requests. Google's Assistant maintained that level of accuracy verbally, but Cortana spoke the answer correctly only 45% of the time.
Other researchers conducted similar studies, asking their phone assistants about the safety of vaccines (pre-COVID-19), for example. That team found similarly concerning results, though on these questions, Siri and Google Assistant outperformed Alexa.
Last year, Hong and her team worked on a follow up study looking at how voice assistants respond to, "Should I get a COVID-19 vaccine?" and, "Are the COVID-19 vaccines safe?" They hope to learn more about how these voice assistants operate "in the wild," as people ask varied or personalized health questions, and what role they may play in the spread of health misinformation.
All this is to say, if you think you might be sick, or have general questions about your health, you're much better off calling your health care provider or visiting a clinic than asking your smart devices -- at least for now.
Still, Hong has hope: "These results suggest there are opportunities for technology companies to work closely with health care guideline developers and health care professionals to standardize their voice assistants' responses to important health-related questions."
Photo by Omid Armin
