Healthcare’s AI problem isn’t technology – it’s trust

Not long ago a novelty in healthcare, artificial intelligence is now an everyday tool for functions ranging from reading scans to drafting clinical notes to managing hospital workflows. There are huge potential advantages in the form of efficiency, quicker diagnoses and – most important of all – better outcomes for patients. But like many new technologies, AI has arrived at hospitals faster than the people fully understand the implications, and this poses issues relating to public trust.

Like human beings, artificial intelligence makes mistakes, and these can have profoundly negative consequences when it comes to healthcare. So how does the public react when AI is involved in adverse events at hospitals?

Research co-authored at Cambridge Judge Business School finds that people attribute more responsibility and are more likely to pursue legal action when adverse hospital events involve AI compared to the same events involving only a human doctor. These reactions are attenuated, however, when people are informed that doctors interactively collaborate with AI.

“We think the findings, in which people are more negative toward the hospital when they find the problem relates to AI, reflects wider issues involving algorithm bias that we have also looked at in other research about AI,” says hospital-AI study co-author Yeun Joon Kim, Associate Professor in Organisational Behaviour at Cambridge Judge. “This algorithm bias manifests in people thinking that AI has no ability to diagnose their own particular situation, yet hospitals engage with AI on many diagnostic matters.

“AI can make mistakes, just as human doctors can, but the public reacts less negatively to these mistakes when physicians are seen as actively involved in how AI is used to diagnose patient conditions. This is important because, in real healthcare settings, AI is rarely used in isolation: physicians remain deeply involved in medical decision-making even when AI tools are used to support diagnosis or care.”

AI errors in healthcare can damage patient trust and expose hospitals to legal risk

The research authored by Joon and colleagues defines adverse events as “unintended injuries caused by medical management rather than the patient’s disease process”, and notes that such events have traditionally stemmed from human error including a breakdown in coordination among health professionals. But as AI is increasingly used in hospital workflows, there is the risk that faulty or incomplete AI outputs are applied to patients and result in injury.

“For example, diagnostic AI models may rely on spurious patterns or misinterpret images, producing misleading diagnostic outputs that can delay appropriate care and contribute to patient harm,” says the research published in the journal npj Digital Public Health. “Moreover, large language models may downplay symptoms for certain demographic groups when generating clinical case notes, which can influence care decisions in ways that disadvantage patients.”

For hospitals, such errors can carry serious consequences including reduced patient trust, reputational damage and lawsuits. The study notes that in the US about 17,000 malpractice lawsuits are filed annually, totalling $4 billion in payouts to patients.

“Taken together, AI-involved adverse events highlight a paradox: although hospitals adopt AI to reduce operational burdens and enhance clinical performance, adverse events involving AI may nonetheless carry reputational, financial, and legal costs that could undermine these anticipated gains,” says the research. “. Despite these potential risks, little is known about how the public responds when these events occur. Much of existing literature centres on adverse events involving human factors and potential interventions aimed at preventing their occurrence.”

Visible physician oversight helps build trust in healthcare AI

The research on AI and adverse events at hospitals is divided into 2 separate studies: the first study examines whether the public responds more negatively to hospitals if adverse events are attributed to AI, a human doctor, or both, and then a companion study looks at these reactions when doctors and AI engage in different types of doctor-AI collaboration.

These different types of collaboration are important because they vary in the degree of physician involvement. In autonomous collaboration, AI performs the primary task, with the physician playing only a limited oversight role. In sequential collaboration, AI first generates an output, which the physician then reviews or builds upon. In interactive collaboration, by contrast, the physician and AI work together more closely, with the physician actively integrating AI input into their own independent assessment.

The research finds that this interactive form, which involves the highest level of physician involvement, leads participants to attribute significantly less responsibility to the hospital and report a lower likelihood of filing complaints. “These results suggest that public responses to AI in healthcare may depend less on the technology itself and more on how its role is structured and integrated into clinical care,” the authors say.

The practical implications of the research include a suggestion that hospitals may benefit from presenting AI systems as part of a collaborative clinical process, rather than as standalone decision-makers. Hospitals are adopting AI to improve efficiency and clinical performance, while the public may react more negatively when AI use is not accompanied by visible and comprehensive physician involvement. The study therefore calls for further research into communication and workflow strategies that can help hospitals capture the benefits of AI while preserving public confidence in physician oversight.

The team behind this hospital research includes Joon and academic colleagues from the University of Cambridge, University of Queensland and the University of Toronto.

Distrust of AI health chatbots can reduce the quality of symptom reporting

A separate study, published in Nature Health and co-authored by Joon and academics from several German universities, examines how people use AI chatbots for medical advice, offering a consumer-facing view of AI in healthcare.

The researchers found that participants who believed they were interacting with an AI tool rather than a human doctor provided “lower-quality symptom reports” for a fictional medical triage situation. Reflecting the algorithm bias also identified in the other research, the findings “could compromise the performance of consumer-facing AI tools in real-world applications, regardless of the underlying model’s actual capacity”, say the authors.

One reason these findings matter, they add, is because the degradation of input-stage information “operates upstream of any clinical algorithm, rendering it invisible to standard model benchmarks”.

“Although the observed effects are of moderate size at the individual level, they could be highly meaningful at scale. This especially applies to consumer-facing applications, where millions of users may rely on AI chatbots or LLM-based assistants for an initial medical urgency assessment before contacting clinicians. In such settings, users’ belief that they are interacting with AI could lead to less suitable symptom reports, resulting in less accurate triage recommendations or risk assessments, regardless of the underlying model’s performance.”

Patient distrust of AI remains poorly understood

“We found that patients who engaged with AI chatbots don’t give enough information about their own symptoms, and this puts them at a disadvantage,” says Joon. “They provide low-quality information or less information because they don’t trust AI, which reflects algorithm aversion. We know the phenomenon but we don’t properly understand why people have such bias.

“Previous research shows that people tend to believe their own disease is unique, and that’s an innate bias, so when they interact with AI they don’t think AI has the ability to diagnose their particular disease. AI does, however, have the capability to collect information from patients, and this information gathering could be much better than what humans can do in terms of errors.”

The benefits of AI-human interaction to boost creativity

Joon says he was not surprised by the finding that people may put themselves at a disadvantage by providing lower-quality information to AI chatbots. A similar pattern appears in his previous research on augmented learning, published in the journal Information Systems Research, which shows that people often fail to make full use of powerful AI technologies.

In that study, Joon found that when people used AI to generate creative ideas over multiple rounds, they did not necessarily become more creative. This was partly because they used AI in overly simplistic ways, or because they relied too heavily on AI during the idea-generation process.

The research introduces a new theory of augmented learning, which explains how humans and generative AI can improve joint creativity over time through a process of learning how best to work together.

“We know from research on automation bias that people often rely on capable machines when they believe those machines can produce good results, even if they do not necessarily regard those machines as intelligent,” says Joon. “But our research shows that human-AI interaction can still produce creative benefits over time. The key is not simply to use AI for one fixed purpose, such as generating ideas or giving feedback.

“Instead, humans and AI need to learn how to divide, perform and recombine different parts of the creative process. Through this repeated process, human-AI teams can gradually discover where each side adds the most value. That is what we mean by augmented learning: a process through which humans and AI come to understand each other’s strengths and limitations in a particular task, and use that understanding to rearrange co-creation activities in ways that improve performance.”

In this sense, the unease toward AI identified in the healthcare research may reflect, at least in part, a lack of augmented learning: people may not yet have a clear understanding of what AI can and cannot do.

“People’s biases toward AI may be reduced if they have opportunities to develop a deeper understanding of the technology through sustained and meaningful interaction over time,” says Joon. “AI has limitations and biases, but it also has strengths, just as humans do. Of course, we should continue working to reduce AI’s limitations. At the same time, we may also need to acknowledge those limitations and consider how human strengths can be built into the broader process of human-AI interaction.”

While these various AI-related studies co-authored by Joon focus on discrete elements, there is a common thread: the focused involvement of humans in AI activities – be it hospital diagnostics, use of a health chatbot or engaging to boost creativity – hold positive benefits for individuals and society alike.