A novel tool to assess patient experience, developed at the Psychometrics Centre at Cambridge Judge Business School, enters advanced pilot stage at Brigham and Women’s Hospital in Boston.
Like other service-oriented institutions, many hospitals ask patients to rate the quality of their experience after attending an appointment. But such survey results are often of little use because patients do not feel comfortable to criticise doctors, nurses and others trusted to make them feel better or even keep them alive.
“We found in research in the UK that about 60 per cent of people who say in a questionnaire that their hospital service was ‘good’ contradicted themselves when we spoke to them later,” says Dr Chris Gibbons of the Psychometrics Centre at Cambridge Judge Business School. “We see the same thing in the US: scores on a five-point scale that range from 4.3 to 4.8 don’t provide much useful information.”
Free text responses given by patients can often contain a more truthful account of their experience, however this data is rarely used due to the complexity of digitising, analysing and summarising textual information at scale. That’s why the Psychometrics Centre hasdeveloped a new digital assessment tool called INSPIRES that goes beyond the usual “good” to “very good” responses normally elicited by hospital questionnaires.
The system uses a three-screen online questionnaire that first asks people to identify their current emotion by pressing one of 12 buttons (such as “happy”, “sad”, “thankful”, “worried” or “frustrated”); then asks people to write in free text why they feel that emotion; and then finally asks patients to identify what made them feel that way (push-button choices include “waiting times”, “cleanliness”, “care provider” and “check-in staff”, or they can type in their own response).
The value of the INSPIRES system lies in how it analyses and tags this data in the background to create clear and actionable reports to administrators. Reported emotions are time-stamped, for example, giving insight into how patient experiences change over time or subsequent to specific interventions. The system, built using the open source Concerto platform, also uses machine learning and natural language processing to codify free text responses into “buckets” representing topics of interest, and then links those topics to emotional sentiment.
While continuing his role as Director of Health Assessment and Innovation at the Psychometrics Centre, Dr Chris Gibbons in 2018 joined the faculty at the Department of Surgery at the prestigious Brigham and Women’s Hospital in Boston, Massachusetts, where he helped implement a major pilot project to assess how INSPIRES can improve patient care.
After introducing INSPIRES late last year to four of 12 outpatient surgery clinics (including urology and general surgery), Brigham and Women’s plans to trial the system in all 12 clinics in 2020 – providing weekly feedback to the nine division heads.
“It looks very promising so far,” says Dr Danny Mou, a resident surgeon at Brigham and Women’s whose research focuses on patient-reported outcome measures (PROMs). “We’re quite excited to see what sort of data we’ll get back, and our hope is that we have actionable data. Rather than ask patients ‘how was your doctor today?’, the platform asks patients to first commit to an emotion, then justify the emotion, and then identify what caused them to feel this emotion.”
The machine learning application has already found that patients who see people eating food near the front desk makes the hospital seem like an unhealthy environment. This sort of issue would probably never have been asked about directly in a questionnaire, yet it’s something that patients felt happy to mention in a free text box.
Taken together, the system’s processes provide useful information to hospital administrators about what specific factors may be influencing the experiences of patients, rather than merely giving a general sense of whether scores are positive or negative. (For example, administrators could see that visitors to their surgery felt both “thankful” for short waiting times and “frustrated” by the lack of parking spaces.)
As the system also picks up on new linguistic topics automatically, it could also detect more complex associations such as an increase in “worried” feelings about a topic that patients have opinions on but which they have not specifically been asked about. This makes the system more likely to fill the gaps in information that result from traditional surveys, flagging up new areas to administrators and thus having a greater overall sensitivity to the diverse factors impacting patient experience. Administrators can also customise the look and feel of the tool to define a distinct set of reportable emotions and event tags for each department. This helps to make the assessment more relevant and hopefully less intimidating for patients.
“We have clearly identifiable items in this platform,” says Dr Aska Arnautovic, Research Fellow at Harvard Medical School who is also working on the Brigham and Women’s pilot. “Beyond doctors and nurses, there is the parking situation, cleanliness, waiting times – the sort of things that people are more likely to talk about through this app, and the type of things we can fix more easily.”
Initially, Brigham and Women’s asked patients to participate in INSPIRES through leaflets and other passive means, but will soon begin texting all outpatients with a link to the INSPIRES online questionnaire that can be completed in just a few minutes on their mobile phone.
Dr Chris Gibbons recently took up a new position at the University of Texas MD Anderson Cancer Center, where he will institute a similar trial of INSPIRES this year. He’ll be back at Cambridge Judge this summer to teach the Psychometrics Centre’s annual data science summer school.
“We hope to adapt and roll out versions of INSPIRES to more hospitals in the coming years,” says Vesselin Popov, Executive Director of the Psychometrics Centre. “By presenting automatic reports and making free text data actionable for hospitals, we believe it has great potential to improve patient experiences and pave the way for further innovation in the health sector.”