Mortality rates don't tell the whole story

I have just finished reading the astonishing book by psychologist Daniel Kahneman Thinking, Fast and Slow − it is a book that takes you on a journey of 30 years of discovery in psychological science. Once you read it you will never “think” about yourself the same way again. The central tenet of the book is particularly salient to the conversation in the health service about mortality and the statistics relating to it following the publication of the Francis report.

‘One of the central themes of Mid Staffs crisis centres on the role, meaning and response to mortality statistics’

Mr Kahneman describes three dichotomous concepts. First, two types of thinking “system 1”, which is fast, intuitive, associative, innate and effortless; and “system 2”, which is low, analytical, calculating, learning and effortful.

Second, he describes two types of people “econs”, who are rational, consistent, logical, ideal economic agents that always make the “right” choice; and “humans”. These are reasonable but subject to biases of their thinking such as priming, framing, narrative fallacy, imabalanced attitudes towards risk of loss and gain, excessive weighting to “available” evidence, and relatively blind to statistics and the “external” view.

Third, there are two “selves”: the experiencing self (the person reading this article here and now) and the remembered self (the person your mind has created through the narrative stitching of remembered events and experiences).

Spectacular corporate failure

What is particularly interesting about his research is that he has demonstrated unequivocally time and again that being an “expert” in any field, be it healthcare, economics or even psychology itself, does not protect you from these innate biases of human thinking − even when you know they exist they still influence you, and in fact being an expert simply puts you in a position where these biases are more consequential (your biases harm other people as well as yourself).

He does, however, describe a number of strategies that organisations can adopt that defend against the consequences of individual judgment bias, strategies that need to be adopted systematically and deliberately. He gives examples of spectacular corporate failure, through “group think”, where these strategies had not been adopted.

Why is this all relevant to us following the Mid Staffordshire crisis, Francis report and the renewed scrutiny of hospitals with “outlying” mortality figures? In the pantheon of spectacular corporate failure, the events in 2007 at Mid Staffordshire Foundation Trust must rank prettily highly, and the principle theme of the second Francis report is that this was an NHS corporate failure, not just a hospital failure. One of the central themes of the crisis centres on the role, meaning and response to mortality statistics for the hospital at the time. It is worth, therefore, taking a little time to understand how mortality statistics are generated.

At the time there was a single statistical measure routinely used to compare mortality in hospitals across the NHS − the hospital standardised mortality ratio (HSMR).

How the ratings work

HSMR is calculated by taking the observed death rate and dividing it by a calculated figure for an “expected” death rate. The expected death rate is essentially the death rate in hospitals for the whole of the UK with a number of adjustments made in order to compensate for differences in case mix (the diagnoses patients are admitted with), age, co-morbidity and social deprivation.

‘An important thing to recognise about mortality statistics is that they are just that: statistics’

The ratio is multiplied by 100, so if your observed death rate is the same as the expected then the ratio is 1.0 and the HSMR would be 100. When observing mortality in a population it is important to recognise that over time mortality will always increase (we all die eventually). Therefore when comparing mortality in different populations we have to know over what period mortality has been counted (usually 30 days, 90 days, one year or five years).

One of the weaknesses of HSMR is that it does not specify the period − it is simply the time spent in hospital − therefore a hospital with a longer than average length of stay will tend to have a higher mortality. Another weakness of HSMR is that it does not have a diagnostic category code for palliative care. The provision of out of hospital end of life services across the UK is at best patchy. Where provision is poor patients are admitted to hospital to die; HSMR does not adjust for these expected deaths and so hospitals in this situation will have a higher than “expected” HSMR.

As a result of these statistical biases, HSMR is being replaced by a newer indicator − the summary hospital mortality indicator (SHMI). This is very similar to HSMR but differs in two important aspects. First, it expands the number of diagnostic codes against which case mix adjustment takes place from 56 to 140, including one for palliative care. Second, it fixes the number of days over which mortality is measured to 30 days after discharge.

The other important thing to recognise about mortality statistics is that they are just that: statistics. As such reliability depends on sample size and confidence intervals. As a rule of thumb a condition with an expected mortality rate of 10 per cent needs a sample size of about 200 cases before the confidence interval falls to a point where a doubling (or halving) of the observed rate can be explained by anything more than random chance. This is best illustrated by a “funnel plot” graph, which is designed to check for publication bias in statistical reviews.

Two stories about mortality

“Hospital A” is a medium-sized district general hospital in a provincial town in the Midlands. It is more than 30 miles from the nearest city and serves a relatively fixed population, bound by open countryside, which is below the threshold for sustaining a comprehensive range of hospital services. It has recently successfully achieved foundation trust status by importing a new board that have railroaded a controversial savings program through that has helped make the long-term finances appear sustainable.

‘There is a deeply held prejudice among leaders and policy makers in the health service: “big is good, small is bad”’

Many staff have been made redundant and staffing ratios on the wards have been reduced. Morale is low and sickness rates are high, the wards are busy with large numbers of elderly dependent patients. The hospital has always struggled with attracting high-quality medical staff because of its location, tenuous affiliation with a university hospital, low numbers of doctors in training and an unexciting specialty portfolio.

There have been a number of recent complaints from relatives unhappy about basic standards of care on the wards. Two of these have been reported in the local newspaper, with the suggestion that neglect of care was a contributing factor in the deaths of patients. The HSMR for this hospital is 127, the second highest in the country.

“Hospital B” is a large university trust in a major city in England. It has a number of highly specialised tertiary services for which it has a national and international profile. One of these is complex paediatric congenital heart surgery, for which it is one of only a dozen centres in the country. The unit is celebrated locally and has a loyal following of patients and their parents who have received treatment there. The surgeons often take on cases others have refused and in doing so avert what would have otherwise been certain death.

The unit has been threatened with closure as a result of a national consultation on re-configuration of paediatric cardiac surgery − aiming to concentrate services from eleven to seven centres. The local newspaper is outraged and has rallied support from local and national celebrities and politicians to keep the unit open. The staff are highly motivated and capable and the surgeons have published acclaimed original research in international journals. The SHMI for paediatric heart surgery at the hospital is 200, the second highest in the country.

Beside ourselves jumping to conclusions

Death is a powerful word − just reading it on the page is likely to result in both a physiological response (increased heart rate, blood pressure and dilated pupils) and an emotional one (fear, disgust and aversion). These responses are usually rapidly attenuated by the rational part of the brain (system 2), however the alertness brought on by the physiological and emotional response will have activated system 1 − your innate, intuitive, fast-thinking brain will be in overdrive (primed to deal with the ‘threat’) as will all of its biases.

So what are you thinking about the two (completely fictional) hospitals above? Where is your outrage directed in each of the stories? Would you allow your grandmother to have her fractured hip treated at hospital A? Would you recommend a friend have their child with VSD be operated on at hospital B? Are you angry with the hospital or the system that is trying to close it down?

Before you answer those questions it is worth taking some time to reflect on how your thinking may be being manipulated:

• First of all, priming − the stories I have written have a ring of familiarity to them, you have made an association with a previous experience for which you know the outcome, however hard you try to avoid it, this will be influencing your thinking in both situations.
• Second, framing − I have (rather clumsily from a literary perspective) set the scene in both stories in a way that influences your thinking, I have told you what some other people think about those hospitals and I have given you some limited facts that probably make you like or dislike each of them.
• Third, narrative fallacy − I have told stories that “explain” the numbers. We all like stories and are primed to discern patterns in randomness − there is a multibillion pound industry built on this tendency called professional sport.
• Fourth, availability bias − this is also variously known as positive reporting bias and/or economy with the truth. It is ruthlessly exploited by the pharmaceutical industry through the suppression of non-supportive evidence for the efficacy of drugs. They are by no means the only culprits though, most published academic research is subject to this bias as well. Even though you “know” I haven’t told the whole story in both cases and can easily entertain the idea that there may be other facts, unreported, which could change your view of the situation, your view is firmly anchored by what I have told you so far and new evidence can only move you from this position.
• Finally, statistical blindness − the fact is that while system 1 thinking is life saving, allowing you to act quickly, decisively and intuitively in almost all everyday situations (and many non-everyday situations), it is designed to jump to conclusions, to abide by rules of thumb and accept evidence, particularly statistical evidence, at face value. As much psychological weight is given to a statistic based on a handful of cases as many thousands even when there is no mathematical justification for doing so.

This last point is particularly pertinent to my two stories, because they both have one thing in common − they are both “small” − one is a small district general hospital, the other is a small highly specialised unit. Their outlier status is almost certainly more to do with their smallness than their quality of care, though this should not be ruled out of hand either.

Prejudice fuelled by bias

Buried within the shrill, insistent and pervasive criticism of the Mid Staffordshire is a deeply held prejudice among leaders and policy makers in the health service: “big is good, small is bad”.

‘The conversation about mortality in hospitals needs to be held in an intelligent, unfrenzied, non-political environment’

This prejudice is affirmed and reaffirmed on a regular basis by the apparent evidence of poor performance and outcomes − the smaller you are the more likely you are to find yourself at the top or bottom of a league table. The fact that this phenomenon is a mathematical inevitability is either not recognised or overlooked because it suits “the system”, it provides supporting “evidence” that not only are small services expensive they offer poor value as well.

Big of course is not inevitably good either − one of the advantages of being a large institution is that you experience the converse of the “small outlier” phenomenon in your outcome statistics; you are generally a “large average” institution, you routinely find yourself (reassuringly) “in the pack”.

Reassured you shouldn’t be though − hiding in every large institution’s aggregated outcome statistics will be some great performance and some dreadful. Some of the shrillness of the commentary will be disguising the fear that every leader holds − that within their own institutions are lurking little bits of Mid Staffs.

The mortality figures at Mid Staffordshire were probably the least surprising and least relevant part of the story. A great deal did go wrong at the hospital, particularly at board level where there is much evidence that it was the victim of group think. Almost all of the pre-requisites and risks were present and it would appear none of the defences.

Outcome statistics: a health warning

As an intensivist I have lived and breathed outcome statistics. They are incredibly useful tools, but they take time to become useful, it took the intensive care community a good decade to start to understand the meaning and utility of the statistics produced by ICNARC.

What is absolutely certain is that they can never tell the whole story, and in fact when constructing a story about the quality and safety of a service they should simply act as pointers for further and deeper investigation. There are many nuances even to the apparently binomial outcome of mortality that have to be unpicked and understood before coming to any meaningful conclusion.

I thoroughly welcome the fact that mortality as an outcome has found the spotlight. It used to frustrate me immensely as a clinical director that I was held to account more for my financial balance than the number of deaths on my unit. But in finding the spotlight it has been picked up, sensationalised and put to political use; not just by the press but by people within the health service that should know better.

The conversation about mortality in hospitals needs to be held in an intelligent, unfrenzied, non-political and unprejudiced environment. We risk doing immense harm to fragile services if we don’t. When thinking about mortality it is vital that we think slow.

Dr Roger Stedman is medical directors at Sandwell and West Birmingham Trust