HSMRs, coding, big assumptions and implausible conclusions....

The second major flaw in Prof Brian Jarman's UK/US mortality comparison is that it assumes HSMRs are reliable and coding is equivalent in the UK and US.  If either of these assumptions is false then the comparison is definitely on extremely dubious foundations.   So firstly to HSMRs, one can certainly do far worse than to read this summary by Prof Speigelhalter:

"The two indices often come up with different conclusions and do not necessarily correlate with Keogh’s findings: for example, of the first three trusts investigated, Basildon and Thurrock was a high outlier on SHMI but not on HSMR for 2011-12(and was put on special measures), Blackpool was a high outlier on both (no action), and Burton was high on HSMR but not on SHMI (special measures). Keogh emphasised “the complexity of using and interpreting aggregate measures of mortality, including HSMR and SHMI. The fact that the use of these two different measures of mortality to determine which trusts to review generated two completely different lists of outlier trusts illustrates this point." It also suggests that many trusts that were not high on either measure might have had issues revealed had they been examined."

HSMRs and SHMIs are not useless but they are far from perfect, even when monitoring the trend of one individual hospital's mortality over time.  They are more problematic when comparing hospitals, as variations in coding can have huge effects on the results.  This BMJ article highlights many more concerns over the use and validity of HSMRs, while there are many excellent rapid responses which also highlight many more associated problems.  Here are some other studies outlining problems with the reliability and/or validity of HSMRs ( paper 1, paper 2, paper 3).  This particular segment highlights a huge flaw in HSMRs and in Jarman's UK/US comparison:

"The famous Harvard malpractice study found that 0.25% of admissions resulted in avoidable  death. Assuming an overall hospital death rate of about 5% this implies that around one in 20 inpatient deaths are preventable, while 19 of 20 are unavoidable. We have corroborated this figure in a study of the quality of care in 18 English hospitals (submitted for publication). Quality of care accounts for only a small proportion of the observed variance in mortality between hospitals. To put this another way, it is not sensible to look for differences in preventable deaths by comparing all deaths."

This is the crux of it, meaning that a 45% difference in acute mortality as a result of poorer care is utterly implausible.  Now to coding, there is a long history of up-coding in the US and inadequate incomplete coding in the UK.   Here is one of the best papers proving that HSMRs is hugely affected by admission practices and coding, something that Prof Jarman seems to be unwilling to consider having any effect on the US/UK HSMR difference:

"Claims that variations in hospital standardised mortality ratios from Dr Foster Unit reflect differences in quality of care are less than credible."

I think this applies to Prof Jarman's latest conclusions on US/UK HSMRs, in my opinion his conclusions are less than credible also.  There is also an excellent NEJM piece on the problems with standardised mortality ratios, rather a reliable and eminent journal.  There is one recurrent theme in the academic literature and it appears to me that HES data and standardised mortality ratios are not reliable.  Another recurring theme is that the one person defending them often seems to be a certain Professor Brain Jarman, read into that what you will.  

There are so many problems with Professor Jarman's work and conclusions that it is hard to sum it up in one piece, really one needs a whole book.  Firstly the underlying HES data is unreliable, secondly HSMRs are not reliable and are highly subject to different coding practices/admission practices, thirdly the US and UK are highly likely to be at opposite ends of the spectrum in terms of coding (up versus down) and are also likely to have extremely different admission/discharge practices, fourthly the differences in UK/US HSMRs being down to care is utterly implausible, and fifthly the UK's massive mortality improvements over the last decade are also utterly implausible.  It appears Professor Jarman has unsuspectingly scored an own goal, the HSMR has revealed itself with such implausible results.