Professor Jarman's reliance on the c-statistic

There is so so much more to this UK/US HSMR comparison that first meets the eye, at first glance there were some significant flaws and assumptions made, but the more digging I have done, the more gaps I seem to be finding in certain peoples' robust conclusions.  Many thanks to Prof Jarman for the links and data he has provided so much, it is well worth while reading up on the methodology behind HSMRs in the first instance.  One thing it is vital to understand is the so-called 'c-statistic' which is a basic measure as to how well a model fits the mortality data for a particular group of patients:

"The success of case-mix adjustment for accurately predicting the outcome (discrimination) was evaluated using the area under the receiver operating characteristic curve (c statistic). The c statistic is the probability of assigning a greater risk of death to a randomly selected patient who died compared with a randomly selected patient who survived. A value of 0.5 suggests that the model is no better than random chance in predicting death. A value of 1.0 suggests perfect discrimination. In general, values less than 0.7 are considered to show poor discrimination, values of 0.7-0.8 can be described as reasonable and values above 0.8 suggest good discrimination."

and

"As a rank-order statistic, it is insensitive to systematic errors in calibration"

The second quote is particularly salient, as one of the key flaws in Professor Jarman's US/UK comparison may be that there were systematic differences between the admissions policy and coding in the different countries, this would not be detected by the c-statistic.

It is then interesting to look at Dr Foster's data concerning the c-statistics they have obtained for the clinical conditions that they use to determine the HSMRs of UK hospitals.  Dr Foster routinely uses 56 diagnostic groups (contributing towards about 83% of UK deaths) and they are defined according to the ICD codes.  Of note much of Prof Jarman's UK/US comparison only used 9 diagnostic codes,  which is a little strange in itself, why not use all 56 diagnostic codes?  These 9 codes covered less than half of the deaths in both countries.  I have listed the codes used the their individual c-statistics based on Dr Foster's 2012 report:

Septicaemia - 0.792 (reasonable)

Acute MI - 0.759 (reasonable)

Acute heart failure - 0.679 (poor)

Acute cerebrovascular event - 0.729 (reasonable)

Pneumonia - 0.838 (good)

COPD/Bronchiectasis - 0.714 (low end of reasonable)

Aspiration pneumonitis -0.711 (low end of reasonable)

Fractured Hip - 0.756 (reasonable)

Respiratory failure - 0.745 (reasonable)

These c-statistics are not very impressive, in must be remembered that 0.5 is effectively the zero, and many of these c-statistics are around the low end of reasonable.  It is interesting that Professor Jarman quotes the overall c-statistic for his 9 code model as being 0.921.  Given that he individually compared each HSMR for each code, surely he should be giving the individual c-statistics for each country's subgroup for each specific code?  Professor Jarman has not provided this data, it would certainly be interested to see the c-statistics for the UK and the US for each code, to see if there was a relationship between c-statistic disparity and HSMR disparity.

It is also interesting that 75 of 247 of Prof Jarman's mortality models failed their statistical Goodness of Fit tests.  The measure of how well the models generally fit is also pretty poor (mean Rsquared of 0.25).  It must also be reiterated that the c-statistic will not pick up errors in calibration, so if one country is systematically up-coded relative to another, then the c-statistic will not detect this. The one key question I would like to see answered, is just how did Professor Jarman select these 9 codes for the US/UK comparison?  There are also other questions, like which models failed the Goodness of Fit tests and did the codes assessed have reasonable r-squared values?  There is so much beneath the surface here, I am convinced this story will run and run.

HSMRs, coding, big assumptions and implausible conclusions....

The second major flaw in Prof Brian Jarman's UK/US mortality comparison is that it assumes HSMRs are reliable and coding is equivalent in the UK and US.  If either of these assumptions is false then the comparison is definitely on extremely dubious foundations.   So firstly to HSMRs, one can certainly do far worse than to read this summary by Prof Speigelhalter:

"The two indices often come up with different conclusions and do not necessarily correlate with Keogh’s findings: for example, of the first three trusts investigated, Basildon and Thurrock was a high outlier on SHMI but not on HSMR for 2011-12(and was put on special measures), Blackpool was a high outlier on both (no action), and Burton was high on HSMR but not on SHMI (special measures). Keogh emphasised “the complexity of using and interpreting aggregate measures of mortality, including HSMR and SHMI. The fact that the use of these two different measures of mortality to determine which trusts to review generated two completely different lists of outlier trusts illustrates this point." It also suggests that many trusts that were not high on either measure might have had issues revealed had they been examined."

HSMRs and SHMIs are not useless but they are far from perfect, even when monitoring the trend of one individual hospital's mortality over time.  They are more problematic when comparing hospitals, as variations in coding can have huge effects on the results.  This BMJ article highlights many more concerns over the use and validity of HSMRs, while there are many excellent rapid responses which also highlight many more associated problems.  Here are some other studies outlining problems with the reliability and/or validity of HSMRs ( paper 1, paper 2, paper 3).  This particular segment highlights a huge flaw in HSMRs and in Jarman's UK/US comparison:

"The famous Harvard malpractice study found that 0.25% of admissions resulted in avoidable  death. Assuming an overall hospital death rate of about 5% this implies that around one in 20 inpatient deaths are preventable, while 19 of 20 are unavoidable. We have corroborated this figure in a study of the quality of care in 18 English hospitals (submitted for publication). Quality of care accounts for only a small proportion of the observed variance in mortality between hospitals. To put this another way, it is not sensible to look for differences in preventable deaths by comparing all deaths."

This is the crux of it, meaning that a 45% difference in acute mortality as a result of poorer care is utterly implausible.  Now to coding, there is a long history of up-coding in the US and inadequate incomplete coding in the UK.   Here is one of the best papers proving that HSMRs is hugely affected by admission practices and coding, something that Prof Jarman seems to be unwilling to consider having any effect on the US/UK HSMR difference:

"Claims that variations in hospital standardised mortality ratios from Dr Foster Unit reflect differences in quality of care are less than credible."

I think this applies to Prof Jarman's latest conclusions on US/UK HSMRs, in my opinion his conclusions are less than credible also.  There is also an excellent NEJM piece on the problems with standardised mortality ratios, rather a reliable and eminent journal.  There is one recurrent theme in the academic literature and it appears to me that HES data and standardised mortality ratios are not reliable.  Another recurring theme is that the one person defending them often seems to be a certain Professor Brain Jarman, read into that what you will.  

There are so many problems with Professor Jarman's work and conclusions that it is hard to sum it up in one piece, really one needs a whole book.  Firstly the underlying HES data is unreliable, secondly HSMRs are not reliable and are highly subject to different coding practices/admission practices, thirdly the US and UK are highly likely to be at opposite ends of the spectrum in terms of coding (up versus down) and are also likely to have extremely different admission/discharge practices, fourthly the differences in UK/US HSMRs being down to care is utterly implausible, and fifthly the UK's massive mortality improvements over the last decade are also utterly implausible.  It appears Professor Jarman has unsuspectingly scored an own goal, the HSMR has revealed itself with such implausible results.

Many big assumptions have been made: 1. HES Data

The US/UK HSMR comparison made by Prof Brian Jarman is continuing to rumble on and probably not for reasons that will please the Professor.  Since the leaking of the data to the media a few days back, several astute observations have been made that cast great doubt upon Prof Jarman's conclusions  This is my take on events and my summary of the problems with Prof Jarman's stance, this is part 1 on HES data:

HES data is of low quality and is unreliable (citation 1,citation 2, citation 3, citation 4, citation 5)

"Concerns remain about the quality of HES data. The overall percentage of admissions with missing or invalid data on age, sex, admission method, or dates of admission or discharge was 2.4% in 2003. For the remaining admissions, 47.9% in 1996 and 41.6% in 2003 had no secondary diagnosis recorded (41.9% and 37.1%, respectively, if day cases are excluded). In contrast to some of the clinical databases, if no information on comorbidity is recorded, we cannot tell whether there is no comorbidity present or if comorbidity has not been recorded. Despite these deficiencies, our predictive models are still good. In the most recent report of the Society of Cardiothoracic Surgeons, 30% of records had missing EuroSCORE variables. Within the Association of Coloproctology database, 39% of patients had missing data for the risk factors included in their final model. A comparison of numbers of vascular procedures recorded within HES and the national vascular database found four times as many cases recorded within HES."

and this from a letter by authors including Bruce Keogh:

"This latest study raises more concerns about hospital episode statistics data, showing that errors are not consistent across the country."

and this from Royal College of Physicians,as recently as 2012:

"This change is necessary because the current process for the collection of data for central returns that feed HES, and the content of the dataset itself, are both no longer appropriate for the widening purposes for which HES are used. "

I can find little to support Brian Jarman's stance claiming that HES data is accurate and reliable.  Prof Jarman's study relies massively on HES data, as it is this very data from which his HSMRs are calculated.  It would be fascinating if Prof Jarman could produce some published evidence to support his stance on HES data.  If the UK data is less complete than the US data, it could well lead to a massive difference in HSMRs that is nothing to do with care standards, but that is purely down to data quality.  The HSMR is another part all in itself.  

Channel 4's 'UK/US hospital mortality' story is based on Jarman's sandy foundations

Last night I sat down to watch the Channel 4 news and was deeply upset by what was then to follow.  The 'news' exclusive on the 'increased mortality' in UK hospitals versus those in the US was presented as if the data to prove this theory was robust, there was no discussion of the huge flaws in the methods used and the panel discussion was completely one sided.  Channel 4 gave no neutral academics a chance to speak and they gave no one the chance to defend the NHS.  It was trial and execution by a biased one man band, it was shoddy journalism at its worst, it was very disappointing and very unlike Channel 4's normally excellent coverage.  I shall be factual and careful with what I shall say next, as it appears some cannot listen to criticism of their pet methodologies without resorting to threats of GMC referral, not the sign of a robust argument I would say.

The story claimed that patients was 45% more likely to die in the UK than the US, when admitted with acute illness such as pneumonia and septicaemia.  This was based on 'research' done by Professor Brian Jarman, he of Dr Foster fame and a big supporter of the HSMR tool (Hospital Standardised Mortality Ratio).  It must be noted that Dr Foster are rather close to the Department of Health, with the latter being obliged to promote the business interests of Dr Foster 'intelligence'.  It is worth reading about the rather cosy relationship involving Jarman, Dr Foster and the government, because it puts Jarman's potential motives into the open, conflicts of interest are often key in understanding such matters.

 Essentially the Channel 4 story was based upon several assumptions that look rather naive, flawed and ignorant to anyone that has a basic grasp of scientific evidence and statistics.  Firstly the UK mortality data is based upon HES (Hospital Episode Statistics) data, which is notoriously inaccurate and unreliable.  For example in the recent past HES data showed there were 17,000 pregnant men in the UK, a truly unbelievable statistic.  There is also abundant evidence showing that HSMRs themselves are very poor tools, even for comparing hospitals within the same country, let alone different continents.  HSMRs are crude tools and many academics feel their use should be abandoned entirely.

"Nonetheless, HSMRs continue to pose a grave public challenge to hospitals, whilst the unsatisfactory nature of the HSMR remains a largely unacknowledged and unchallenged private affair."

The above quote shows exactly what the problem is, here we have a flawed and scientifically dubious measure being used when it should not be used, as a political bandwagon and some vested interests are running out of control.  Brian Jarman's baby is the HSMR and he is a powerful man with a large sphere of influence.  It is notable that even the Keogh review was rather critical of the use of HSMRs and the way in which they had been inappropriately use to create divisive anti-NHS propaganda.  There are so many things that can change the HSMR, and many are absolutely nothing to do with the actual quality of care provided.

In simple terms if you put poor quality in, then you get dubious data out and will reach dodgy flawed conclusions.  Firstly HES data, on which the UK mortality rates are based, is poor and this means that mortality cannot be adequately adjusted for confounding factors.  The data is so inaccurate that information about illness type, severity of illness and co-morbidities cannot be adequately adjusted for in the statistical modelling.  The way data is coded differently in the US and UK is also likely to have a massive effect on the results.  Generally HES data is poor and patients are under-coded, ie their illness and co-morbidities are not coded to be as bad as they actually are.  However the exact opposite is true in US coding, they have a vast army of bureaucrats and have had a marketised system for decades, meaning that over-coding is rife, ie hospitals exaggerate the illness and co-morbidities of patients in order to increase their revenues.  There have also been huge issues with the accuracy of the US data.  There is also a huge volume of evidence showing that over-charging as a result of over-coding has been rife in the US, estimates put the cost of over-coding in the multi billion dollar ball park.

Overall this is a case study in dodgy data and dodgy journalism leading to what appears a hugely erroneous conclusion.  Brian Jarman has been reckless and irresponsible in my opinion, he has tried to justify his leaking of unpublished data but his squirming doesn't cut the mustard for me.  In honesty Brian Jarman simply doesn't know why the HSMRs are so different in the UK and US, interestingly the UK outperformed the US on fractured hip mortality, a fact sadly ignored by the news and not mentioned by Jarman in his interview.  This is also rather salient as hip fractures are one of the few things coded accurately in the UK, as it has been attached to a specific tariff for years, again yet more evidence that coding is responsible for the differences in HSMRs and not care quality.  Overall the likely explanation for the differences is the huge discrepancies in terms of coding practice between the US and the UK, in the US they tend to over-code and in the UK we tend to under-code. The huge effects of coding and admission practices on HSMRs have been well made in the BMJ in the past.  There is also the fact that many US patients are discharged from acute hospitals to die elsewhere, and Brian Jarman has admitted this hasn't been taken into account in his calculations.

Brian Jarman should not be leaping to such conclusions and he should be publishing in a reputable journal before recklessly leaking such propaganda to the media.  I wonder if Brian Jarman has done any work at all to investigate the differences between coding in the US and the UK, or did he simply say what he was nudged towards by the Department of Health?  I also wonder if Brian Jarman can claim his data is adequately corrected for confounders when it is based on unreliable and inaccurate HES data?  Personally speaking I have lost all respect for Brian Jarman and am now highly suspicious of the cosy nepotistic little relationship involving the government, Dr Foster and Brian Jarman.  It is just extremely sad that Victoria McDonald was played and used in such a game.  This whole episode has shown very clearly that one thing the NHS needs is good independent statistics and high quality data, and Dr Foster is definitely not going to be part of a progressive solution.