Monday, 16 September 2013

Many big assumptions have been made: 1. HES Data

The US/UK HSMR comparison made by Prof Brian Jarman is continuing to rumble on and probably not for reasons that will please the Professor.  Since the leaking of the data to the media a few days back, several astute observations have been made that cast great doubt upon Prof Jarman's conclusions  This is my take on events and my summary of the problems with Prof Jarman's stance, this is part 1 on HES data:

HES data is of low quality and is unreliable (citation 1,citation 2, citation 3, citation 4, citation 5)

"Concerns remain about the quality of HES data. The overall percentage of admissions with missing or invalid data on age, sex, admission method, or dates of admission or discharge was 2.4% in 2003. For the remaining admissions, 47.9% in 1996 and 41.6% in 2003 had no secondary diagnosis recorded (41.9% and 37.1%, respectively, if day cases are excluded). In contrast to some of the clinical databases, if no information on comorbidity is recorded, we cannot tell whether there is no comorbidity present or if comorbidity has not been recorded. Despite these deficiencies, our predictive models are still good. In the most recent report of the Society of Cardiothoracic Surgeons, 30% of records had missing EuroSCORE variables. Within the Association of Coloproctology database, 39% of patients had missing data for the risk factors included in their final model. A comparison of numbers of vascular procedures recorded within HES and the national vascular database found four times as many cases recorded within HES."

and this from a letter by authors including Bruce Keogh:

"This latest study raises more concerns about hospital episode statistics data, showing that errors are not consistent across the country."

and this from Royal College of Physicians,as recently as 2012:

"This change is necessary because the current process for the collection of data for central returns that feed HES, and the content of the dataset itself, are both no longer appropriate for the widening purposes for which HES are used. "

I can find little to support Brian Jarman's stance claiming that HES data is accurate and reliable.  Prof Jarman's study relies massively on HES data, as it is this very data from which his HSMRs are calculated.  It would be fascinating if Prof Jarman could produce some published evidence to support his stance on HES data.  If the UK data is less complete than the US data, it could well lead to a massive difference in HSMRs that is nothing to do with care standards, but that is purely down to data quality.  The HSMR is another part all in itself.  

1 comment:

Anonymous said...

I wonder if the Ferret would be as critical of Prof. Jarman's work if he'd said English Hospital mortality compared well with other countries?

Its not hard to find examples of HES accuracy. (See

The citations given are less than compelling evidence that HES is not accurate for the purposes of Jarman's analysis.

Citation 1 indicates that HES CAN be used to model mortality in certain relatively common procedures (CABG, aortic aneurism & colorectal surgery)

Citation 2 misses the point as it concludes that CPRD data on suicides is inaccurate (CPRD is primary care data)

Citations 3-5 deal with use of HES in paediatric heart surgery, a controversial area, but a very small part of NHS hospital activity.

No one who's worked with HES would claim it's perfect. But, in common diagnoses/procedures its often good enough to highlight significant variations in practice & hospital mortality.

If the Ferret doesn't like Prof. Jarman's HSMR what publically available measures & data sources on patient safety in NHS hospitals would he/she prefer?