Health Education England, Fiona Patterson and Situational Judgement Tests (SJTs)

 

I got a bit distracted this morning when I started reading HEE's latest idea on medical selection called 'valued based recruitment' (VBR).  Here is HEE's long and shiny briefing on VBR.  I was particularly interested to read their evidence summary section in which it is stated:

 

"Situational Judgement Tests: high reliability and high validity"

 

(high validity in context of 'only if based on robust psychometric methodology')

This led me to question just where this evidence has come from.  HEE kindly replied to my query very quickly and gave me the source.  It was also interesting to note that all methods of assessing values were pretty highly susceptible to coaching, I would interpret that to mean that these methods aren't really very good at assessing people's real values as these should not be very modifiable by coaching!

Here is the full evidence section, it is important to note that Fiona Patterson and her Work Psychology Partnership Group (infamous due to their role in the disastrous MTAS selection process) are thanked by HEE for their 'support' in reviewing the evidence.  Of note Fiona Patterson and her group are keen on SJTs, have developed them for use in the UK and have made a lot of money out of SJTs.  No conflicts of interest there then.

Anyway I went on to read the evidence to see where this evidence of high reliability and validity of SJTs has come from.  Well, surprise, surprise, the evidence on SJTs is Fiona Patterson's very own review!  I then read this review to see if the evidence would back up HEE's summary, alas it did not.

Patterson's review shows a very different picture on SJTs to that painted by the HEE summary.  The reliability of SJTs is mixed, certainly some studies have shown good reliability, however many have shown extremely poor reliability (Cronbach's alphas of around 0.2).  The same is true for SJT validity, it is mixed at best, and that is arguably too optimistic, numerous studies have shown extremely poor reliability (spearman's r's of around 0.2/0.3). 

Overall the evidence in Patterson's review does not support the use of SJTs in medical selection, in my biased opinion.  It is worth noting that Patterson appears to be promoting a selection tool from which she stands to make significant amounts of money from. There is one aspect of the review I strongly agree with and that is the solid evidence demonstrating that selection centres are good selection tools:

"When designed appropriately, selection centres are valid predictors of job performance and could be used for VBR. Construct validity is enhanced by ensuring the content is directly relevant to the attributes of interest in the target job role. However, selection centres are relatively expensive to design and deliver. "

The bottom line for me is that SJTs should not be used, they are unreliable and not sufficiently valid.  However for various reasons including a distorted and cherry picking view of the evidence, HEE had come out advising that SJTs be used as part of VBR.  HEE have got it wrong, SJTs should not be used, I am sure Fiona Patterson and her group will do very nicely if SJTs are used though, what a coincidence......