The last few weeks has seen MTAS (medical training application service) start off with more than a few glitches. In fact the system has been called a ‘shambles’ on the broadsheet front pages (1). The potential human cost of such system failure appears to be massive, so it begs the question: is there any evidence that backs up the MTAS process?
This study (2) analysed a new system used to select GP registrars, which incorporated application forms using competency based statements, structured interviews and a lengthy selection day at an assessment centre. Saliently the use of the application forms is all but ignored in the results and discussion. The focus is upon the finding that those candidates who did well at the very long assessment centre selection, which involved several 20-40 minute stations, did better in their jobs three months down the line. This is not a particularly dramatic result; as most people would assume that the more rigorous the assessment centre, the better it would be at weeding out the weaker candidates. There is no mention of a correlation between the application form scores and job performance, this is surprisingly omitted.
“Furthermore, as the comparison and matched group were demographically not equivalent to the initial intervention sample (354 applicants), caution is needed in generalisability of the results”
Quoting directly from the study (2), the authors are admitting that great caution should be taken in generalising from these results, possibly advice that MTAS should not have ignored. There is another similar study (3) that has been published since this, which found that a lengthy rigorous assessment centre process was useful in selecting paediatric doctors for higher training. From what I have gleaned, the evidence shows that lengthy and rigorous assessment centre selections are good at selecting candidates; hardly rocket science is it?
MTAS has used an application form which gives a rather large amount of weight to competency based questions, and a rather minimal amount of weight to concrete provable achievements like royal college exams, prizes, publications and courses. Is there any evidence base on which the MTAS form has been sculpted? This study (4) is about the only evidence around concerning the use of competency based statements. It demonstrated that applicants who added competency based statements to their résumés got higher scores from Human Resources departments. The relevance of this to MTAS seems pretty contentious, to say the least.MTAS does not seem to have much evidence behind its application forms. The recurring theme from the research seems to be that thorough selection days at assessment centres are very useful in getting the best candidates for the jobs. The process of allocating such a high percentage of short listing marks for competency based questions does not seem to have ever been validated or proven, and this makes the current bunch of juniors look increasingly like experimental subjects.
What is wrong with the Curriculum Vitae and good old references? After all, these formats have stood the test of time and are still used as a selection tool to this day by numerous big companies and businesses around the globe. A good reference is the highest commendation one can be given, the referee is putting his reputation on the line for you, and this cannot be summed up into numbers and highly structured scores. It is also insane to pretend that it makes no difference which university one attended or which jobs one has had, this is political correctness gone raving mad.If one assumes that the MTAS process is incorruptible, then we are still left with a new selection tool that has no evidence behind its use. However if one entertains the possibility of a corrupt MTAS process, then we have an even bigger disaster on our hands. There have been rumours that some candidates have had access to confidential information, thus giving them an unfair advantage in this selection process. So either way, the CV is looking like a utopian tool in comparison with MTAS.
The potential human cost of a deeply flawed selection process is massive. Whatever the tool used, the unemployment of junior doctors was going to be a big problem; but with a flawed selection tool, some of the very best junior doctors may be amongst those signing on. How on earth did the MTAS application form come into existence in its current form? As it appears that scientific evidence had very little do with it..If everything that is subjective continues to be brutally destroyed in this manner, it will soon become apparent that not much remains. If things cannot be translated into numbers, it does not mean they are meaningless, far from it. We are humans after all, and by trying to rigidly standardize everything under the sun, we are forgetting this important fact. The subjective human touch is often impossible quantify, but this doesn’t make it worthless; it makes it priceless and we might not be in the pickle we are in today if certain people had remembered this.
1. The Daily Telegraph front page 01/03/2007 http://www.telegraph.co.uk/news/main.jhtml?xml=/news/2007/03/02/nhs02.xml2. A new selection system to recruit general practice registrars: preliminary findings from a validation study. Patterson et al. BMJ 2005;330:711-714.3. Randall, R, Davies, H, Patterson, F, Farrell, K (2006). Selecting doctors for postgraduate training in paediatrics using a competency based assessment centre.. Arch. Dis. Child. 91: 444-448.4. Jim Bright and Sonia Hutton. The impact of competency statements on resumees for short-listing decisions. International journal of selection and assessment. Volume 8 Issue 2 Page 41 - June 2000.
No comments:
Post a Comment