Friday 13 June 2014

John Cook's responds but reliability issue is clearly a key study flaw

Here is John Cook's response to my questions:

"Re rater reliability and data breakdowns, we’ve released the raw individual ratings as well as the final ratings of all 11,944 papers at http://www.skepticalscience.com/tcp.php

Re how the endorsement levels were created, this was the result of a long, collaborative discussion between the authors - attempting to resolve the issue that different authors expressed endorsement of the scientific consensus in different ways. By allowing for different expressions of endorsement, it allowed us to have our cake and eat it too.

Re the proportion of studies providing primary evidence, we didn’t tag such papers - but it is an interesting idea worth exploring."
 
The reliability issue appears a big thorn in Cook's side.  In my opinion intra-rater and inter-rater reliability should have been published as part of the original study, the peer review should have picked this up and the fact that it didn't is worrying.
 
Cook's suggestion that the data is there for analysis is potentially misleading in my opinion.  Analysis is not possible without with days and days of re-working the data into a manageable format from which the reliability analysis could be performed.  Also I don't think the data is even there for calculating intra-rater reliability as it appears that raters did not re-rate the studies.
 
In my opinion the data should be provided in a format in which rater reliability can be easily calculated, this is clearly not the case as things stand.  Given that this is something fundamental to the original study and that should have been published in the first place, I find Cook's stance on the data rather unhelpful and the cynic could interpret this as Cook trying to hide the problem that is a far from reliable rating system.
 
At least Cook does acknowledge my point that it would be well worth exploring which studies actually provided strong primary evidence to back up their subjective opinion.  A study that reviewed the primary evidence for man-induced global warming would be far far more valuable than a study which simply surveys subjective opinion that may be based upon no meaningful evidence at all.

Thursday 12 June 2014

Why won't John Cook reply to my simple questions?


Strangely or not so strangely, having been 'redirected' to send my letter's questions directly to the author, John Cook, I have received a response, but no attempt to respond to my queries about his study's methodology.  John Cook simply asked if I was one of many people 'that referred to Skeptical Science as "That Propaganda Site”?'.

I fail to see what my opinion on John Cook's website has to do with him answering some very simple questions about his study's methodology.  Perhaps Dr Cook should just answer the questions or is there a reason why he cannot?  Here are the questions that John Cook chooses to ignore for whatever reason:

"I read the study by Cook et al with great interest (1).  Firstly the study used levels of endorsement of global warming as outlined in their Table 2, however I can see no mention as to how these levels were created and how reliable they were in terms of both inter-rater and intra-rater reliability (Cohen’s kappa); would it be possible for the authors to clarify?  Secondly the authors ‘simplified the analysis’ by breaking down ratings into three groups, however they have not included the data breaking down the results into the original 7 categories: would it be possible to see this data?  Finally the study showed that 62.7% of all papers endorsed the consensus, but it does not mention how what proportion of these studies actually provided primary evidence to support the consensus: did the authors gather this information? "


Friday 6 June 2014

Strange response from Environmental Research Letters on Cook et al's 97% paper


 
"I read the study by Cook et al with great interest (1).  Firstly the study used levels of endorsement of global warming as outlined in their Table 2, however I can see no mention as to how these levels were created and how reliable they were in terms of both inter-rater and intra-rater reliability (Cohen’s kappa); would it be possible for the authors to clarify?  Secondly the authors ‘simplified the analysis’ by breaking down ratings into three groups, however they have not included the data breaking down the results into the original 7 categories: would it be possible to see this data?  Finally the study showed that 62.7% of all papers endorsed the consensus, but it does not mention how what proportion of these studies actually provided primary evidence to support the consensus: did the authors gather this information? 

1.                  Cook J et al. Quantifying the consensus on anthropogenic global warming in the scientific literature. Environ. Res. Lett. 8 (2013) 024024. "
 
I wrote the above letter to ERL and received the following response:
 
"We regret to inform you that the Board Member has recommended that your article should not be published in the journal, for the reasons given in the enclosed report. Your manuscript has therefore been withdrawn from consideration."
 
The Board member states:
 
"The "methodological queries" is not a manuscript suited for publication at all, it simply is a set of questions to the authors of Cook et al. I would advise the author to pose these questions directly to John Cook, as is the normal procedure if someone has further questions about a publication - the corresponding author's contact address is provided with each paper on ERL."
 
I have sent the following response back to the Journal outlining why I feel the response from their Board member is grossly inadequate:
 
"I appreciate your response however the referee's comments are grossly inadequate in my opinion and I would like to request that my letter be reviewed by another reviewer who is independent to the Journal.
 
My questions relate to significant methodological flaws in the study by Cook and it is perfectly acceptable to submit this as a letter, so that such methodological flaws are discussed openly in a public forum.
 
I have written to several Journals and had several similar letters published in the past, I have never heard of this excuse for rejecting a submission.
 
Of note the reviewer did not even comment on the validity of my methodological questions, something I find rather strange.
 
I would appreciate a swift response to this letter"
 
This whole things appears rather fishy.  I have also emailed John Cook to see if he can answer my questions, for some reason I suspect I shall receive nothing back from him.  It is 'normal procedure' for many many scientific Journals to publish letters such as mine, which outline methodological concerns, so that this can be discussed openly and any problems with the study are noted in the public domain.  For some reason this ERL Board member doesn't want concerns about Cook's paper to be aired in public, can't imagine why?

Thursday 5 June 2014

Telegraph gives away its nasty agenda

 
The above headline is a total disgrace.  The contaminated drips were supplied by a private firm, ITH Pharma, and the NHS is totally blame free in the poisoning of 15 babies.  However the Telegraph chose to cynically mislead readers and smear the NHS.  Of note this is not an isolated example, the Telegraph has been denigrating and smearing the NHS for a long time now, strangely the private sector never gets the same treatment, even when it kills babies.
 
I would urge everyone to complain to the PCC by using this link:
 
 
Feel free to use the text below in your complaint:
 
"i) The Press must take care not to publish inaccurate, misleading or distorted information, including pictures."
 
The headline is inaccurate and misleading.  The drips were used in the NHS but they were not manufactured by the NHS.  They were manufactured by a private firm, ITH Pharma.

The lazy and inaccurate headlines misleads readers and it directly implies that the NHS is at fault for the harm to the babies, when it is not in anyway to blame.

"ii) A significant inaccuracy, misleading statement or distortion once recognised must be corrected, promptly and with due prominence, and - where appropriate - an apology published. In cases involving the Commission, prominence should be agreed with the PCC in advance."
The Telegraph should issue a front page correction and apology as a lead article.

Of note the Telegraph clearly knows the headline was a mistake as it has already change the headline on its website to "One baby dead and 14 with blood poisoning from contaminated drip".  Alas the damage has been done with this inaccurate and misleading headline in the print edition.