John Cook's responds but reliability issue is clearly a key study flaw

Here is John Cook's response to my questions:

"Re rater reliability and data breakdowns, we’ve released the raw individual ratings as well as the final ratings of all 11,944 papers at http://www.skepticalscience.com/tcp.php

Re how the endorsement levels were created, this was the result of a long, collaborative discussion between the authors - attempting to resolve the issue that different authors expressed endorsement of the scientific consensus in different ways. By allowing for different expressions of endorsement, it allowed us to have our cake and eat it too.

Re the proportion of studies providing primary evidence, we didn’t tag such papers - but it is an interesting idea worth exploring."

 

The reliability issue appears a big thorn in Cook's side.  In my opinion intra-rater and inter-rater reliability should have been published as part of the original study, the peer review should have picked this up and the fact that it didn't is worrying.

 

Cook's suggestion that the data is there for analysis is potentially misleading in my opinion.  Analysis is not possible without with days and days of re-working the data into a manageable format from which the reliability analysis could be performed.  Also I don't think the data is even there for calculating intra-rater reliability as it appears that raters did not re-rate the studies.

 

In my opinion the data should be provided in a format in which rater reliability can be easily calculated, this is clearly not the case as things stand.  Given that this is something fundamental to the original study and that should have been published in the first place, I find Cook's stance on the data rather unhelpful and the cynic could interpret this as Cook trying to hide the problem that is a far from reliable rating system.

 

At least Cook does acknowledge my point that it would be well worth exploring which studies actually provided strong primary evidence to back up their subjective opinion.  A study that reviewed the primary evidence for man-induced global warming would be far far more valuable than a study which simply surveys subjective opinion that may be based upon no meaningful evidence at all.