Science and statistics

When dealing with research that involves statistical analysis, there are many ways people can get it wrong (unintentionally and intentionally). In this article on abcnews.go.com's Technology section, John Allen Paulos has written about the issue of combining data sets that have been claimed to measure something but for which the thing supposedly being measured is defined differently in the production of each set of data. This means that despite the same word being used for what was intended to be measured in both cases, the different sets of data did not measure the same thing (regardless of what words are used to describe both sets of measurements).

As Frederick Williams and Peter Monge have written in Reasoning With Statistics: How to Read Quantitative Research:

Just because a study has used statistics is no guarantee of its worth. In fact statistics can be misused either intentionally or unwittingly, and it is not difficult to locate quantitative studies where ststistics were not really needed at all. On the other hand, statistics can be a powerful tool for description or hypothesis testing. Probably the most valuable general skill individuals can have in statistical methods is the ability to understand the foregoing when reading research articles, books, or reports in their respective specialties. Of course a valuable specific skill is the ability to use statistical methods in your research if you need them.

When dealing with claims based on statistics, if the details are not provided to determine how the data is claimed to have been collected (ie who is being claimed to have observed what and what methods they used to record quantified measurements of selected aspects of these observations) then the claims are no better than opinions unsupported by statistics. If the claims of who observed and recorded what are clear, judgments can be made about the likely reliability or otherwise of the data and the claims based on them, and data (but more importantly, observations underlying the data) can be tested by replicating experiments.