The Process of Peer Review
Followup to my post about global warming skepticism specifically and scientific skepticism in general. If someone discovers anomalous data supposedly correcting, modifying, or outright disproving a theory it is important to ask the following questions:
No single experiment proves that a theory is true, no single anomalous data point proves that a theory is false, and all data is verified independently. Any shortcut, for example issuing a public press release without peer review, is asking for trouble. Just ask Pons and Fleischman or Hwang Woo-Suk. Deviation from the process is not prima facie evidence that the new data is fraudulent or that the new theory is incorrect, but keep the corks on your champagne anyway. Scientists can be wrong for all kinds of reasons from excitement and misinterpretation to outright fraud, and the peer review process is a safeguard against this known failure mode.
"A lie can make it half way around the world before the truth has time to put its boots on." - frequently misattributed to Mark Twain, but a good one nonetheless.
- Have the data and claims been accepted for publication in a reputable, professional, peer-reviewed journal?
- Have the data and claims been replicated and verified by independent observers or labs?
- Does the anomalous data suggest a revised, predictive, falsifiable, testable theory?
- Have the replications used improved methodologies to better isolate the effect based on the predictions of the new theory? Does the effect size increase under examination by these methods?
- What is the revised concensus based on the replicated, peer-reviewed, published data?
No single experiment proves that a theory is true, no single anomalous data point proves that a theory is false, and all data is verified independently. Any shortcut, for example issuing a public press release without peer review, is asking for trouble. Just ask Pons and Fleischman or Hwang Woo-Suk. Deviation from the process is not prima facie evidence that the new data is fraudulent or that the new theory is incorrect, but keep the corks on your champagne anyway. Scientists can be wrong for all kinds of reasons from excitement and misinterpretation to outright fraud, and the peer review process is a safeguard against this known failure mode.
"A lie can make it half way around the world before the truth has time to put its boots on." - frequently misattributed to Mark Twain, but a good one nonetheless.