The Overlooked Peril of Big Data
We're growing familiar with the problem of how Big Data (the compilation of enormous amounts of statistical information) can erode privacy, often by correlating information in ways that enable to identify individuals in unexpected ways. I've often compared it to Lovecraft's famous quote about correlating the contents of our mind -- sometimes it's better if the unknown knowns remain unknown.
But there's another problem -- our conclusions are only as good as the data on which they are based, but we tend to overestimate the reliability of our data. A case in point: Robert McNamara, JFK's Secretary of Defense, was a numbers man, an expert in the analysis of enormous amounts of statistical data. However, he did not realize that a lot of the data on which he was basing his conclusions about the situation in Vietnam, and thus his directives for the military, was faulty. Some of it was deliberately so, as a result of units trying to make themselves look better, and some of it was the results of unconscious biases or even just the limitations of the intelligence-gathering technology of the time. But it led to some very bad policy decisions that cost thousands of lives and may well have contributed to the anti-war movement and the ultimate withdrawal of US forces from Vietnam, with disastrous results.
But there's another problem -- our conclusions are only as good as the data on which they are based, but we tend to overestimate the reliability of our data. A case in point: Robert McNamara, JFK's Secretary of Defense, was a numbers man, an expert in the analysis of enormous amounts of statistical data. However, he did not realize that a lot of the data on which he was basing his conclusions about the situation in Vietnam, and thus his directives for the military, was faulty. Some of it was deliberately so, as a result of units trying to make themselves look better, and some of it was the results of unconscious biases or even just the limitations of the intelligence-gathering technology of the time. But it led to some very bad policy decisions that cost thousands of lives and may well have contributed to the anti-war movement and the ultimate withdrawal of US forces from Vietnam, with disastrous results.