You know you've taken methods for too long when...
...you're watching the HoC and hear this:
Mr David Nuttall (Bury North) (Con): May we please have a debate on the Ministry of Justice’s 2011 compendium of reoffending statistics and analysis, so that the fact that prison works can be highlighted? The report contains proof that those who serve longer sentences are less likely to reoffend than those who serve shorter ones. (Emphasis mine)
And your first reaction is this:
The Ministry of Justice’s 2011 compendium of reoffending statistics and analysis is fifty pages of number-porn for stats nerds. That's just great. Stats are good. Interestingly, the Guardian used exactly the same report to argue that prison doesn't work. (Ok, well they added some stuff about longer sentences being found to be better than shorter sentences at the end of the article, but I doubt most people will have read that far).
So what did the report actually find? Regarding sentence length and 'if prison works', they looked at offenders who were given community orders, who were sentenced to less than twelve months in prison, who were sentenced to between one and two years and at offenders who were sentenced to two to four years. And they used matched pairs, which is marvellous. Basically, those given community orders re-offended less than those given less than twelve months (which pleases Mr Kenneth Clarke). But, after comparing a < 12 months group with a 1-2 years group, and comparing a1-2 years group with a 2-4 years group, they found that longer prison sentences lead to less re-offending that shorter prison sentences. (Re-offending here, btw, could just mean 'they get better at not being caught').
Eyes glazed over yet? Excellent.
This is essentially cherry-picking at it's finest. Mr Nuttall likes people to go to prison, so he picks out the data in favour of prisons. The Guardian prefer a wishy-washy namby-pamby liberal approach, so they pick out the data in favour of community orders.
This isn't what made me facepalm, however. It was the idea of proof. Ironically, all the long lists of numbers in the long, long, shiny report go almost nowhere near proving anything.
Here's the deal: if you want to compare two groups then the groups have to be basically equal to begin with. If you're going to compare which exercise program makes people better runners, you don't put professional athletes through one program and professional nerds though the other (we need to get out more. And by that I mean I need to get out more). It doesn't take a genius to work out that, despite the Ministry of Justice's carefully matched pairs, there might be some sort of difference between offenders given difference sentence lengths.
So how do we get to know what works best? We randomise offenders to different sentences. As Ben Goldacre puts much better than I can:
Randomised trials are our best way to find out if something works: by randomly assigning participants to one intervention or another, and measuring the outcome we're interested in, we exclude all alternative explanations for any difference between the two groups. If you don't know which of two reasonable interventions is best, and you want to find out, a trial will tell you... Do long prison sentences work? At the moment, sentences are hugely variable anyway: randomise properly and run a trial.
In other words: get a thousand odd people who have all committed the same crime. Punch their names in a computer, then get that computer to randomly divide them into a short-sentence and a long-sentence group. Follow the prisoners up for ten years, measuring re-offending rates. Then we'll actually know what works best on re-offending rates. Or what teaches criminals not to get caught the best. One way or the other.
It'll never happen, but I can dream.
Mr David Nuttall (Bury North) (Con): May we please have a debate on the Ministry of Justice’s 2011 compendium of reoffending statistics and analysis, so that the fact that prison works can be highlighted? The report contains proof that those who serve longer sentences are less likely to reoffend than those who serve shorter ones. (Emphasis mine)
And your first reaction is this:
The Ministry of Justice’s 2011 compendium of reoffending statistics and analysis is fifty pages of number-porn for stats nerds. That's just great. Stats are good. Interestingly, the Guardian used exactly the same report to argue that prison doesn't work. (Ok, well they added some stuff about longer sentences being found to be better than shorter sentences at the end of the article, but I doubt most people will have read that far).
So what did the report actually find? Regarding sentence length and 'if prison works', they looked at offenders who were given community orders, who were sentenced to less than twelve months in prison, who were sentenced to between one and two years and at offenders who were sentenced to two to four years. And they used matched pairs, which is marvellous. Basically, those given community orders re-offended less than those given less than twelve months (which pleases Mr Kenneth Clarke). But, after comparing a < 12 months group with a 1-2 years group, and comparing a1-2 years group with a 2-4 years group, they found that longer prison sentences lead to less re-offending that shorter prison sentences. (Re-offending here, btw, could just mean 'they get better at not being caught').
Eyes glazed over yet? Excellent.
This is essentially cherry-picking at it's finest. Mr Nuttall likes people to go to prison, so he picks out the data in favour of prisons. The Guardian prefer a wishy-washy namby-pamby liberal approach, so they pick out the data in favour of community orders.
This isn't what made me facepalm, however. It was the idea of proof. Ironically, all the long lists of numbers in the long, long, shiny report go almost nowhere near proving anything.
Here's the deal: if you want to compare two groups then the groups have to be basically equal to begin with. If you're going to compare which exercise program makes people better runners, you don't put professional athletes through one program and professional nerds though the other (we need to get out more. And by that I mean I need to get out more). It doesn't take a genius to work out that, despite the Ministry of Justice's carefully matched pairs, there might be some sort of difference between offenders given difference sentence lengths.
So how do we get to know what works best? We randomise offenders to different sentences. As Ben Goldacre puts much better than I can:
Randomised trials are our best way to find out if something works: by randomly assigning participants to one intervention or another, and measuring the outcome we're interested in, we exclude all alternative explanations for any difference between the two groups. If you don't know which of two reasonable interventions is best, and you want to find out, a trial will tell you... Do long prison sentences work? At the moment, sentences are hugely variable anyway: randomise properly and run a trial.
In other words: get a thousand odd people who have all committed the same crime. Punch their names in a computer, then get that computer to randomly divide them into a short-sentence and a long-sentence group. Follow the prisoners up for ten years, measuring re-offending rates. Then we'll actually know what works best on re-offending rates. Or what teaches criminals not to get caught the best. One way or the other.
It'll never happen, but I can dream.