More thoughts on educational assessment
So I recently got into the Room for Debate blog on nytimes.com. So I thought I'd comment on one that particularly struck me, about testing, particularly standardized testing.
A lot of my thinking on this subject has been influenced by a book I had to read for my Assessment class. I disagree with him on some points, but we'll get to that in a moment. First the part I think he got right. The purpose of assessment, at least summative assessments (at the end of the unit/semester/year...all standardized tests are meant to be summative rather than formative. Formative assessments are expected to influence instruction going forward, such as indicating that additional review is needed before proceeding.) is to operationalize educational objectives. Psychologists want to measure happiness, but there is no ruler for that, so they devise a questionairre or some-such that measures happiness, or that they think is a good stand-in for measuring happiness. Educational objectives are things like "student can do any long division problem involving a three digit dividend and a two digit divisor" (that is more specific than usual, but it serves my purposes). The theoretical best test of that objective is asking the student to divide every number from 100 to 999 by every number from 10 to 99. However, 81000 long division problems would be a little tedious. And long before they did problem number 81000, or problem 8100 or problem 100, I'd know whether or not they could do it. So there will be sampling. I could ask 20 questions, assuming they were well selected, that might indicate the degree to which my educational objective has been achieved. The high-stakes assessments (eg final exams, state mandated exams, etc) are simply testing a larger number of educational objectives from a previously agreed upon list, as Sandra Stotsky notes in the first link.
Then we approach the question that is addressed in thecollection of experts quoted in the NYT blog: What of "teaching to the test"? Here I think we approach something Popham, author of the above cited book, got wrong. Stotsky gives the example of a reading comprehension exam where during instruction the teacher teaches students skills of reading comprehension. This is clearly what we expect teachers to do and best practices. We expect students to have learned things such that they are better able to succeed on the test post-instruction. Popham would say this isn't teaching to the test as Stotsky says but teaching to the skills neccesary to succeed on the test. Popham gives two criteria for determining if the test prep a teacher is providing is appropriate and then 5 possible methods of test prep. The first criteria is professional ethics. In essence, teachers should behave ethically. This is obvious and cannot be violated without also violating the second criterion, and as such adds little to nothing. We are thus left with the second criterion: test prep must not improve test scores without also improving the underlying skill.
Among his five test prep practices, he lists (and bashes) previous form prep, which anyone who has taken a standardized test prep class has likely experienced. To prepare you for this year's SAT, you are given old SATs so you become familiar with the types of questions asked. He has similar objection to same-format preparation, where questions are asked in the same way (and only that way) as they are asked on the exam. He claims that this will boost scores artificially. However, he also claims that if it became widely known that these methods were used, there would be public outrage. That seems to me patently ridiculous. The public is aware and does not say: That is outrageous, but instead: I must buy my child one of those exam books, so that s/he too can prepare that way. Booksellers sell these books publically. Clearly no public outrage has yet occurred. Yes, it does to some degree raise test scores without raising the underlying skills, such as reading comprehension, vocabulary, quantitative comparison, etc. (unless you assume the skill being tested is the ability to take standardized tests) However, Popham has no problem with generalized test-taking prep and varied format prep. Generalized test-taking prep will be things like how to eliminate bad choices on multiple-choice items (although get a good night's rest & eat a good breakfast should also fall into this category), while varied format includes such minor changes as presenting subtraction problems horizontally if the test always presents them vertically. I don't see such a minor change as turning a not-OK test prep method into an acceptable one.
Real issues would arise if the sampling nature of the exam were in some way broken. If I know which 20 questions will be asked with regard to long division, and I were to reveal any part of that information, then the exam ceases to be a random sampling of long division problems.
A lot of my thinking on this subject has been influenced by a book I had to read for my Assessment class. I disagree with him on some points, but we'll get to that in a moment. First the part I think he got right. The purpose of assessment, at least summative assessments (at the end of the unit/semester/year...all standardized tests are meant to be summative rather than formative. Formative assessments are expected to influence instruction going forward, such as indicating that additional review is needed before proceeding.) is to operationalize educational objectives. Psychologists want to measure happiness, but there is no ruler for that, so they devise a questionairre or some-such that measures happiness, or that they think is a good stand-in for measuring happiness. Educational objectives are things like "student can do any long division problem involving a three digit dividend and a two digit divisor" (that is more specific than usual, but it serves my purposes). The theoretical best test of that objective is asking the student to divide every number from 100 to 999 by every number from 10 to 99. However, 81000 long division problems would be a little tedious. And long before they did problem number 81000, or problem 8100 or problem 100, I'd know whether or not they could do it. So there will be sampling. I could ask 20 questions, assuming they were well selected, that might indicate the degree to which my educational objective has been achieved. The high-stakes assessments (eg final exams, state mandated exams, etc) are simply testing a larger number of educational objectives from a previously agreed upon list, as Sandra Stotsky notes in the first link.
Then we approach the question that is addressed in thecollection of experts quoted in the NYT blog: What of "teaching to the test"? Here I think we approach something Popham, author of the above cited book, got wrong. Stotsky gives the example of a reading comprehension exam where during instruction the teacher teaches students skills of reading comprehension. This is clearly what we expect teachers to do and best practices. We expect students to have learned things such that they are better able to succeed on the test post-instruction. Popham would say this isn't teaching to the test as Stotsky says but teaching to the skills neccesary to succeed on the test. Popham gives two criteria for determining if the test prep a teacher is providing is appropriate and then 5 possible methods of test prep. The first criteria is professional ethics. In essence, teachers should behave ethically. This is obvious and cannot be violated without also violating the second criterion, and as such adds little to nothing. We are thus left with the second criterion: test prep must not improve test scores without also improving the underlying skill.
Among his five test prep practices, he lists (and bashes) previous form prep, which anyone who has taken a standardized test prep class has likely experienced. To prepare you for this year's SAT, you are given old SATs so you become familiar with the types of questions asked. He has similar objection to same-format preparation, where questions are asked in the same way (and only that way) as they are asked on the exam. He claims that this will boost scores artificially. However, he also claims that if it became widely known that these methods were used, there would be public outrage. That seems to me patently ridiculous. The public is aware and does not say: That is outrageous, but instead: I must buy my child one of those exam books, so that s/he too can prepare that way. Booksellers sell these books publically. Clearly no public outrage has yet occurred. Yes, it does to some degree raise test scores without raising the underlying skills, such as reading comprehension, vocabulary, quantitative comparison, etc. (unless you assume the skill being tested is the ability to take standardized tests) However, Popham has no problem with generalized test-taking prep and varied format prep. Generalized test-taking prep will be things like how to eliminate bad choices on multiple-choice items (although get a good night's rest & eat a good breakfast should also fall into this category), while varied format includes such minor changes as presenting subtraction problems horizontally if the test always presents them vertically. I don't see such a minor change as turning a not-OK test prep method into an acceptable one.
Real issues would arise if the sampling nature of the exam were in some way broken. If I know which 20 questions will be asked with regard to long division, and I were to reveal any part of that information, then the exam ceases to be a random sampling of long division problems.