schools test to be scored electronically
How Australia plans to mark NAPLAN with cognitive computing
By Paris Cowan on Apr 29, 2015 12:25 PM
Filed under Software
National schools test to be scored electronically from 2017.
Read more: http://www.itnews.com.au/News/403322,how-australia-plans-to-mark-naplan-with-cognitive-computing.aspx#ixzz3dFBvRduw
Australia’s national curriculum authority, ACARA, has announced plans to apply cognitive computing technology to marking the annual benchmarking exams sat by Australian schoolkids.
Each year, students in years three, five, seven and nine sit common NAPLAN tests to map their progress in reading, writing and arithmetic - the results of which also inform the mySchool performance website.
The Australian Curriculum Assessment and Reporting Authority (ACARA) plans to start marking two-to-three page written components of the test using cognitive computing from 2017.
It is yet to select a specific technology, ACARA general manager Dr Stanley Rabinowitz told iTnews, but the organisation has already run a proof of concept with an unnamed platform and is working towards reproducing the accuracy of similar experiments overseas.
“There are a lot of good options out there,” he said of the burgeoning artificial intelligence and machine learning market.
Solutions ACARA is considering involve teachers marking and scoring around 1000 exam papers according to a detailed rubrik.
The completed exams, the teachers' marks and the marking criteria are then fed into the program, which will reverse engineer or ‘learn’ a criteria for spotting the score points in each paper from the input data.
Rabinowitz said the computer would identify the same sorts of writing elements that a teacher looks out for, like “grammar, the structure of sentences, the variation in vocabulary, right down to more mundane elements like spelling,” he said.
The results of the trials have been assessed according to two criteria: whether the computer scores correlate to the human scores within the same margin as two different human markers; and whether the scores generated by the computer distribute in the same way as an equivalent number marked by humans.
Rabinowitz said the trials show the artificial intelligence solutions perform as well, or even better, than the teachers involved.
He sought to reassure parents and teachers that protections would be built into the auto-marking process to ensure quality and consistency, just as they are built into existing human marking procedures.
Teachers groups have already responded angrily to the plan, arguing computer-based marking undermines the creative aspect of student performance and learning.
Rabinowitz said the initial calibration of the program would throw up a set of indicators pointing to unusual papers that are better off being marked by human markers.
Five or ten percent of papers would be double marked by expert assessors to ensure the program remains within marking parameters, he said, or to indicate that it needs to be “re-trained”, mirroring the process currently used to double check the work of human markers.
Teachers will “still be very much involved” in the process, Rabinowitz insisted.
“We are not the only ones doing this. It is happening more and more around the world,” he said.
The use of cognitive computing is expanding in Australia: ANZ Bank and Deakin University were two of the early adopters of IBM's Watson technology, and Australia's Customs agency this week announcing it would also deploy Watson to gain better insight into large amounts of public data.
Copyright © iTnews.com.au . All rights reserved.
By Paris Cowan on Apr 29, 2015 12:25 PM
Filed under Software
National schools test to be scored electronically from 2017.
Read more: http://www.itnews.com.au/News/403322,how-australia-plans-to-mark-naplan-with-cognitive-computing.aspx#ixzz3dFBvRduw
Australia’s national curriculum authority, ACARA, has announced plans to apply cognitive computing technology to marking the annual benchmarking exams sat by Australian schoolkids.
Each year, students in years three, five, seven and nine sit common NAPLAN tests to map their progress in reading, writing and arithmetic - the results of which also inform the mySchool performance website.
The Australian Curriculum Assessment and Reporting Authority (ACARA) plans to start marking two-to-three page written components of the test using cognitive computing from 2017.
It is yet to select a specific technology, ACARA general manager Dr Stanley Rabinowitz told iTnews, but the organisation has already run a proof of concept with an unnamed platform and is working towards reproducing the accuracy of similar experiments overseas.
“There are a lot of good options out there,” he said of the burgeoning artificial intelligence and machine learning market.
Solutions ACARA is considering involve teachers marking and scoring around 1000 exam papers according to a detailed rubrik.
The completed exams, the teachers' marks and the marking criteria are then fed into the program, which will reverse engineer or ‘learn’ a criteria for spotting the score points in each paper from the input data.
Rabinowitz said the computer would identify the same sorts of writing elements that a teacher looks out for, like “grammar, the structure of sentences, the variation in vocabulary, right down to more mundane elements like spelling,” he said.
The results of the trials have been assessed according to two criteria: whether the computer scores correlate to the human scores within the same margin as two different human markers; and whether the scores generated by the computer distribute in the same way as an equivalent number marked by humans.
Rabinowitz said the trials show the artificial intelligence solutions perform as well, or even better, than the teachers involved.
He sought to reassure parents and teachers that protections would be built into the auto-marking process to ensure quality and consistency, just as they are built into existing human marking procedures.
Teachers groups have already responded angrily to the plan, arguing computer-based marking undermines the creative aspect of student performance and learning.
Rabinowitz said the initial calibration of the program would throw up a set of indicators pointing to unusual papers that are better off being marked by human markers.
Five or ten percent of papers would be double marked by expert assessors to ensure the program remains within marking parameters, he said, or to indicate that it needs to be “re-trained”, mirroring the process currently used to double check the work of human markers.
Teachers will “still be very much involved” in the process, Rabinowitz insisted.
“We are not the only ones doing this. It is happening more and more around the world,” he said.
The use of cognitive computing is expanding in Australia: ANZ Bank and Deakin University were two of the early adopters of IBM's Watson technology, and Australia's Customs agency this week announcing it would also deploy Watson to gain better insight into large amounts of public data.
Copyright © iTnews.com.au . All rights reserved.