Marking free text

One of the frequent criticisms on e-Assessment is the perceived limits in item types that can be supported by technology. While there are long debates to be had about assessing higher order skills with constrained response item types, I don't think these debates are going to take away the prime concern: Free text items.

I must say that I have serious doubts about marking free text by computer. I don't know enough about the principles involved to say this with any sort of authority, but I am aware of the kind of heuristics used in automated essay marking for instance. These heuristics are often grammatical and rhetorical in nature, and have fairly little to do with the subject matter (although it must be said that many human markers have been shown to use similar heuristics). Nevertheless, interesting progress is being made in this area, and eventually I am sure that language processing will be commonplace.

One of the interesting project that I recently became aware of, is the OpenComment project, which is lead by Denise Whitelock at the Open University. The project is looking to use latent semantic analysis to analyse learners responses to open ended questions in history and philosophy. Another interesting fact is that the project is developing this as a question type in Moodle, and so it should be relatively easy for everyone to reap the benefits of this technology within their own learning environments.

Automated marking is by no means the only value of using technology in assessment. The OpenMentor project, again from the Open University, is a great example. OpenMentor compares a mark assigned to a piece of work to the amounts of positive and negative feedback given, and checks this for consistency. In this way it can help in the coaching process of new teachers. Given the importance of feedback, I think it's wonderful to have explicit standards and training in giving it.

The ABC (Assess By Computer) software has so far escaped my radar. I wasn't aware of it until queried by the Times Higher Education for the article they were doing. The software has a support role similar to OpenMentor, but this time the support is provided around the marking process. The software can highlight keywords, compare answers to model answers and more. All of this for the sole purpose of making it easier on the human marker, but also improve consistency between human markers. Especially the latter is very welcome I think, as marking open ended questions and assignments can sometimes be somewhat of a dark art.

I only just discovered that bits of the e-mail I sent to the reporter actually appear in the article. If I would have known that I probably would have paid a bit more attention to my grammar :S