In physics the Heisenberg uncertanty principle is a well known limitation of measurements. The principle explains the fundamental conflict between establishing a particle's speed, and it's position. The more we focus on making one of these explicit, the more uncertain our understanding of the other. This is not a shortcomming of our instruments or anything like that, it is a fundamental property of the universe. Perhaps it is time that we realise that in education our ability to measure things like student attainment is even more limited. It is not a limitation that we can overcome by measuring more. In fact that just makes the situation worse, as our measurements then start to influence what we are trying to observe and ussualy not for the better. This effect is called the observer effect, and it is a crucial element to take inot consideration when delivering high stakes assesments.
With the increasing pressures on education to measure and report, calls to take into account the observer effect (although ussualy not referred to as such) are becomming louder. The National Union of Teachers conference has spoken out against the practice, and I have raised the issue on this blog before in the post titled High stake national assessments and ranking. A very thoughtful analysis of the problem is given by Wesley Fryer in his post Raising expectations. Wesley argues for the return of teachers designing and delivering high stakes tests, in stead of these being set by governments and awarding bodies. While a lot can be said in favor of this idea, I do think it is important to realise that this is only possible if we combine this with a very serious upgrade of the staff development that is given to our teaching staff on the subject of assessment. Nevertheless, Wesleys post is definetly worth a read. Especially this little gem:
"... bestowed upon the plebeian masses by the academic elites filling the hallowed halls of commercial companies now profiting handsomely from our myopic focus on summative, simplistic, high-stakes assessments". That must be the best and most colourful descriptions of our asessment culture that I have ever read.
Tuesday, 25 March 2008
Monday, 24 March 2008
Marking free text
One of the frequent criticisms on e-Assessment is the perceived limits in item types that can be supported by technology. While there are long debates to be had about assessing higher order skills with constrained response item types, I don't think these debates are going to take away the prime concern: Free text items.
I must say that I have serious doubts about marking free text by computer. I don't know enough about the principles involved to say this with any sort of authority, but I am aware of the kind of heuristics used in automated essay marking for instance. These heuristics are often grammatical and rhetorical in nature, and have fairly little to do with the subject matter (although it must be said that many human markers have been shown to use similar heuristics). Nevertheless, interesting progress is being made in this area, and eventually I am sure that language processing will be commonplace.
One of the interesting project that I recently became aware of, is the OpenComment project, which is lead by Denise Whitelock at the Open University. The project is looking to use latent semantic analysis to analyse learners responses to open ended questions in history and philosophy. Another interesting fact is that the project is developing this as a question type in Moodle, and so it should be relatively easy for everyone to reap the benefits of this technology within their own learning environments.
Automated marking is by no means the only value of using technology in assessment. The OpenMentor project, again from the Open University, is a great example. OpenMentor compares a mark assigned to a piece of work to the amounts of positive and negative feedback given, and checks this for consistency. In this way it can help in the coaching process of new teachers. Given the importance of feedback, I think it's wonderful to have explicit standards and training in giving it.
The ABC (Assess By Computer) software has so far escaped my radar. I wasn't aware of it until queried by the Times Higher Education for the article they were doing. The software has a support role similar to OpenMentor, but this time the support is provided around the marking process. The software can highlight keywords, compare answers to model answers and more. All of this for the sole purpose of making it easier on the human marker, but also improve consistency between human markers. Especially the latter is very welcome I think, as marking open ended questions and assignments can sometimes be somewhat of a dark art.
I only just discovered that bits of the e-mail I sent to the reporter actually appear in the article. If I would have known that I probably would have paid a bit more attention to my grammar :S
I must say that I have serious doubts about marking free text by computer. I don't know enough about the principles involved to say this with any sort of authority, but I am aware of the kind of heuristics used in automated essay marking for instance. These heuristics are often grammatical and rhetorical in nature, and have fairly little to do with the subject matter (although it must be said that many human markers have been shown to use similar heuristics). Nevertheless, interesting progress is being made in this area, and eventually I am sure that language processing will be commonplace.
One of the interesting project that I recently became aware of, is the OpenComment project, which is lead by Denise Whitelock at the Open University. The project is looking to use latent semantic analysis to analyse learners responses to open ended questions in history and philosophy. Another interesting fact is that the project is developing this as a question type in Moodle, and so it should be relatively easy for everyone to reap the benefits of this technology within their own learning environments.
Automated marking is by no means the only value of using technology in assessment. The OpenMentor project, again from the Open University, is a great example. OpenMentor compares a mark assigned to a piece of work to the amounts of positive and negative feedback given, and checks this for consistency. In this way it can help in the coaching process of new teachers. Given the importance of feedback, I think it's wonderful to have explicit standards and training in giving it.
The ABC (Assess By Computer) software has so far escaped my radar. I wasn't aware of it until queried by the Times Higher Education for the article they were doing. The software has a support role similar to OpenMentor, but this time the support is provided around the marking process. The software can highlight keywords, compare answers to model answers and more. All of this for the sole purpose of making it easier on the human marker, but also improve consistency between human markers. Especially the latter is very welcome I think, as marking open ended questions and assignments can sometimes be somewhat of a dark art.
I only just discovered that bits of the e-mail I sent to the reporter actually appear in the article. If I would have known that I probably would have paid a bit more attention to my grammar :S
e-Assessment centre for the South-West?
Last Thursday I attended a conference / workshop that was exploring the business case of setting up an e-Assessment centre in the South West of England. The conference was organised by Dr. Kevin Hapeshi from the University of Gloustershire with talks from Denise Whitelock (always inspiring) and myself. I must say my presentation skills are somewhat out fo practice, and I coudl have probably done with reviewing the art of speaking.
The day took an unexpected turn during the afternoon sessions, where we were going to discuss the details of a regional e-assessment centre. Both work groups came to the surprising conclusion that perhaps this was actually not such a good idea after all. There are plenty of challenges that we need to face in this domain, but none of them really benefit from a regional approach.
The biggest challenge is the development of mature tools, standards and practice. I've blogged about this in the past ( see Standards in assessment, Open Source Assessment tools, The ideal assessment engine). This is not a challenge we can face as universities, or regions however, it is something that requires (inter-) national collaboration. Many of the other challenges are institutional. They revolve around generating awareness and changing culture and practice. This is not something you can do from the outside. We find it hard to change practice on other other campuses of the University. Changing practice requires a proximity to practitioners, and to the learning and teaching strategies and strategic stakeholders. I don't think that proximity is something you can achieve with a regional (and thus external) centre.
There are of course hybrid models, whereby Universities could collaborate in virtual networks, nominating and funding members of their own staff to work in and with the centre. but this might just all become a rather artificial model tailored mostly towards fitting the proposed solution, and not the problem.
The day took an unexpected turn during the afternoon sessions, where we were going to discuss the details of a regional e-assessment centre. Both work groups came to the surprising conclusion that perhaps this was actually not such a good idea after all. There are plenty of challenges that we need to face in this domain, but none of them really benefit from a regional approach.
The biggest challenge is the development of mature tools, standards and practice. I've blogged about this in the past ( see Standards in assessment, Open Source Assessment tools, The ideal assessment engine). This is not a challenge we can face as universities, or regions however, it is something that requires (inter-) national collaboration. Many of the other challenges are institutional. They revolve around generating awareness and changing culture and practice. This is not something you can do from the outside. We find it hard to change practice on other other campuses of the University. Changing practice requires a proximity to practitioners, and to the learning and teaching strategies and strategic stakeholders. I don't think that proximity is something you can achieve with a regional (and thus external) centre.
There are of course hybrid models, whereby Universities could collaborate in virtual networks, nominating and funding members of their own staff to work in and with the centre. but this might just all become a rather artificial model tailored mostly towards fitting the proposed solution, and not the problem.
Subscribe to:
Posts (Atom)
