Standards in assessment

The attempts to define standards for computer based assessments have so far been largely unsuccessful. I think that one of the problems is the lack of clarity in the functional domain. Do we really understand the ontology of an assessment, and a question? I don't really think we do, and perhaps we never will. It is easy enough to find a way to define a multiple choice question in XML, but to do the same for 'any' question... I think it's a bit much to ask. You inevitable end up constraining what you can do.

This was one of the main problems with IMS QTI 1.2. The specification was incredibly limited, and thus any systems supporting the standard by definition were as limited. Worse, most systems did not even implement the standard fully, or correctly, and so QTI 1.2 never really got anywhere.

Version 2 was supposed to solve this. The specification (currently still a draft, version 2.1) is indeed a lot better, and allows for much more questions types, feedback, and scoring strategies. The problem is that to make all this possible in a standard XML definition, the specification has gotten rather complicated. I'm not sure it is a viable proposition to expect any vendor to support the standard in full. To make matters worse, all the big vendors, but also the Open University's OpenLearn, seem to be pushing the Common Cartridge, which includes an amended version of IMS QTI. 1.2. While it would be nice to be able to exchange and run questions that are embedded in learning materials from Blackboard or Moodle, it it does strike me as very unlikely that any vendor will now have a serious incentive to support anything beyond the Common Cartridge.

And so we might have to live with the fact that we are not going to have any standard for the exchange of question and/or assessment information. I'm not sure that s a bad thing though. We would probably be better of designing a decent system first, in stead of trying to standardise functionality that hasn't even been implemented anywhere yet. What use is interoperability, if there isn't anything to exchange?

Open Source Assessment tools

I attended a JISC - CETIS workshop today discussing the latest set of Open Source assessment tools that JISC has commissioned. The triads of projects is to deliver Authoring, item banking and delivery tools based on the IMS QTI 2.1 standard. For more information on the individual projects, they are:

• AQuRate (The authoring tool, developed by Kingston University)
• Minibix (The item banking tool, developed by Cambridge University)
• ASDEL (The delivery engine, developed by Southampton University)

While I applaud the 3 project teams for the work that they have done, I must also say that I was concerned.

There have been a lot of projects funded by the sector that were supposed to kick start the development and uptake of standards-based e-assessment. Projects like TOIA, APIS, R2Q2. None of these project ever became much more then a proof of concept. The current set of projects seems to be on course to be heading that same way. None of these projects ever have the institutional backing of a stakeholder that understands the long term business need for such a solution. In stead they are research bids by researchers and developers who's only mandate is to fulfill the requirements of the project plan, and who's only resources are those granted by, in this case, JISC. And so after the kick start the project dies, as the funding dries up.

Are we then forever in the hands of the commercial vendors? I certainly hope not, as so far they have been completely unable to impress me with their products. Most commercial tools offer little of the pedagogical affordances and support that they should be giving and are often even technically rather weak. I deeply believe that the only serious hope that we have in ever getting a valuable and usable set of assessment tools is by collaboratively developing them ourselves. Unfortunately the success that Moodle has become in the world of VLE's seems unlikely to be repeated in the area of e-Assessment anytime soon.

Ideas anyone?