The ideal assessment engine

I've been looking into criteria for assessment technologies a lot lately. One reason is that we are looking into migrating our current system to a new platform (as the old one, Authorware, is no longer supported). The other reason is that I have been invited by the Joint Research Centre to take part in a workshop on quality criteria for computer based assessments. I will be posting on the outcomes of that workshop next week. For now though, here are some of my thoughts on the topic.

Flexibility
The main strength of our current system is flexibility. This has several aspects, that are all important in their own right:

• Flexibility in design: The layout of the question can be modified as desired, using media and such to create an authentic and relevant presentation
• Flexible interactions: There is no point in systems that have parameterized 5 question types for you, and all you can do is define a title, question text, alternatives and select the right answer. Interactions testing and supporting higher order skills are, or should be, more complex then that.
• Detailed and partial scoring: A discriminating question does not just tell you whether you were completely right, or completely wrong. It can tell you the degree to which you were right, and what elements of your answer had any value. It might also penalize you for serious and fundamental mistakes.
• Detailed feedback: A lot of mistakes learners make are predictable. If we allow assessment systems to capture these mistakes and give targeted feedback, learners can practice their skills while lecturers can focus there time on more in depth problems that require their personal engagement.
  
• Extensive question generation and randomization options: For the re-usability of assessments, generating questions using rules and algorithms given a single question almost infinite re usability. On the assessment level, the same is true for assessment generation based on large banks with questions tagged with subject matter and difficulty.

So far, no real news for TRIADS users (although no proprietary system I know of really supports this well).

Questions without assessments
As Dylan Wiliam so eloquently worded at the ALT-C conference (you can find his podcast on the matter on http://www.dylanwiliam.net/), the main value in learning technology lies in "to allow teachers to make real-time instructional decisions, thus increasing student engagement in learning, and the responsiveness of instruction to student needs." I could not agree more. However, this means that questions should not just exist within the assessment, but instead be embedded within the materials and activities. Questions become widgets that can of course still function within an assessment, but also work on their own without loosing the ability to record and respond to interaction. This, as far as I'm aware, is unchartered territory for assessment systems. Territory that we hope to explore in the next iteration of our assessment engine.

Simple tools

It's not about the technology. We often say it, but we rarely really mean it I think. Let's be fair now: Technology is kinda fun. I know I get carried away far to easily with new shiny things. But sometimes I forget that not everyone is paid to play with new shiny things. Some of us actually have to teach students on a regular basis. So for those colleagues, here's some lovely simple tools that I think can give you a lot of bang for just a little investment.

Course Genie
Course Genie is a lovely tool. It allows you to create nice looking materials and quizzes without any skills other then MS Word. It's a great tool if you want to develop something slightly more interactive then just uploading a module specification or PowerPoint to you VLE, and it integrates nicely with Moodle, WebCT and Blackboard. The quiz options are relatively simple, but actually have some powerful capabilities such as rich feedback based on answers given. The downside is that Course Genie does not save answers, scores, or track progress

QUIA
God knows who thought up the name for this one. In all honesty, QUIA would not be a tool that I'd use myself. it does a lot of things (games, quizes, surveys), but it does none of them very well. Nevertheless if you want a cheap tool that instantly will allow you to create a lot of simple interactive resources this might be the tool for you. The absolute plus to QUIA is that you instantly tap in to the whole QUIA community, which allows you to share and rate all developed materials. QUIA also has a very decent result tracking option, which even allows you to expert a detailed result analysis to Excel.

Electronic Voting Systems
A great way to make teaching, especially in larger groups, more interactive. It's a great way to find out what students think, understand or want during your lesson, so that you can adapt to their needs in the appropriate manner. We've been using Turning Point for a short while now, and so far I am quite impressed. It's extremely easy to set up a simple poll supported by PowerPoint. it is however also possible to much more complex thing, such as linking in responses to demographics.