The ideal assessment engine

I've been looking into criteria for assessment technologies a lot lately. One reason is that we are looking into migrating our current system to a new platform (as the old one, Authorware, is no longer supported). The other reason is that I have been invited by the Joint Research Centre to take part in a workshop on quality criteria for computer based assessments. I will be posting on the outcomes of that workshop next week. For now though, here are some of my thoughts on the topic.

Flexibility
The main strength of our current system is flexibility. This has several aspects, that are all important in their own right:

• Flexibility in design: The layout of the question can be modified as desired, using media and such to create an authentic and relevant presentation
• Flexible interactions: There is no point in systems that have parameterized 5 question types for you, and all you can do is define a title, question text, alternatives and select the right answer. Interactions testing and supporting higher order skills are, or should be, more complex then that.
• Detailed and partial scoring: A discriminating question does not just tell you whether you were completely right, or completely wrong. It can tell you the degree to which you were right, and what elements of your answer had any value. It might also penalize you for serious and fundamental mistakes.
• Detailed feedback: A lot of mistakes learners make are predictable. If we allow assessment systems to capture these mistakes and give targeted feedback, learners can practice their skills while lecturers can focus there time on more in depth problems that require their personal engagement.
  
• Extensive question generation and randomization options: For the re-usability of assessments, generating questions using rules and algorithms given a single question almost infinite re usability. On the assessment level, the same is true for assessment generation based on large banks with questions tagged with subject matter and difficulty.

So far, no real news for TRIADS users (although no proprietary system I know of really supports this well).

Questions without assessments
As Dylan Wiliam so eloquently worded at the ALT-C conference (you can find his podcast on the matter on http://www.dylanwiliam.net/), the main value in learning technology lies in "to allow teachers to make real-time instructional decisions, thus increasing student engagement in learning, and the responsiveness of instruction to student needs." I could not agree more. However, this means that questions should not just exist within the assessment, but instead be embedded within the materials and activities. Questions become widgets that can of course still function within an assessment, but also work on their own without loosing the ability to record and respond to interaction. This, as far as I'm aware, is unchartered territory for assessment systems. Territory that we hope to explore in the next iteration of our assessment engine.

e-APEL article in Response

The new online journal Response has published a 'work in progress' report I wrote on the e-APEL project that I'm involved in. I'm afraid it is rather dated, as the journal took more then 8 months to actually publish this version. Still, for those interested in the accreditation of prior learning, or IT projects in education in general, it might be a worthwhile read.

FREMA

At a JISC meeting this Thursday I was reminded of the FREMA project. I had been aware of their attempts to map the domain of e-Assessment for a number of years now, but I was not aware of some recent developments. Most interesting to me was their use of a semantic wiki. I had never heard of the concept in all honesty, but I found the idea fascinating. In particular for the purpose of Knowledge management and dissemination I think the possibilities here are truly significant.

for the FREMA project in particular, one of the things they were able to do as a result of using this technology, is a gap analysis of their understanding of the domain, but also of the available solutions within the domain. Unfortunately there are still quite a few gaps to be filled in the area of e-assessment, but at least through resources like these we can maximize our efficiency in finding existing solutions, and focusing our efforts on those gaps where the needs are most pressing.

Is web 2.0 dumbing us down?

As some of you might know, I'm an avid listened of podcasts (mainly to make my daily hour long commute seem a little less wasteful). Two of the recent casts I listened to grabbed my specific attention.

The first is a presentation titled Republic 2.0 by Cass Sunstein. In this presentation mr. Sunstein explains the risks of web 2.0 to democracy. While the increased access to the expression and consumption of information and opinion seems like a wonderful thing, there are downsides to how we engage with blogs, wiki's and social networks. Due to the vast amount of information out there, but also because of the nature of these new social artifacts, we tend to expose ourselves only to information and opinions from those that we are close to (ideologically or otherwise). Research has shown that in homogeneous groups like these, polarization takes place: views and opinions become more singular and extreme.

This is a concern in itself, and something to keep in mind when considering aspects of our education system, such schools based on subgroups of our society over dimensions such as religion, class or even geography. The concern got a new dimension for me however after listening to one of the seminars of the Long Now Foundation In his talk, Ignore Confident Forecasters, Philip Tetlock shares some insights from his research on peoples ability to make appropriate predictions about complex future events (in this case in world politics). He found 2 types of thinking, leading to 2 distinct patterns of predictions. one group was classified as 'hedgehogs'. These were people that had a single specialism or conviction, and tried to explain everything in the world from this single perspective. The second group, the 'foxes' were more broad in their thinking, and the constructs they applied to solving problems. Foxes significantly outperformed the hedgehogs.

So this begs the question: If we allow ourselves to be exposed only to those views and people that we have sympathy with, something the web increasingly allows us to do, are we really depriving ourselves of the tools for a balanced and effective mental development?

Simple tools

It's not about the technology. We often say it, but we rarely really mean it I think. Let's be fair now: Technology is kinda fun. I know I get carried away far to easily with new shiny things. But sometimes I forget that not everyone is paid to play with new shiny things. Some of us actually have to teach students on a regular basis. So for those colleagues, here's some lovely simple tools that I think can give you a lot of bang for just a little investment.

Course Genie
Course Genie is a lovely tool. It allows you to create nice looking materials and quizzes without any skills other then MS Word. It's a great tool if you want to develop something slightly more interactive then just uploading a module specification or PowerPoint to you VLE, and it integrates nicely with Moodle, WebCT and Blackboard. The quiz options are relatively simple, but actually have some powerful capabilities such as rich feedback based on answers given. The downside is that Course Genie does not save answers, scores, or track progress

QUIA
God knows who thought up the name for this one. In all honesty, QUIA would not be a tool that I'd use myself. it does a lot of things (games, quizes, surveys), but it does none of them very well. Nevertheless if you want a cheap tool that instantly will allow you to create a lot of simple interactive resources this might be the tool for you. The absolute plus to QUIA is that you instantly tap in to the whole QUIA community, which allows you to share and rate all developed materials. QUIA also has a very decent result tracking option, which even allows you to expert a detailed result analysis to Excel.

Electronic Voting Systems
A great way to make teaching, especially in larger groups, more interactive. It's a great way to find out what students think, understand or want during your lesson, so that you can adapt to their needs in the appropriate manner. We've been using Turning Point for a short while now, and so far I am quite impressed. It's extremely easy to set up a simple poll supported by PowerPoint. it is however also possible to much more complex thing, such as linking in responses to demographics.

Student complaints

I had a chat with a colleague last week, as I was looking for some feedback on the e-assessment she had run this year. She told me about this student complaint, and how it was handled. In all honesty I still can't quite believe we can be this stupid.

Some of our lecturers are experimenting with feedback during summative exams. This means learners immediately know whether they answered a question right of wrong, and sometimes why. It also allows them to instantly view their result at the end of the exam. So far, results from the experiment have been quite good (except for learners who really haven't mastered any of the subject matter, who obviously get rather depressed by this whole affair).

A student that had previously taken part in one of these experimental exams, was now taking part in a regular exam; no feedback and no immediate score. The lecturer, as usual, had received the transcripts from us and, after a possible moderation, had published these to the learners. This learner however was apparently convinced that the lecturer had fiddled with the results. Why else would they not have published the results upon the exam finishing? Apparently the student made quite a scene, upon which the programme leader decided to adhere to the students demands and make the exam available again for his perusal for another week. He was then also granted a resit. I really don't understand the problem here.

Let's look at a normal exam. When you hand in the paper, do you get an immediate result? No, of course you don't; a lecturer takes it away, marks it, maybe a colleague moderates it, and the mark is published. It's the lecturer's job to fiddle with the results, that's what we are paying them for isn't it? They look at the answers provided, and make a judgment on the extent to which they satisfy the assessment criteria? Introducing (partial) marking by a computer can only make this process more objective, not less!

It's no wonder lecturers and teachers sometimes complain about the lack of respect learners give them. Because this isn't just about caving in to a student and giving them their way. This also sends a message that this lecturer was wrong. It sends the message that it is the lecturer who has to cater to the students every whim, as she has now been instructed to do. I understand this lecturer will not be teaching here next year, and if this story is true, I can understand why.

It's funny how things coincide sometimes. Today a student came to our office, wondering if we could help her develop an assessment in support of her dissertation. For some reason several students have made this request this year, while none used to before. Personally I think it's a wonderful thing, and testament to how students now view technology as an integral and important part of their lives and careers. It also shows how they are much less subject to pigeonholing technology.

For years the Centre for Interactive Assessment Development has been supporting lecturers by developing rich assessments. The university of Derby has in general adopted a far more innovative approach then most, progressing e-Assessments far beyond the domain of the multiple choice quiz. still the applications sought for innovative assessment practice have been rather limited. Primarily assessments were measurements of learning, mostly summative or formative only in the sense of providing practice and a benchmark for a later summative exercise. Assessments that actually teach, or diagnose are a relative new addition to our portfolio. Assessments for other purposes, such as research or evaluation, have never even been considered as part of the centres value and expertise. This is something I am desperate to change. I'm glad at least students seem to agree with me on that one.