Proceedings from Work-based Learning Futures

I blogged in April about my excellent experience attending and presenting to the work-based Learning Futures conference in Buxton. I announced then that the proceedings would most likely be published as a special UVAC publication, and it now has. You can read op on the contribution from the e-APEL project-team that I am involved in by following this link. I thoroughly recommend having a look at the entire publication, as I think us learning technologists would benefit from realizing that innovation does not always mean technology.

The echo of teaching

I thought I'd have a go at answering the The Learning Circuits Blog: December Big Question - What did you learn about learning? One of the projects I have worked on this year, is the development of a tool supporting the Accreditation of Prior Learning (APL). It has been truly enlightening for me in many ways.

APL is going to be a core activity of a lot of Universities I recon. Content no longer seems to be the core business of the sector, as has been shown by initiatives such as Open Learn. Coming to grips with this is a bit like trying to understand Open Source business models I think, it requires a fundamental rethink of what is valuable. For most universities I think that value is going to increasingly lie in guidance and coaching on the one side, and assessment and accreditation on the other.

There seems to be a problem with accreditation of learning that has not taken place within the controlled environment of a course though. Very few universities are serious about APL, and I can't help but wonder why. Part of it, I am sure, is to do with fees and such, but not all. After some reflection I think we must also admit that APL exposes some critical weaknesses of our assessment processes. In theory our assessments are supposed to discriminate between those learners that have attained certain outcomes, and those who haven't. If that was all there was to it, then surely learners claiming APL could as simple as doing the regular assessment, but without attending the course.

The reason this isn't common practice I think, is that most assessments don't really assess the right outcomes. Most assessments I think are designed to trigger an echo of teaching, and not of learning. And of course our teaching is so good, that if the learner echo's a confirmation of our teaching, then surely that means the intended learning has taken place. But what if learning has not been a result of our teaching? Suddenly we cannot short circuit the inherent difficulty of assessing competence by resorting to looking for the echo of teaching.

I think it would be interesting to dig into assessment practices used by recruitment agencies. In a way they are asked to make assessments that employers aren't confident we have made. Furthermore, whatever they assess is always without the luxury safety net of knowing what has probably been learned and by which means.

Psychometrics versus pedagogy

One of the things I noticed during the workshop I attended last week, is the fundamental differences to the approach of computer based assessment between psychometrists and educators. it all boils down to the use and value of statistics.

I think within education we often don't evaluate our teaching and assessment practice enough, in particular by means of objective standards. For assessment practice the most well known methods of scientific evaluation are Classical Test Theory and, most importantly, Item Response Theory. I don't think many educators really bother with these, and some will stare very blankly should I ever bring up these terms in conversation. In stead we rely on our 6th pedagogic sense that rather mysteriously enables us to divine what assessment methods and questions work, and which do not.

The psychometric approach is radically different. Almost religiously sometimes, items are tested and analyzed using IRT. the most meticulous detail (question order, item order, delivery medium etc.) is reviewed for it's influence on responses. These statistical analysis only focus on one thing though, and that is the alignment of the item with the overall test. What the church of IRT seems to sometimes forget however, is to question whether or not the test in itself actually measures what it is assumed to measure. To a degree IRT is a bit of a circle argument, if not used carefully and in conjunction with other arguments.

It seems to me we could both do with a bit of each others zeal. Educators should really try and build in some structured objective evaluation of their assessment practices, and psychometrists should perhaps question the appropriateness of their means more fundamentally.