Evidence based teaching

One of the topics that came up several times over the past days in Reykjavik, is that of the differences in culture around assessment. Different countries have different ways in which they perceive and deal with assessment, and this can have a significant impact on the effect of the assessments, and the success of the educational system as a whole.

One particularly interesting approach was outlined by Jakob Wandall, who's work in the Danish national tests I have blogged about last year in High stake national assessments and ranking. I tried to capture Jakob's slide on a picture, but unfortunately that failed rather miserably, so I have tried to recreate his message in the graphic below:




The graph outlines how both the focus of the assessment (on the horizontal axis) and the purpose for which the results are primarily used (on the vertical axis) vary from country to country. I thought the visualisation was very interesting. Comparing this to, for instance, the outcomes of the 2006 PISA, it is interesting to note that neither the approach of the Scandinavian schools (who focus primarily on learner focused formative assessment) nor the Anglo-Saxon approach 9that is much more heavy on the measurements of indicators for performance, tied in to funding) really yields the best results.

The starts of PISA are of course the Finnish, and the unique approach is apparent from this graph. in stead of moving somewhere between the top left and the bottom right of the graph, they sit toward the top right. The Finish system highly values national measurements, evaluating the success of the system by objective measurements. However these measurements are not tied to any control, either through formal channels or more informal ones such as public rankings. In stead the measurements made in the Finnish system have the purpose to inform teaching and learning. An evidence based approach to teaching shall we say.

When I translate this to our own practice, I can't help but relate this to demands to increase the amounts of formative assessment in our teaching. And while I am sympathetic to these demands, these assessments are similar to those in the top left of the above graph, informing and supporting individual learning processes. And so perhaps in stead of focusing primarily on formative individual assessment, we should focus (also) on assessment and evaluation that informs teaching. Building an infrastructure through which lecturers can stay in touch with the progress, successes and difficulties of all their students, and modify their teaching based on this understanding continuously.

Free at last?

Research done by PISA has shown convincingly that school systems whit a high degree of autonomy perform better (see my post on SAT troubles for a bit more info) . It seems that the Liberal Democrats have now formally adopted this position, and have outlined plans to scrap the national curriculum. A brave move. It will be interesting to see how this discussion unfolds, and if it will survive the inevitable backlash from the control brigade.

SAT troubles

There's been a lot of upheaval this week about SAT tests. After a report published by the Children, Schools and Family committee of the House of Commons, MPs warn that national Sats tests distort education, which then lead to the schools minister defending the Sats, followed by technical difficulties with the tests. Personally I am not convinced the tests are really the problem.

One of the keynotes at the Blackboard Europe 2008 conference was given by Andreas Schleicher, the director of the PISA program for the OECD. He presented a very compelling set of ideas around successful (secondary) educations. Some of the conditions he identified (and all of these are based on the data gathered by the programme over the past years) are:

• No stratification. Education systems that have separate streams, schools and or qualifications for learners based on their performance tend to do poorly. An example of this is the Dutch system, where secondary education is stratified over VMBO, HAVO and VWO based on a learners performance in primary school. The British system actually comes out quite well here (if we ignore the stratification that takes place because of the divide between private and public schools that is).
• Standards. It is important to work to common standards. Central examinations are one way of enforcing common standards, and so the SAT tests do satisfy this condition.
• Autonomy. It is crucial for schools and teachers to have a high degree of autonomy as long as their performance raises no concerns. Here we obviously fail completely as the British system dictates how schools teach and assess to a very high degree.
• High Expectations, challenge and support. Both for teachers and learners, education should provide challenge, the expectation of high performance, but also plenty of support (staff development for instance). I think this is another area in which we fail to deliver.

Our main problem lies in the area of autonomy. We no longer trust our teachers and schools do do what they do best based on their professional judgments. In stead there is this weird notion that education is better served by central generic judgments made by policymakers. The problem with SATs isn't that they provide a common high stakes benchmark for learners. The problem is that this information is abused for public league tables and the like, which inevitably leads to pressures on learners that have nothing to do with their personal learning. It's the same pressures that lead to Universities coercing students into filling out the national student survey more favorably.

In Finland schools have no idea about their performance related to their neighbors. Funny enough in Finland it doesn't really make a difference. Only 4% of the variance in scores on the PISA tests can be assigned to the difference in quality between schools. Finnish schools have around 9 applicants for every position offered, and this is not because of higher salaries or anything like that. It is because the system in Finland provides a challenging environment in which people are valued, can grow and develop and actually make a difference.

Assessment standards: a manifesto for change

A group of 34 prominent academics has taken a laudible stance against our current assessment culture (see also this THE article). You can find the manifesto and it's supporters at the bottom of this post. Point 3 especially I think is very poignant within the context of e-Assessment, where our obsession with the measurable (I'm thinking Item Response Theory here) has gotten way out of hand at the expense of validity.

The Weston Manor Group

Assessment standards: a manifesto for change

1. The debate on standards needs to focus on how high standards of learning can be achieved through assessment. This requires a greater emphasis on assessment for learning rather than assessment of learning.

2. When it comes to the assessment of learning, we need to move beyond systems focused on marks and grades towards the valid assessment of the achievement of intended programme outcomes.

3. Limits to the extent that standards can be articulated explicitly must be recognised since ever more detailed specificity and striving for reliability, all too frequently, diminish the learning experience and threaten its validity. There are important benefits of higher education which are not amenable either to the precise specification of standards or to objective assessment.

4. Assessment standards are socially constructed so there must be a greater emphasis on assessment and feedback processes that actively engage both staff and students in dialogue about standards. It is when learners share an understanding of academic and professional standards in an atmosphere of mutual trust that learning works best.

5. Active engagement with assessment standards needs to be an integral and seamless part of course design and the learning process in order to allow students to develop their own, internalised, conceptions of standards, and monitor and supervise their own learning.

6. Assessment is largely dependent upon professional judgement, and confidence in such judgement requires the establishment of appropriate forums for the development and sharing of standards within and between disciplinary and professional communities.

Supporters:

Professor Trudy Banta

Dr Simon Barrie

Professor Sally Brown

Ms Cordelia Bryan

Dr Colin Bryson

Ms Jude Carroll

Professor Sue Clegg

Professor Linda Drew

Professor Graham Gibbs

Professor Anton Havnes

Dr Mary Lea

Dr Janet Macdonald

Professor Ranald Macdonald

Dr Debra Macfarlane

Dr Susan Martin

Professor Marcia Mentkowski

Dr Stephen Merry

Professor David Nicol

Professor Andy Northedge

Professor Lin Norton

Ms Berry O’Donovan

Dr Thomas Olsson

Dr Susan Orr

Dr Paul Orsmond

Professor Margaret Price

Professor Phil Race

Mr Clive Robertson

Dr Mark Russell

Dr Chris Rust

Professor Gilly Salmon

Professor Kay Sambell

Professor Brenda Smith

Professor Stephen Swithenby

Professor Mantz Yorke

Efficiency or effectiveness

The BBC reports that our government will be reviewing the efficiency of our exam system. I'm developing a rather serious aversion to efficiency, as it usually translates rather neatly into degradation.

It would be nice if the government would review the effectiveness of our exam system. Effectiveness is about reaching intended outcomes, not just about saving pennies. As the general secretary of the Association of School and College Leaders, John Dunford, said in the article: "It is vitally important that the government not only conducts a cost-benefit analysis of the current exam system but evaluates its effect on teaching and learning." Perhaps (god forbid) we could also review the effects of all the links to targets, KPIs and league tables on the quality of learning, as they certainly compromise the validity of the whole system. I will again point to the efforts of colleagues in Denmark, who seem to have understood this a whole lot better.

Marking free text

One of the frequent criticisms on e-Assessment is the perceived limits in item types that can be supported by technology. While there are long debates to be had about assessing higher order skills with constrained response item types, I don't think these debates are going to take away the prime concern: Free text items.

I must say that I have serious doubts about marking free text by computer. I don't know enough about the principles involved to say this with any sort of authority, but I am aware of the kind of heuristics used in automated essay marking for instance. These heuristics are often grammatical and rhetorical in nature, and have fairly little to do with the subject matter (although it must be said that many human markers have been shown to use similar heuristics). Nevertheless, interesting progress is being made in this area, and eventually I am sure that language processing will be commonplace.

One of the interesting project that I recently became aware of, is the OpenComment project, which is lead by Denise Whitelock at the Open University. The project is looking to use latent semantic analysis to analyse learners responses to open ended questions in history and philosophy. Another interesting fact is that the project is developing this as a question type in Moodle, and so it should be relatively easy for everyone to reap the benefits of this technology within their own learning environments.

Automated marking is by no means the only value of using technology in assessment. The OpenMentor project, again from the Open University, is a great example. OpenMentor compares a mark assigned to a piece of work to the amounts of positive and negative feedback given, and checks this for consistency. In this way it can help in the coaching process of new teachers. Given the importance of feedback, I think it's wonderful to have explicit standards and training in giving it.

The ABC (Assess By Computer) software has so far escaped my radar. I wasn't aware of it until queried by the Times Higher Education for the article they were doing. The software has a support role similar to OpenMentor, but this time the support is provided around the marking process. The software can highlight keywords, compare answers to model answers and more. All of this for the sole purpose of making it easier on the human marker, but also improve consistency between human markers. Especially the latter is very welcome I think, as marking open ended questions and assignments can sometimes be somewhat of a dark art.

I only just discovered that bits of the e-mail I sent to the reporter actually appear in the article. If I would have known that I probably would have paid a bit more attention to my grammar :S

High stake national assessments and ranking

It's been a long day full of many, many presentations. Fortunately the last presentation was actually one of the more interesting ones, and I did not have to fight off the embarrassment of falling asleep too hard. It was a presentation by Jakob Wandall from Skolestyrelsen.
about the new national computer based assessments in secondary education that have been introduced in Denmark.

While the technical side of this was interesting (they were using computer adaptive testing for instance), the most interesting bit of the talk had nothing to do with technology at all. It had to do with how the test was used, presented used and regulated.

In England, high stakes tests are a very big deal. The main reason is that they are always is that they are inevitable translated into rankings and funding consequences, leading to teachers and school becoming completely obsessed with assessments, drilling students until they are green in the face in the idle expectation it might raise the school a place or 2 in the oh so important regional league tables. It is this abomination that I think the Danish have elegantly addressed (apparently with the English system as the example of what they wanted to avoid at all costs, and understandably so!)

The publication of the results of these national benchmarks is strictly regulated. The national average is published and used for policy purposes, but no regional or individual result is public. Teachers can review all results of all their students, and even responses to individual questions, but are forbidden to communicate these results other then to the student and their parents (and this communication is not in the form of a grade, but of a textual report with feedback). Students have to be given their result by a qualified teacher that discusses the results and provides relevant feedback on the performance.

So it is impossible for a school, a local authority or the press, to rate and rank scores just on the numerical outcomes of a single test. It provides stakeholders on every level with the relevant information, without the detrimental effects of publication that we see in the US and UK. I think we've got a lot to learn from the Scandinavian approach to education