Thursday, November 26, 2015

When The Money Runs Out

This paper asks a difficult question, and comes up with ... an answer. I don't think it is the answer, but it it an answer to a question which many of us are going to have to address soon.

Pull quote:
"Our students pay high fees, have high workloads and want face-to-face feedback from experienced tutors. There is a subtle problem here. Student-centred approaches involving self-regulation and active learning may promote deep learning and high generalisation of outcomes, but skills acquisition seems to require at least some degree of authoritative feedback. The question is how to provide the latter as efficiently as possible without diverting learners from the former."

What if best practice is too expensive? Feedback on oral presentations and efficient use of resources, (2015) Assessment & Evaluation in Higher Education, doi: 10.1080/02602938.2015.1109054

Thursday, November 12, 2015

Keynes: the return of the master

Keynes the return of the master I've been a Keynsian since before I had labels to put on these things, but I've been slow catching up with Robert Skidelsky's commentary on the current financial crisis.

Robert Skidelsky. (2010) Keynes: the return of the master is divided into three sections. The first is scene-setting history and biography. The second is likely to be rather heavy going for non-economists, but it was the third section that really grabbed me. Here, Skidelsky goes beyond analysis of the current crisis to propose solutions for our woes (and also has a pop at a few neo-Keynsians such as Stiglitz). But Skidelsky also goes way beyond economics with a discussion of religion, duty, ethics and post Utilitarianism, particularly G.E. Moore's influence on Keynes.

Skidelsky also has a shot at asking How much is enough?, suggesting that Keynes didn't quite hit that nail on the head. How Much is Enough? Money and the Good Life (2012) is on my C-word reading list now.

Monday, November 09, 2015

Chalk and Cheese

Contrast In a week when all was doom and gloom over #HEgreenpaper, something good happened. Paul Orsmond and Stephen Merry published a paper.

One day last week I was ranting at one of my project students about contrasts - as a piece of statistical jargon and as a vehicle to construct hypotheses. So let's have some contrasts. Orsmond and Merry have let the cat out of the bag - it's not all about teachers, it's about the students too. I misread one sentence of their paper and temporarily thought they were calling for "non-constructive alignment" - then I was disappointed that they hadn't. Anyhow, the contrast between the reductive approach of #HEgreenpaper and the constructive approach in this paper could not be greater. You've read one, now read the other:

Tutors’ assessment practices and students’ situated learning in higher education: chalk and cheese. Assessment & Evaluation in Higher Education, 28 Oct 2015. doi: 10.1080/02602938.2015.1103366
This article uses situated learning theory to consider current tutor assessment and feedback practices in relation to learning practices employed by students outside the overt curriculum. The case is made that an emphasis on constructive alignment and explicitly articulating assessment requirements within curricula may be misplaced. Outside of the overt curriculum students appear to be interdependent learners, participating in communities of practice and learning networks, where sense-making occurs through negotiation and there is identity development. Such negotiation may translate curriculum requirements articulated by tutors into unexpected meanings. Hence, tutors’ efforts might be better placed on developing students’ ability to self-assess and to effectively evaluate and negotiate information, rather than primarily on their own delivery of the curriculum content and feedback. Tutors cannot be fully effective if they fail to consider students’ learning outside the overt curriculum, and ways to facilitate such learning processes are suggested together with future research directions.

Friday, November 06, 2015

How to fix feedback

Checklist Yet another paper telling us how to (start to) fix the feedback problem. This one contains some very sensible recommendations which I have highlighted below. Those of you who been playing along for many years may feel that this manuscript is rather similar to this and this (not referenced, but perhaps that's understandable given that the HEA have binned all their publicly-funded open-access journals). And so we reinvent the wheel. Again. Anyway, here's how to fix feedback:

Clear Transferability - Programme-level assessment. Yes please. Can't see it happening.

Feedback On Draft Work - Yes please, it's feedback, not assessment. Lift and separate.

Directly Linked To Criteria - Rubrics. Hmm... maybe...

Wasted Opportunities - Separate feedback and assessment. It's simple isn't it? I'll say it again. Separate feedback and assessment. Want me to say it again? OK, separate feedback and assessment.

Making connections: technological interventions to support students in using, and tutors in creating, assessment feedback. (2015) Research in Learning Technology, 23: 27078 -
This paper explores the potential of technology to enhance the assessment and feedback process for both staff and students. The ‘Making Connections’ project aimed to better understand the connections that students make between the feedback that they receive and future assignments, and explored whether technology can help them in this activity. The project interviewed 10 tutors and 20 students, using a semi-structured approach. Data were analysed using a thematic approach, and the findings have identified a number of areas in which improvements could be made to the assessment and feedback process through the use of technology. The findings of the study cover each stage of the assessment process from the perspective of both staff and students. The findings are discussed in the context of current literature, and special attention is given to projects from the UK higher education sector intended to address the same issues.

Thursday, November 05, 2015

Let's be honest

Bloxham, S., den-Outer, B., Hudson, J., & Price, M. (2015) Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria. Assessment & Evaluation in Higher Education, 23 Mar 2015, 1-16
Unreliability in marking is well documented, yet we lack studies that have investigated assessors’ detailed use of assessment criteria. This project used a form of Kelly’s repertory grid method to examine the characteristics that 24 experienced UK assessors notice in distinguishing between students’ performance in four contrasting subject disciplines: that is their implicit assessment criteria. Variation in the choice, ranking and scoring of criteria was evident. Inspection of the individual construct scores in a sub-sample of academic historians revealed five factors in the use of criteria that contribute to marking inconsistency. The results imply that, whilst more effective and social marking processes that encourage sharing of standards in institutions and disciplinary communities may help align standards, assessment decisions at this level are so complex, intuitive and tacit that variability is inevitable. We conclude that universities should be more honest with themselves and with students, and actively help students to understand that application of assessment criteria is a complex judgement and there is rarely an incontestable interpretation of their meaning.

"Accepting the inevitability of grading variation means that we should review whether current efforts to moderate are addressing the sources of variation. This study does add some support to the comparison of grade distributions across markers to tackle differences in the range of marks awarded. However, the real issue is not about artificial manipulation of marks without reference to evidence. It is more that we should recognise the impossibility of a ‘right’ mark in the case of complex assignments, and avoid overextensive, detailed, internal or external moderation. Perhaps, a better approach is to recognise that a profile made up of multiple assessors’ judgements is a more accurate, and therefore fairer, way to determine the final degree outcome for an individual. Such a profile can identify the consistent patterns in students’ work and provide a fair representation of their performance, without disingenuously claiming that every single mark is ‘right’. It would significantly reduce the staff resource devoted to internal and external moderation, reserving detailed, dialogic moderation for the borderline cases where it has the power to make a difference. This is not to gainsay the importance of moderation which is aimed at developing shared disciplinary norms, as opposed to superficial procedures or the mechanical resolution of marks."

It's quite easy to criticize this paper - small scale study (n=24), no attempt at statistical analysis or validation. But there's still an inescapable feeling that as the stakes have escalated, HE is kidding itself about assessment practices.

Thursday, October 22, 2015

Peer-assessment in higher education

pier After my mini-rant yesterday, here's a new review article on peer-assessment in higher education. In my view the author is possibly too circumspect in his conclusions - if a student wrote this for me I'd tell them to get off the fence. However, he does us a service by pointing out that peer assessment isn't necessarily a magic bullet that will save staff time. Too darn right. Nevertheless, this is a very useful bibliography for those of us convinced that peer assessment has to be the future of higher education but are struggling to make it work.

Peer-assessment in higher education – twenty-first century practices, challenges and the way forward. Assessment & Evaluation in Higher Education 19 Oct 2015 doi: 10.1080/02602938.2015.1100711
Peer assessment in higher education has been studied for decades. Despite the substantial amount of research carried out, peer assessment has yet to make significant advances. This review identifies themes of recent research and highlights the challenges that have hampered its advance. Most of these challenges arise from the manual nature of peer assessment practices, which prove intractable as the number of students involved increases. Practitioners of the discipline are urged to forge affiliations with closely related fields and other disciplines, such as computer science, in order to overcome these challenges.

Wednesday, October 21, 2015

PeerMark Frustration


Frustration because it's so close to being a brilliant tool, apart from being mind-blowingly complex to set up (which means that few will use it).

Sadly, as reported, it is buggy as hell generating dozens of emails from students :-(