Monday, September 11, 2017

New jobs, new blog

Is digital a thing? As of a week or so ago, I have two new jobs:

Digital Learning Lead
Metrics lead

for the School of Biological Sciences. What all this means (and whether digital is actually a thing) we will see in the next few months. While we work it all out, we have a new blog which we invite you to follow:

Biological Sciences SoTL

As you know, this blog has been quiet for a long time. Whether that will continue or whether I will start to use this site for personal views and the new blog as a corporate organ ... time will tell.

Tuesday, August 08, 2017

Collaborative testing promotes higher order thinking

Graduates I'm not sure I completely understand why, but the English education system installs a fierce sense of competition in students. Although we attempt to make them aware that we use criterion- rather than norm-referenced assessment, we just can't seem to convince them that assessment is not a zero sum game. Consequently, group assessment, especially summative group assessment, generates fireworks. But it turns out that collaborative testing is a valuable exercise, assuming we're interested in higher order skills rather than just cramming their heads with facts?

The effects of collaborative testing on higher order thinking: Do the bright get brighter? Active Learning in Higher Education 04.08.2017 doi: 10.1177/1469787417723243
Collaborative testing has been shown to enhance student performance compared to individual testing. It is suggested that collaborative testing promotes higher order thinking, but research has yet to explore this assumption directly. The aim of this study was to explore the benefits of collaborative testing on overall performance, as well as performance on higher order thinking questions. It was hypothesised that, compared to individual test results, students would perform better overall and on higher order thinking questions under collaborative testing conditions. It was expected that these differences would be equal when comparing students of different academic abilities (i.e. ‘upper’, ‘middle’ and ‘lower’ performers). Undergraduate students completed an individual followed by a collaborative test as part of summative assessment. Analyses revealed that with the exception of upper performers, students performed better overall on the collaborative test. Additionally, regardless of their academic abilities, students performed better on the higher order thinking questions under collaborative conditions. This improvement was equal across different academic abilities, suggesting that collaborative testing promotes higher order thinking even when taking into account previous academic achievement. The acceptability and application of collaborative testing is discussed.

Wednesday, August 02, 2017

Making claims for assessment [Commentary]

Usain Bolt In a recent article Dylan Wiliam does what Dylan Wiliam does best – takes a sideways but highly practical look at assessment.

The starting point for his reflection is a recent article discussing models of assessment and learning (Baird, J., Andrich, D., Hopfenbeck, T. & Stobart, G. (2017) Assessment and learning: fields apart? Assessment in Education: Principles, Policy & Practice, 24:3, 317-350, DOI: 10.1080/0969594X.2017.1319337). This argues that learning is substantially driven by assessment and that theories of assessment and theories of learning have, for the most part, proceeded entirely separately. The paper argues that that if theories of assessment take into account theories of learning, assessments will somehow be more valid. This is the point that Wiliam takes issue with:

“There is no a priori reason that processes of development would be enhanced by attention to the processes by which the results of that development are assessed, nor that the processes of assessing the results of development would be improved by considering how that development took place. For example, if we consider the case of the 100 m at the Olympic Games, a theory of learning would provide insights into how people improve their sprinting ability, and would, as a result, help us improve the quality of sprinters, so that they record lower times for the 100 m at the Olympic Games. We could also look at ways of improving the accuracy of the measurement of the time taken by sprinters to run 100 m. However, the important point here is that these two processes are entirely separate. Improvements in the measurement of time do not help us improve the performance of athletes, and improvements in the performance of athletes do not contribute to measuring sprint times more accurately. The aetiology of a performance may be entirely irrelevant to the measurement of that performance.”

Wilam goes on providing good value:

“… it may be that the introduction of more authentic assessments were driven by a desire to make the assessment more closely aligned to the learning environment. But it is at least as likely that the changes were driven by a desire to extend the kinds of inferences that test outcomes would support – to address construct underrepresentation in the assessment design. In other words, the developments of assessment theory were driven by problems of assessment, not of learning. The assessment problems may have been raised by a concern for learning, but there is little evidence that that anything in learning theory prompted the changes. Indeed, the most convincing narrative is, in my view, that developments in each of these fields have been almost completely unaffected by developments in the other. Reasons for such lack of mutual engagement and influence are easy to propose. Academics work in silos, and the pressure to publish creates incentives to make incremental improvements within a narrowly defined field, not least because criteria of quality and rigour in different fields are often very different.”

“The real problem is that measures of student progress are inherently less reliable than status measures, because the former involve subtracting one number that is measured with error from another number that is measured with error (Cronbach & Furby, 1970). The problem is not that the theory is inadequate. The problem is that the theory gives answers that people don’t like.”

“At its heart, as Lee Cronbach pointed out almost 50 years ago, an assessment is a procedure for drawing inferences: ‘One validates, not a test, but an interpretation of data arising from a specified procedure’ (Cronbach, 1971, p. 447, emphasis in original). Any educational assessment is therefore a procedure for making inferences about learning.”

Dylan Wiliam (2017) Assessment and learning: some reflections. Assessment in Education: Principles, Policy & Practice, 24:3, 394-403, DOI: 10.1080/0969594X.2017.1318108

Monday, May 08, 2017

Statement bank feedback: worthwhile?

Feedback checklist After a long sleep, time to dust off the old blog. This paper (just published) is highly relevant to research I'm currently engaged in, so it's of interest to me. It should be of interest to you too, given that staff time is the major pressure on feedback and tricks such as statement banks are only going to grow in prominence. An interesting study, but it uses the language of "assessment literacy" which I don't buy into. My mental model of the "all or nothing" behaviour observed is a different one - student responses to assessment are yet another proxy of engagement, evidenced by the higher marks of the email responders. Apart from not reading their feedback or email, we don't know what the non-responders were doing. So if students aren't going to read it, let's save staff time by using statement banks.

Response of students to statement bank feedback: the impact of assessment literacy on performances in summative tasks.
Assessment & Evaluation in Higher Education 07.05.17, doi: 10.1080/02602938.2017.1324017

Efficiency gains arising from the use of electronic marking tools that allow tutors to select comments from a statement bank are well documented, but how students use this type of feedback remains under explored. Natural science students (N = 161) were emailed feedback reports on a spreadsheet assessment that included an invitation to reply placed at different positions. Outcomes suggest that students either read feedback completely, or not at all. Although mean marks for repliers (M = 75.5%, N = 39) and non-repliers (M = 57.2%, N = 68) were significantly different (p < .01), these two groups possessed equivalent attendance records and similar submission rates and performances in a contemporaneous formatively assessed laboratory report. Notably, average marks for a follow-up summative laboratory report, using the same assessment criteria as the formative task, were 10% higher for students who replied to the original invite. It is concluded that the repliers represent a group of assessment literate students, and that statement bank feedback can foster learning: a simple ‘fire’ analogy for feedback is advanced that advocates high-quality information on progress (fuel) and a curricular atmosphere conducive to learning (oxygen). However, only if students are assessment literate (ignition) will feedback illuminate.