Pages

Friday, December 11, 2015

More Than a Literature Review: Final Year Projects for Bioscience Students

Final year projects are very much the capstone experience of the undergraduate bioscience experience. Yet many academics cling doggedly to the belief that they are training Mini-Mes and that projects are only an entry into PhD programmes. But alternative (non-laboratory) types of project remain very much second class options. This is because, in my personal opinion which does not represent that of my employer, activity is valued over thinking. And in the weird world we live in, scholarship does not count as activity. But the vast and increasing majority of our graduates will never don a lab coat or set foot in a laboratory again after they graduate. They will, however, be required to think. And to make judgements and have opinions. In other words, we live in hope that our graduates will live their lives in a scholarly way. The least we can do is to try to train them to do that.



Julia Lodge (2015) More Than a Literature Review: An Alternative Final Year Project for Bioscience Students. Education in Practice, Vol. 2 No. 1.
With increasing pressures on staff time and increased diversity in the student population it is important that universities explore different ways of providing final year projects. This case study describes a successful format developed in the School of Biosciences at the University of Birmingham. The project consists of a literature review followed by an in-depth critical analysis of five key papers and the subsequent development of a research proposal. In addition to written reports and an oral presentation students are asked to write both a lay and a technical abstract for their research proposal. This challenges the student to explain the importance and also the scientific approach to different audiences. The analysis of five key papers has been found to be an excellent tool to encourage a deep understanding and critical analysis of research papers and that the skills demonstrated in this section are distinct from those demonstrated in a traditional literature review. The research proposal allows students to develop many of the skills usually associated with a practical project including identification of gaps in current knowledge and developing a hypothesis. Overall this format allows students not choosing a lab or field based final year project to apply the skills and knowledge accumulated during their degree in a discipline related context and despite not having a practical element it remains true to its roots in experimental science.


Education in Practice Education in Practice - December 2015
https://intranet.birmingham.ac.uk/staff/teaching-academy/education-in-practice/eip-dec2015.aspx







Wednesday, December 02, 2015

Assessing Teamwork

Teamwork I'm in the middle of a big team work evaluation right now, so the title of this paper immediately grabbed my attention. The Team-Q tool described looks very good, although sadly it remains the case that the procedures involved are still too cumbersome for easy widespread adoption.

Oh, and then there's the whole business of student "satisfaction". Team assessment? Students hate it.




Assessing teamwork in undergraduate education: a measurement tool to evaluate individual teamwork skills. (2015): Assessment & Evaluation in Higher Education, doi: 10.1080/02602938.2015.1116497
Effective teamwork skills are essential for success in an increasingly team-based workplace. However, research suggests that there is often confusion concerning how teamwork is measured and assessed, making it difficult to develop these skills in undergraduate curricula. The goal of the present study was to develop a sustainable tool for assessing individual teamwork skills, with the intention of refining and measuring these skills over time. The TeamUp rubric was selected as the preliminary standardised measure of teamwork and tested in a second year undergraduate course (Phase One). Although the tool displayed acceptable psychometric properties, there was concern that it was too lengthy, compromising student completion. This prompted refinement and modification leading to the development of the Team-Q, which was again tested in the same undergraduate course (Phase Two). The new tool had high internal consistency, as well as conceptual similarity to other measures of teamwork. Estimates of inter-rater reliability were within a satisfactory range, although it was determined that logistical issues limited the feasibility of external evaluations. Preliminary evidence suggests that teamwork skills improve over time when taught and assessed, providing support for the continued application of the Team-Q as a tool for developing teamwork skills in undergraduate education.




Tuesday, December 01, 2015

Audio feedback - let's not oversell it

Audio It's nice to read a paper which doesn't make wildly unsubstantiated claims which it then fails to justify.


Investigating expectations and experiences of audio and written assignment feedback in first-year undergraduate students. Teaching in Higher Education, 27 Nov 2015. doi: 10.1080/13562517.2015.1115969
Previous research suggests that audio feedback may be an important mechanism for facilitating effective and timely assignment feedback. The present study examined expectations and experiences of audio and written feedback provided through turnitin for iPad® from students within the same cohort and assignment. The results showed that although initially sceptical of audio compared to written feedback, there were no significant differences in students' experiences of audio and written feedback. Students' performance on the assignment was not associated with their experiences of audio feedback but first-class performing students ( > 70%) had more positive experiences of written feedback than those who received an upper second-class grade (60–69%). In general, the results imply that audio feedback provided through turnitin for iPad® is a viable alternative to written feedback. The findings are discussed in relation to past research findings.





Monday, November 30, 2015

Assessment and learning without grades

Can you eliminate grades and stress from higher education and make university about learning rather than grades? Yes you can. Can you persuade a government which wants to reduce higher education to a meaningless league table of misleading metrics that it's a good idea?



Assessment and learning without grades? Motivations and concerns with implementing gradeless learning in higher education. (2015) Assessment & Evaluation in Higher Education, doi: 10.1080/02602938.2015.1114584
The relationship between assessment and learning in higher education often comes down to a single thing: a grade. Despite widespread criticism of grades as inexact tools, whose overemphasis undermines student learning and negatively affects student well-being, they continue to be the norm in the assessment of student learning. This paper analyses an alternate form of assessment: so-called ‘gradeless learning’. This study theoretically and geographically contextualises the recent implementation of a gradeless learning policy at a large public university in Asia, and presents findings from a student opinion survey about the policy. The paper shows that respondents overwhelmingly understand and often agree with the central claims of gradeless learning, including its potential to ease students into college life, allow them to make more daring choices in their studies and even develop as lifelong learners. However, the aim of relieving stress among one group of students has increased stress for others. The study explains the circumstances that create this divergence in student stress levels, which are both locally specific and common to all gradeless systems. The paper concludes by discussing the effectiveness of the gradeless system in achieving its aims and suggesting future research avenues.



Thursday, November 26, 2015

When The Money Runs Out

This paper asks a difficult question, and comes up with ... an answer. I don't think it is the answer, but it it an answer to a question which many of us are going to have to address soon.

Pull quote:
"Our students pay high fees, have high workloads and want face-to-face feedback from experienced tutors. There is a subtle problem here. Student-centred approaches involving self-regulation and active learning may promote deep learning and high generalisation of outcomes, but skills acquisition seems to require at least some degree of authoritative feedback. The question is how to provide the latter as efficiently as possible without diverting learners from the former."


What if best practice is too expensive? Feedback on oral presentations and efficient use of resources, (2015) Assessment & Evaluation in Higher Education, doi: 10.1080/02602938.2015.1109054





Thursday, November 12, 2015

Keynes: the return of the master

Keynes the return of the master I've been a Keynsian since before I had labels to put on these things, but I've been slow catching up with Robert Skidelsky's commentary on the current financial crisis.

Robert Skidelsky. (2010) Keynes: the return of the master is divided into three sections. The first is scene-setting history and biography. The second is likely to be rather heavy going for non-economists, but it was the third section that really grabbed me. Here, Skidelsky goes beyond analysis of the current crisis to propose solutions for our woes (and also has a pop at a few neo-Keynsians such as Stiglitz). But Skidelsky also goes way beyond economics with a discussion of religion, duty, ethics and post Utilitarianism, particularly G.E. Moore's influence on Keynes.

Skidelsky also has a shot at asking How much is enough?, suggesting that Keynes didn't quite hit that nail on the head. How Much is Enough? Money and the Good Life (2012) is on my C-word reading list now.





Monday, November 09, 2015

Chalk and Cheese

Contrast In a week when all was doom and gloom over #HEgreenpaper, something good happened. Paul Orsmond and Stephen Merry published a paper.

One day last week I was ranting at one of my project students about contrasts - as a piece of statistical jargon and as a vehicle to construct hypotheses. So let's have some contrasts. Orsmond and Merry have let the cat out of the bag - it's not all about teachers, it's about the students too. I misread one sentence of their paper and temporarily thought they were calling for "non-constructive alignment" - then I was disappointed that they hadn't. Anyhow, the contrast between the reductive approach of #HEgreenpaper and the constructive approach in this paper could not be greater. You've read one, now read the other:


Tutors’ assessment practices and students’ situated learning in higher education: chalk and cheese. Assessment & Evaluation in Higher Education, 28 Oct 2015. doi: 10.1080/02602938.2015.1103366
This article uses situated learning theory to consider current tutor assessment and feedback practices in relation to learning practices employed by students outside the overt curriculum. The case is made that an emphasis on constructive alignment and explicitly articulating assessment requirements within curricula may be misplaced. Outside of the overt curriculum students appear to be interdependent learners, participating in communities of practice and learning networks, where sense-making occurs through negotiation and there is identity development. Such negotiation may translate curriculum requirements articulated by tutors into unexpected meanings. Hence, tutors’ efforts might be better placed on developing students’ ability to self-assess and to effectively evaluate and negotiate information, rather than primarily on their own delivery of the curriculum content and feedback. Tutors cannot be fully effective if they fail to consider students’ learning outside the overt curriculum, and ways to facilitate such learning processes are suggested together with future research directions.




Friday, November 06, 2015

How to fix feedback

Checklist Yet another paper telling us how to (start to) fix the feedback problem. This one contains some very sensible recommendations which I have highlighted below. Those of you who been playing along for many years may feel that this manuscript is rather similar to this and this (not referenced, but perhaps that's understandable given that the HEA have binned all their publicly-funded open-access journals). And so we reinvent the wheel. Again. Anyway, here's how to fix feedback:

Clear Transferability - Programme-level assessment. Yes please. Can't see it happening.

Feedback On Draft Work - Yes please, it's feedback, not assessment. Lift and separate.

Directly Linked To Criteria - Rubrics. Hmm... maybe...

Wasted Opportunities - Separate feedback and assessment. It's simple isn't it? I'll say it again. Separate feedback and assessment. Want me to say it again? OK, separate feedback and assessment.



Making connections: technological interventions to support students in using, and tutors in creating, assessment feedback. (2015) Research in Learning Technology, 23: 27078 - http://dx.doi.org/10.3402/rlt.v23.27078
This paper explores the potential of technology to enhance the assessment and feedback process for both staff and students. The ‘Making Connections’ project aimed to better understand the connections that students make between the feedback that they receive and future assignments, and explored whether technology can help them in this activity. The project interviewed 10 tutors and 20 students, using a semi-structured approach. Data were analysed using a thematic approach, and the findings have identified a number of areas in which improvements could be made to the assessment and feedback process through the use of technology. The findings of the study cover each stage of the assessment process from the perspective of both staff and students. The findings are discussed in the context of current literature, and special attention is given to projects from the UK higher education sector intended to address the same issues.






Thursday, November 05, 2015

Let's be honest

Bloxham, S., den-Outer, B., Hudson, J., & Price, M. (2015) Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria. Assessment & Evaluation in Higher Education, 23 Mar 2015, 1-16
Unreliability in marking is well documented, yet we lack studies that have investigated assessors’ detailed use of assessment criteria. This project used a form of Kelly’s repertory grid method to examine the characteristics that 24 experienced UK assessors notice in distinguishing between students’ performance in four contrasting subject disciplines: that is their implicit assessment criteria. Variation in the choice, ranking and scoring of criteria was evident. Inspection of the individual construct scores in a sub-sample of academic historians revealed five factors in the use of criteria that contribute to marking inconsistency. The results imply that, whilst more effective and social marking processes that encourage sharing of standards in institutions and disciplinary communities may help align standards, assessment decisions at this level are so complex, intuitive and tacit that variability is inevitable. We conclude that universities should be more honest with themselves and with students, and actively help students to understand that application of assessment criteria is a complex judgement and there is rarely an incontestable interpretation of their meaning.


"Accepting the inevitability of grading variation means that we should review whether current efforts to moderate are addressing the sources of variation. This study does add some support to the comparison of grade distributions across markers to tackle differences in the range of marks awarded. However, the real issue is not about artificial manipulation of marks without reference to evidence. It is more that we should recognise the impossibility of a ‘right’ mark in the case of complex assignments, and avoid overextensive, detailed, internal or external moderation. Perhaps, a better approach is to recognise that a profile made up of multiple assessors’ judgements is a more accurate, and therefore fairer, way to determine the final degree outcome for an individual. Such a profile can identify the consistent patterns in students’ work and provide a fair representation of their performance, without disingenuously claiming that every single mark is ‘right’. It would significantly reduce the staff resource devoted to internal and external moderation, reserving detailed, dialogic moderation for the borderline cases where it has the power to make a difference. This is not to gainsay the importance of moderation which is aimed at developing shared disciplinary norms, as opposed to superficial procedures or the mechanical resolution of marks."



It's quite easy to criticize this paper - small scale study (n=24), no attempt at statistical analysis or validation. But there's still an inescapable feeling that as the stakes have escalated, HE is kidding itself about assessment practices.



Thursday, October 22, 2015

Peer-assessment in higher education

pier After my mini-rant yesterday, here's a new review article on peer-assessment in higher education. In my view the author is possibly too circumspect in his conclusions - if a student wrote this for me I'd tell them to get off the fence. However, he does us a service by pointing out that peer assessment isn't necessarily a magic bullet that will save staff time. Too darn right. Nevertheless, this is a very useful bibliography for those of us convinced that peer assessment has to be the future of higher education but are struggling to make it work.


Peer-assessment in higher education – twenty-first century practices, challenges and the way forward. Assessment & Evaluation in Higher Education 19 Oct 2015 doi: 10.1080/02602938.2015.1100711
Peer assessment in higher education has been studied for decades. Despite the substantial amount of research carried out, peer assessment has yet to make significant advances. This review identifies themes of recent research and highlights the challenges that have hampered its advance. Most of these challenges arise from the manual nature of peer assessment practices, which prove intractable as the number of students involved increases. Practitioners of the discipline are urged to forge affiliations with closely related fields and other disciplines, such as computer science, in order to overcome these challenges.


Wednesday, October 21, 2015

PeerMark Frustration

PeerMark

Frustration because it's so close to being a brilliant tool, apart from being mind-blowingly complex to set up (which means that few will use it).

Sadly, as reported, it is buggy as hell generating dozens of emails from students :-(



Tuesday, September 29, 2015

How to write an essay

writing
Problem 1: Students write descriptive essays which do not demonstrate critical thinking.
Solution: Title as question.

Problem 2: Massive over reliance on essays as an assessment format in higher education.
Solution: ???



Henri D, Morrell L. and Scott G. Ask a clearer question, get a better answer. F1000Research 2015, 4: 901 doi: 10.12688/f1000research.7066.1
Many undergraduate students struggle to engage with higher order skills such as evaluation and synthesis in written assignments, either because they do not understand that these are the aim of written assessment or because these critical thinking skills require more effort than writing a descriptive essay. Here, we report that students who attended a freely available workshop, in which they were coached to pose a question in the title of their assignment and then use their essay to answer that question, obtained higher marks for their essay than those who did not attend. We demonstrate that this is not a result of latent academic ability amongst students who chose to attend our workshops and suggest this increase in marks was a result of greater engagement with ‘critical thinking’ skills, which are essential for upper 2:1 and 1st class grades. The tutoring method we used holds two particular advantages: First, we allow students to pick their own topics of interest, which increases ownership of learning, which is associated with motivation and engagement in ‘difficult’ tasks. Second, this method integrates the development of ‘inquisitiveness’ and critical thinking into subject specific learning, which is thought to be more productive than trying to develop these skills in isolation.


I'm not quite sure how peer review works on the F1000 education channel. I have been asked to peer review articles in the past and have done so, but I'm not sure how open the process is. So here's my open peer review of this paper.

This is an interesting and potentially valuable study of a method to improve the quality of student writing. The sample size is relatively small and the major weaknesses are pointed out by the authors in the Discussion:
"For the purpose of this study we assumed that students who posed a question in the title of their essay had attended the workshop and understood the underlying concepts of the workshop, and this has been used as the independent factor in our analysis. We acknowledge that this lack of certainty in the allocation of students to the did/did not attend category does need to be borne in mind when interpreting our results. Another possible confounding factor is that voluntary workshop attendance may be skewed towards individuals who are more engaged or motivated with the module; and these individuals are more likely to obtain higher grades because of this higher engagement with the module content"
To counteract these factors, the authors should cite an effect size to validate the p-values quoted in the results.




Thursday, September 17, 2015

The 3 P's of Feedback - again

Since I wrote about the 3 P's of good feedback a couple of weeks ago I've had several conversations with colleagues which seem to need some amplification.


Prompt
Not much to add to what I said previously - I suspect that for students prompt means 21 seconds or 21 minutes after submission, not 21 days - except that electronic submission of work has changed the game and is probably responsible for some of the dissatisfaction we face. When students had to trek across campus to a printer, print their work, then track back to a departmental office to stand in a queue to hand it in, 21 days seemed like a fair interval to wait for a response. But that's not how it is any more. Clicking a Submit button online generates an expectation of a much quicker interaction. For anything other than simple automatically marked MCQs, this just isn't going to happen in the interval (seconds or minutes) that students want. I don't know what we do about this, other than to get better at explicit telling students what to expect "After submission you will receive your marks and feedback within 24 hours / one week / whenever."


Personal
Group feedback does not satisfy the feedback beast. However, it could buy us some time, if it's sufficiently prompt. Group feedback on essays (good points and common errors) within 24 hours with detailed personal feedback to follow in 7 days? Feedback on exams is another huge problem that group feedback won't solve. The examination process used to take a few days, or maybe even hours for a small class. Now it takes weeks. This is partly because of student numbers, but mostly because of the huge backend bureaucracy we have built. Students don't understand this, it further distances us from them. And yet I can't see the bureaucracy going away, so the only inadequate solution I have is to better communicate with students why it takes us so long.


Positive
This is the idea that has generated most heat in discussions. "If we don't tell students what's wrong with their work how can they improve?". I was raised in an academic era when my work was bluntly, sometime brutally, criticized (when needed). Crap was frequently scrawled in the margins, and it's hard for me to break out of that mould. It was tough at the time, but you could argue that it made me resilient (or maybe I was already resilient, which enabled me to survive). Either way, it doesn't work any more. If feedback is not positive, non-resilient students switch off and disengage. But that doesn't mean you can't tell students what's wrong with their work. Rather than writing "You didn't include any diagrams", say "If you include some diagrams in your next essay you will get better marks". One consequence of this is that the classic sh*t sandwich feedback formula is now beyond the pale. We're in the unbearably upbeat Have A Nice Day era. We need to adapt. Feedback is not peer review. Feedback is not performance monitoring, it is mentoring. Once again we have fallen down the crack between feedback and assessment.





Friday, September 11, 2015

Student use of Wikipedia as an academic resource

Wikipedia I am encouraged rather than discouraged by:


Selwyn, N., and Gorard, S. (2015) Students' use of Wikipedia as an academic resource - patterns of use and perceptions of usefulness. The Internet and Higher Education. 5 September 2015 doi: 10.1016/j.iheduc.2015.08.004
Wikipedia is now an established information source in contemporary society. With initial fears over its detrimental influence on scholarship and study habits now subsiding, this paper investigates what part Wikipedia plays in the academic lives of undergraduate students. The paper draws upon survey data gathered from students across two universities in Australia (n=1658), alongside follow-up group interview data from a subsample of 35 students. Analysis of this data suggests that Wikipedia is now an embedded feature of most students’ study, although to a lesser extent than other online information sources such as YouTube and Facebook. For the most part, Wikipedia was described as an introductory and/or supplementary source of information – providing initial orientation and occasional clarification on study topics. While 87.5 per cent of students reported using Wikipedia, it was seen to be of limited usefulness when compared with university-provided library resources, e-books, learning management systems, lecture recordings and academic literature databases. These findings were notably patterned in terms of students’ gender, year of study, first language spoken and subject of study.
  • Draws on survey data examining 1658 undergraduate students’ uses of digital technologies for academic purposes.
  • 87.5 per cent of students report having used Wikipedia to find information for their academic work, with 24.0 per cent of these considering Wikipedia to have been ‘very useful’.
  • Use and perceived usefulness of Wikipedia is most prevalent amongst students who are male, in advanced years of study, from non-English speaking households, and those studying engineering, science and medicine subjects.
  • Rather than constituting a primary source of information, students report Wikipedia mainly playing introductory or clarificatory roles in their information gathering and research.

See: Citing Wikipedia in Academic Work




Friday, September 04, 2015

Formative and shared assessment in higher education

Back to school "The aim of this article is to review the use of formative and shared assessment within higher education."

  • Setting clear learning goals that students can achieve
  • Providing students with feedback to guide them in their learning
  • Involving students in the learning process, self-evaluation and assessment
  • Promoting feedback as a process of dialogue
  • Finding a balance between the ideal time spent in formative assessment processes and conditions in which the course is developed


Formative and shared assessment in higher education. Lessons learned and challenges for the future. Assessment & Evaluation in Higher Education 03 Sep 2015 doi: 10.1080/02602938.2015.1083535
Formative assessment is the process by which teachers provide information to students during the learning process to modify their understanding and self-regulation. An important process within this is shared assessment, which refers to student involvement in the assessment and learning practice, a process of dialogue and collaboration between teacher and students aimed at improving the learning process, both individually and collectively. The purpose of this paper is to review the current state of affairs in depth. This paper therefore highlights, on the one hand, the lessons learned through research and development in higher education (i.e. providing clear learning goals and feedback, guiding learning, involving students in learning and assessment, promoting feedback as a process of dialogue, and making processes viable). On the other hand, these lessons also suggest some challenges and difficulties that must be addressed in the future in order to further improve formative and shared assessment in higher education. These include the need for more research on its effects, further conceptual clarification, the intersubjectivity of the process, recognition of the divergent processes and ethical principles, students’ involvement not only in assessment but also in determining academic grades, and broadening learning goals and objectives in FA & SA.



Wednesday, August 26, 2015

Facebook Addiction

Bergen Facebook Addiction Scale

Andreassen, C.S., Torsheim, T., Brunborg, G.S., & Pallesen, S. (2012) Development of a Facebook addiction scale 1, 2. Psychological Reports, 110(2), 501-517
The Bergen Facebook Addiction Scale (BFAS), initially a pool of 18 items, three reflecting each of the six core elements of addiction (salience, mood modification, tolerance, withdrawal, conflict, and relapse), was constructed and administered to 423 students together with several other standardized self-report scales (Addictive Tendencies Scale, Online Sociability Scale, Facebook Attitude Scale, NEO–FFI, BIS/BAS scales, and Sleep questions). That item within each of the six addiction elements with the highest corrected item-total correlation was retained in the final scale. The factor structure of the scale was good (RMSEA = .046, CFI = .99) and coefficient alpha was .83. The 3-week test-retest reliability coefficient was .82. The scores converged with scores for other scales of Facebook activity. Also, they were positively related to Neuroticism and Extraversion, and negatively related to Conscientiousness. High scores on the new scale were associated with delayed bedtimes and rising times.

Citations




Tuesday, August 25, 2015

The 3 P's of Good Feedback

Butterfly Yet another paper about feedback in higher education - because it's still one of the major problems.

This fairly low-power study uses a budgeting methodology to ask what students value, in other words gives them a notional budget and ask them how they would spend it.
For "Lecturer qualities", Good feedback comes top, Interactive lecturing style bottom. So all those years of being told to be interactive in lectures don't mean much - students want your boring PowerPoints (and to know what's in the exam).
For "Feedback information" Highlights the skills I need to improve for future assignments is top and Corrects grammatical errors is bottom.

For me however the most striking message from this paper is an almost throwaway comment in the Introduction on what students want from feedback:
  • Prompt - fair enough, although I suspect that for students prompt means 21 seconds or 21 minutes after submission, not 21 days.
  • Personal - group feedback for that class of over 300 is a stopgap which really isn't going to satisfy demand.
  • Positive - it doesn't matter if they can't write (in spite of what employers say), you can only engage them if you give them good news quickly.


Winstone, N.E., Nash, R.A., Rowntree, J., & Menezes, R. (2015) What do students want most from written feedback information? Distinguishing necessities from luxuries using a budgeting methodology. Assessment & Evaluation in Higher Education, 20 Aug 2015 doi: 10.1080/02602938.2015.1075956
Feedback is a key concern for higher education practitioners, yet there is little evidence concerning the aspects of assessment feedback information that higher education students prioritise when their lecturers' time and resources are stretched. One recent study found that, in such circumstances, students actually perceive feedback information itself as a luxury rather than a necessity. We first re-examined that finding by asking undergraduates to "purchase" characteristics to create the ideal lecturer, using budgets of differing sizes to distinguish necessities from luxuries. Contrary to the earlier research, students in fact considered good feedback information the single biggest necessity for lecturers to demonstrate. In a second study, we used the same method to examine the characteristics of feedback information that students value most. Here, the most important perceived necessity was guidance on improvement of skills. In both studies, students? priorities were influenced by their individual approaches to learning. These findings permit a more pragmatic approach to building student satisfaction in spite of growing expectations and demands.




Friday, August 14, 2015

Assessment and feedback - what do students want?

Ray os sunshine We have moved to online marking (mostly via Turnitin) substantively over the last 18 months. In reality, it's probably the only way we could cope with numbers of students we have. But what do students think of this change? We don't know (because we haven't asked them, although they have not complained). Has this change helped with the NSS feedback question? (No.) This new paper addresses some of these questions and comes up with some interesting findings.

Individual students like or dislike online marking - individual preferences are negatively correlated. That means that as a population, students are broadly neutral, which has been our experience.

On the question of feedback, the findings are more interesting. Students who like online marking tend to view it as a gateway to staff contact - the start of a conversation. This is problematic because online assessment is primarily seen by staff as a file and forget exercise. So even with the students who are pro-online marking, we are not meeting their expectations. But most importantly of all - STUDENTS HATE NEGATIVE FEEDBACK ... which explains the NSS results.
"We suggest that markers should consider developing a small bank of brief but positive comments (for example “nicely written” “good argument” ) that can be readily added to the assignment in the place of the ticks that might have been given on a traditionally submitted assignment. Appropriate positive comments specific to particular sections of the assignment could then easily be added to a pdf (through annotation), word document (in a comment box), or included in the suite of QuickMarks used in submission services such as Turnitin. These recommendations notwithstanding, we also advocate that university budget centres acknowledge that although online marking has many benefits, relative to offline marking, more time will be needed by markers if students are to receive appropriate positive feedback on their work, and for the benefits of online assignments to be fully realised."


Assignments 2.0: The Role of Social Presence and Computer Attitudes in Student Preferences for Online versus Offline Marking. The Internet and Higher Education, 8 August 2015. doi: 10.1016/j.iheduc.2015.08.002
This study provided the first empirical and direct comparison of preferences for online versus offline assignment marking in higher education. University students (N= 140) reported their attitudes towards assignment marking and feedback both online and offline, perceptions of social presence in each modality, and attitudes towards computers. The students also ranked their preferences for receiving feedback in terms of three binary characteristics: modality (online or offline), valence (positive or negative), and scope of feedback (general or specific). Although attitudes towards online and offline marking did not significantly differ, positive attitudes toward one modality were strongly correlated with negative attitudes toward the other modality. Greater perceptions of social presence within a modality were associated with more positive attitudes towards that modality. Binary characteristics were roughly equally weighted. Findings suggest that the online feedback modality will most effectively maximise student engagement if online assignment marking and feedback tools facilitate perceptions of social presence.





Monday, August 03, 2015

Hello World

Hello World
As of today I am no longer a member of the Department of Biology (which doesn't exist any more), I am now a member of the Department of Neuroscience, Psychology and Behaviour.

What does that mean? Well I am still a member of the School of Biological Sciences and of the College of Medicine, Biological Sciences and Psychology, so in the short term, it doesn't mean as much as you might think. In the medium term, we are under new management, and in the long term ... well, what does anyone know of the long term?

But as of today, Hello World.







Tuesday, July 28, 2015

#Google+Pivot - what went wrong?

Google+ Google introduced yet another social network that no-one (except Google) really needed.

But Google+ was too complicated rather than following the simple design that won Google its prominence.

Google forgot how to be Google.

Sic transit gloria mundi.

Simples.


Open education: the need for a critical approach

Open access One agrees with the following:
"Crucially, the field has lacked coherent definitions of ‘open’, and too often tended towards optimism, advocacy, and conviction, rather than a critical understanding of what openness might mean for education. Moreover, it is the vagaries of the term itself that have allowed it to be attached to other ideas so readily: to notions of self-directed learning and cohesive community interaction; and to technology and the presumed capacities of the digital networks that enable educational activity to take place. In these ways, ‘open’ has too often accounted for the assumed ease with which educational hierarchies can be horizontalised, and economic and geographic barriers can be dissolved. But more than this, openness has too often assumed that institutional structures, financial constraints, and distance are the only issues preventing the instinctive and effortless uptake of self-directed learning. It is precisely in this way that an uncritical championing of openness fails to adequately analyse educational closures.

Many approaches to open education have been guided by the assumption that students fall into a universal category of rational, self-directing, and highly motivated individuals. Much less common is the acknowledgement that openness reconfigures or maintains particular notions of learning, teaching, and human being; that it is involved in the production of our contemporary understanding of ourselves as educated and educating beings. Part of the appeal of openness has derived from its association with a broader restructuring of education around the idea of the ‘learner’. This ‘learnification’ (Biesta 2010) has tended to assume autonomous students, whose independent activity requires educational opportunity with instantly and universally accessible material, anytime admittance, and teachers who merely ‘facilitate’ the process. The assumption is that ‘we’ are naturally open, and for all this time it has been the institution that has disciplined us into being closed. However, this idea of openness relies too heavily on the logic of self-direction, and fails to engage with the de-emphasis of teacher contact, problematic forms of student isolation, the appropriation of academic labour, and a neglect of the social and political dimensions of an education that surfaces in globalised classrooms. Such openness is only a solution for the imagined autonomous subject, and is only imaginable where education is divorced from the complexities of culture, sociality, and the power of the political. Furthermore, the restructuring of the role of the teacher and the implicit de-professionalisation of teaching is overlooked in this drive for openness. The open education movement can at times seem worryingly amenable to forces of neoliberalism within the university."


Open education: the need for a critical approach. (2015) Learning, Media and Technology 40(3) doi: 10.1080/17439884.2015.1065272



Tuesday, July 21, 2015

Returning to the problem of information literacy

Information literacy Since I started running "key skills" modules umptynine years ago, delivery of information literacy skills has been a constant concern. Leaving aside arcane discussions of what information literacy actually is, the realities of very large student numbers often forced me down the MCQ route, both because "If it's not (summatively) assessed it doesn't exist" and because of the impossible logistics of more appropriate alternatives.

Returning this week once again to the problem of information literacy delivery for the coming year, I turned to the research literature. The last time I did that I found it unhelpful because publications were dominated by arcane (unkindly: airy-fairy) discussion of the nature of academic literacies. This time around, by focusing my searches onto "information literacy" AND "science" (a spot of Boolean karma there), I came up with a more satisfactory outcome. Sifting of the literature in this restricted area now gives a more clear cut consensus on the best and possibly only solution: a pragmatic approach involving learning by doing - an authentic assessment apprenticeship model. Which unfortunately causes rather than solves a problem.

Numbers - that's the problem. If I had the luxury of delivering this content for half a dozen students on a single degree course, I'd have them write essays recursively until I knew they'd got it. Not viable with 400 students. So my alternative is ... peer assessment, either using the Blackboard Self and Peer Assessment tool or the Turnitin PeerMark system (comments on these welcome please). Which is fine, except it leads me into a discussion about formative versus summative assessment. Have students peer assess and only force completion by completion, or going down the hairy (and possibly unacceptable) route of having students summatively peer assess?


Information literacy assignment
1000 word Microsoft Word report on assigned topic:
  1. List and justify which keywords you used for your search (2 marks)
  2. List and justify which synonyms you used for your search (2 marks)
  3. List and justify which wildcards you used for your search (2 marks)
  4. List and justify which Boolean operators you used for your search (2 marks)
  5. Display search results (screenshots) from each of the following databases: PubMed, Google Scholar, WoK. (3 marks)
  6. Annotated Bibliography - list your choice of the 10 most relevant and important papers for your assigned topic - explaining why you chose each one. (10 marks)


I would really welcome your comments and insight on this.


Thompson, L, and Blankinship, L.A. (2015). Teaching Information Literacy Skills to Sophomore-Level Biology Majors. Journal of Microbiology & Biology Education, 16(1): 29. doi: 10.1128/jmbe.v16i1.818



Friday, July 17, 2015

So remind me again, what is the purpose of a university education?

Defensive position There's an interesting new paper in Assessment & Evaluation in Higher Education by Royce Sadler touching on ideas of authentic assessment, from which I have cheekily abstracted and editorialised what I consider to be the most important bits. Food for thought.

Three in-course assessment reforms to improve higher education learning outcomes. Assessment & Evaluation in Higher Education 17 Jul 2015 doi: 10.1080/02602938.2015.1064858


Reform 1: Assessment task design and specifications - ffs, let's bin "the essay".
"Creating demanding assessment tasks from scratch is hard work if the tasks are to tap into higher order operations on ideas and information. ... Potential sources also include real-life problems in the relevant field."

Reform 2: Grading at the ‘Pass’ level - "markers constantly need to make sound judgments about the quality of work in order to infer underlying competence or capability. Not always easy with real-life problems."

Reform 3: Redesigning course assessment plans
"Accumulation of marks - The common arguments for accumulation are essentially instrumentalist. The purpose is not so much to help learners attain adequate levels of complex knowledge and skills by the end of a course, as to keep them working and provide multiple opportunities for feedback. ... However, notwithstanding its superficial appeal, accumulation actually diverts attention from the goal of achieving a satisfactory level by course end.
Formative assessment - Given a set of course objectives, formative assessment is commonly viewed narrowly as giving students assessment tasks and then feedback so that they can improve. Despite all the effort typically invested in creating better and better feedback, it too often makes practically no difference ... the principal reason is that feedback is basically about telling students – the transmission model of teaching transposed into an assessment setting. The alternative is to offer students formative assessment opportunities that provide authentic evaluative experience of the type they need in order to become better able to recognise, monitor and control the quality of works they themselves are to produce."

Re-inventing end-of-course summative assessment - exam essays? Forget it.

Implications for students
Goal setting - are we locking them into league table mentality even before they graduate?
"Student sense of agency - students need to see and appreciate the purpose to be served, experience success in moving towards its attainment, and be motivated, with grit and determination, to follow through to completion." And not play point scoring games.

Inhibitors of change - well, we all know what they are.
"In recent decades, the focus for evaluating teaching quality has been heavily weighted towards inputs (student entry levels, participation rates, facilities, resources and support services) and a select group of outcomes (degree completions, employability, starting salaries and student satisfaction, experience or engagement). Conspicuously absent is anything to do with actual academic achievement in courses. This has allowed a number of sub-optimal assessment practices to become normalised into assessment cultures. One of the consequences is that too many students have been able to graduate without the capabilities expected of graduates, yet this is not necessarily apparent from their transcripts.
The focus in this article is on student outcomes rather than inputs, with particular emphasis on the higher order capabilities of students. Many students fail to master these, yet they gain credit in course after course and eventually graduate. Directly addressing the deficient aspects of assessment culture and practice could radically alter this state of affairs, but it would require a transformation in thinking and practice on the part of many academics. The ultimate aim is to ensure that all students accept a significant proportion of the responsibility for achieving adequate levels of higher order outcomes. Bluntly put, no student would be awarded a pass in a course without being able to demonstrate these levels. For some students, this would necessitate a major change in their priorities. For academics, both their assessment practices and the nature of the student–teacher relationship would change.
Undoubtedly, determination to pursue this end would have significant washback effects on teaching, learning, and course and programme objectives, but that is intended. The likelihood of success depends on finding a rational, ethical and affordable way to do it. This may require re-engineering some parts of the transition path, creating other parts from scratch, and reworking priorities, policies and practices to a considerable extent. In particular, it would entail rebalancing institutional resource allocations in order to cater for student cohorts that have become much more diversified. Except for aims geared narrowly to economic and employment considerations, this goal is broadly consistent with older and many recent statements of the real purposes of higher education."



Thursday, July 09, 2015

Online technology for teaching and learning - gains and losses

Small worlds This commentary describes recent developments in the use of online technologies, in particular social media and mobile devices, for teaching and learning and considers what has been gained and lost.


Online technology for teaching and learning - gains and losses. Alan Cann FEMS Microbiology Letters (16 June 2015), fnv099, doi: 10.1093/femsle/fnv099


Although Marc Prensky's 2001 description of "digital natives" (Prensky, 2001) - those who were "born digital" - is now largely disowned (including by Prensky himself), it has inevitably come to shape thinking about educational technology. It is undeniable that the Internet has changed most aspects of our lives in some way, yet there is scant evidence that the online world has fundamentally altered how we think or learn. Teenagers now hang out with friends on Facebook rather than in parks or on street corners, but the interactions are much the same as they ever were. In an introduction to a new F1000Research channel, Graham Scott writes:
"We assume that digital literacy and access are common to all who teach and communicate their science and to their audiences. We also assume that our digital communication is effective and that by using digital technologies learning experiences are enhanced. But are these reasonable assumptions to make?" (Scott, 2015)
The flipped teaching approach requires content delivery, whether text- or multimedia-based, to occur outside of the classroom and to replace traditional live lectures. Academic–student contact time is then used for interactive activities aimed at facilitating deeper conceptual understanding of the subject material (O'Flaherty & Phillips, 2015). Ideally, a successful flipped class approach improves student engagement, both within and outside the class. In reality, flipping is frequently a challenging task for students and academic staff alike. The potential for educational gains (and possible cost saving) are there, but support and persistence is likely to be needed to achieve success. Massive open online course (MOOC) style of flipped learning has received the most media attention over the past two years (Haggard et al, 2013). Thankfully, the initial hysteria around MOOCs revolutionizing education and sweeping universities away has now subsided. In its place have emerged more realistic thoughts about the impact of free online education. However, MOOCs have refocused attention onto open educational resources (OER), a potentially valuable area where content provision has ridden ahead of an underlying pedagogy (Knox, 2013). Valuable OER resources are out there - images on Flickr, videos on YouTube, explanations on Wikipedia. This article discusses social media and mobile technologies and considers their positive and negative aspects.

Social Media
Traditional media usually portray social media (computer systems that allow users to create and share information, ideas and content via online virtual communities and networks) as flat and monolithic in structure. In reality, nowhere has the complexity of the Internet and its influence on young people's lives been portrayed as well as by danah boyd in her recent book It's Complicated: The Social Lives of Networked Teens (boyd, 2014). Boyd reminds us that the hysteria surrounding the Internet is only the latest incarnation of the end of the world, and that Socrates warned of the dangers of the alphabet and writing, citing implications for memory and the ability to convey truth (sound familiar?). She is particularly eloquent on technological determinism and the Idea that the Internet is unique and cannot be compared with anything that went before it:
"Utopian and dystopian views assume that technologies possess intrinsic powers that affect all people in all situations the same way. ... These extreme rhetorics are equally unhelpful in understanding what actually happens when new technologies are broadly adopted. Reality is nuanced and messy, full of pros and cons. Living in a networked world is complicated."
Social media is, generally speaking, “free” to education providers because successful projects tend to hook into existing sites where student attention may already be focussed rather than trying to create artificial, closed, small scale networks (Bik & Goldstein, 2013; Maleko et al, 2013). This means that there are no infrastructure limitations (beyond the provision of adequate wifi connectivity) as social media access tends towards “bring your own device” (byod) models where students provide their own phones or tablets. On the other hand this means a variety of user experiences which must be taken into account when designing any activity. This apparent loss of control over students goes hand in hand with perceived institutional risk (reputation) in the public sphere. Although such risks are usually overstated, there is certainly a need to pay attention to professional development of students to minimize risk of career damage.
And yet real changes are occurring on the Internet as it becomes a mature medium which has discovered what is effective online, particularly in terms of monetization. Doors are closing around the Internet. Net neutrality is a serious concern - will we have one internet or many? The big players - Google, Facebook and Buzzfeed - have discovered that monetization requires building a lobster pot for online attention - if you send your students to Facebook will you ever get them back? However, if you're willing to take the risk, there is a lot of high quality microbiology information available via Facebook (Table 1).

Table 1. Microbiology pages on Facebook.
American Society for Microbiology https://www.facebook.com/asmfan
Morbidity and Mortality Weekly Report (MMWR) https://www.facebook.com/CDCMMWR
FEMS (Federation of European Microbiological Societies) https://www.facebook.com/FEMSmicro
MicrobiologyBytes https://www.facebook.com/MicrobiologyBytes
Society for Applied Microbiology https://www.facebook.com/sfamfb
Society for General Microbiology https://www.facebook.com/SocGenMicro

The growing neoliberalization of the online world is about control, generating tension compared to the early blue sky days. This narrative runs parallel to other changes in education (Kahn, 2015). We now have the Internet as infrastructure, commoditized and packaged. How would your working week go if the Internet were not there on Monday morning? The economies of scale are attractive to the online giants and education providers alike - more users, more students. The online world works because of scale rather than depth. Billions of people use Facebook daily, but what do they use it for? Popular sites such as Buzzfeed (just Google that if it's not familiar to you) are notorious for dumbing down and the invention of the listicle, short-form writing that uses a list as its structure, fleshed out with enough copy to be published as an article: "10 things you should know about E. coli". But as boyd (2014) has warned us, any over simplistic view of the Internet is flawed. At the same time as becoming notorious for items such as "29 Cats That Forgot How To Cat", Buzzfeed has also invested heavily (US$50m) in longform journalism, including science journalism. Does it matter where people read about microbiology? (Ghorayshi, A. (2015) Mail-Order Viruses Are The New Antibiotics http://www.buzzfeed.com/azeenghorayshi/mail-order-viruses-are-the-new-antibiotics; Dan Vergano, D. (2015) Africa At Risk Of Ebola Flare-Up, WHO Leaders Say http://www.buzzfeed.com/danvergano/africa-at-risk-of-ebola-flare-up-who-leaders-say)

Mobile technologies
For all the buzz around social media the really big impact of recent years has been from mobile technologies and information interfaces. The byod movement stresses reduced infrastructure costs, convenience and the potential for engagement with new forms of delivery. Set against this are the challenges presented by mobile technologies – an expectation of reduced response times or immediate responses, and the always on nature leading to reduced separation of work and private life, causing stress and anxiety for staff and students alike (Lepp et al, 2014).
Academic staff frequently feel that they lag far behind students in understanding and familiarity with mobile technologies, although the true picture is complex (Cochrane, 2014). Commercial organizations have adapted to the 24/7 mobile world much faster than academics. While it is unthinkable to someone of my generation that anyone would choose to read an academic paper on a mobile phone, that is a frequent occurrence in my experience. Although students push the boundaries of content delivery on mobile devices, the old formats do not work effectively on small screens. Longform text needs to be broken into small chunks and visual or audio content is frequently preferable to any text at all (Ekanayake & Wishart, 2014). However, mobile bandwidth is frequently limited (or expensive), and so video is not always a viable choice to replace text. Audio content shows much promise, for example for academic feedback, and there has been a resurgence of interest in podcasts, for example Vincent Racaniello's This Week In Virology (http://www.microbeworld.org/podcasts/this-week-in-virology). Adapting to mobile technology requires considerable thought and planning - it's not simple to switch from text-dominated to multimedia content, which is one reason why academia has lagged behind in this area.

Summary
In an era where so much information is available from so may different sources, characterizing science education dichotomously as either “formal” or “informal” no longer makes sense (Falk et al, 2015). For all the rapid evolution of technology in recent years, academics have been slow to adapt and even slower to adopt the most effective formats and techniques (Shelton, 2014). However, the adoption of new technology has frequently had a negative effect in promoting over assessment! Even though constructive alignment of the interdependence of learning outcomes, teaching methods and assessment is widely accepted (Biggs, 1996), the availability of shiny new electronic gizmos sometimes overrides sense.
Yet the overall picture is not so bleak and there have been wins. YouTube is an example of this with its accidental invention of the perfect OER - ideal unit size, hosted by a trusted brand, cross platform availability (Grant, 2015). Millions of people are now receiving microbiology education via a plethora of informal channels. Once academia catches up with the altered technology landscape, the future looks bright for an educated public (Jandu, 2012).

References
Biggs, J. (1996) Enhancing teaching through constructive alignment. Higher education, 32(3): 347-364.
Bik, H.M., & Goldstein, M.C. (2013) An introduction to social media for scientists. PLoS Biology, 11(4), e1001535.
Boyd, D. (2014) It's Complicated: The Social Lives of Networked Teens. Yale University Press. ISBN 0300166435.
Cochrane, T.D. (2014) Critical success factors for transforming pedagogy with mobile Web 2.0. British Journal of Educational Technology, 45(1): 65-82.
Ekanayake, S.Y., & Wishart, J. (2014) Mobile phone images and video in science teaching and learning. Learning, Media and Technology, 39(2): 229-249.
Falk, J.H., Dierking, L.D., Osborne, J., Wenger, M., Dawson, E., & Wong, B. (2015) Analyzing Science Education in the United Kingdom: Taking a System‐Wide Approach. Science Education, 99(1): 145-173.
Grant, W. (2015) Science communication on YouTube: Factors that affect channel and video popularity. Public Understanding of Science, 19 February 2015. doi: 10.1177/0963662515572068.
Haggard, S., Brown, S., Mills, R., Tait, A., Warburton, S., Lawton, W., & Angulo, T. (2013) The Maturing of the MOOC: literature review of massive open online courses and other forms of online distance learning. Department for Business, Innovation and Skills, UK Government.
Jandu, N. (2012) Microbiology for the masses: teaching concepts and skills for a general audience. Trends in microbiology, 20(10): 459-460.
Kahn, P. (2015) Critical perspectives on methodology in pedagogic research. Teaching in Higher Education 20 (4): 442-454. doi: 10.1080/13562517.2015.1023286.
Knox, J. (2013) Five critiques of the open educational resources movement. Teaching in Higher Education, 18(8): 821-832.
Lepp, A., Barkley, J.E., & Karpinski, A.C. (2014) The relationship between cell phone use, academic performance, anxiety, and satisfaction with life in college students. Computers in Human Behavior, 31: 343-350.
Maleko, M., Nandi, D., Hamilton, M., D'Souza, D., & Harland, J. (2013) Facebook versus Blackboard for supporting the learning of programming in a fully online course: the changing face of computing education. In Learning and Teaching in Computing and Engineering (LaTiCE), (pp. 83-89). IEEE.
O'Flaherty, J., & Phillips, C. (2015) The use of flipped classrooms in higher education: A scoping review. The Internet and Higher Education, 25, 85-95.
Prensky, M. (2001) Digital natives, digital immigrants part 1. On the horizon, 9(5): 1-6.
Scott, G. (2015) Teaching and communicating science in a digital age. F1000Research 2015, 4: 83 doi: 10.12688/f1000research.6323.1.
Shelton, C. (2014) “Virtually mandatory”: A survey of how discipline and institutional commitment shape university lecturers’ perceptions of technology. British Journal of Educational Technology, 45(4): 748-759.





Wednesday, June 17, 2015

Openness needs a business model

Cover Open is not free. Openness needs a business model. Guess which one the neolibs have in mind.
"Laurillard is careful to separate education from access to resources, but she presents the relationship between the OER movement and web technology as largely unproblematic. I want to argue that the relationship is more layered and more complex, and that openness is an affordance of an assemblage and related to broad social conditions, national politics, economics and law."


Openness, technologies, business models and austerity. Learning, Media and Technology 15 Jun 2015 doi: 10.1080/17439884.2015.1051307
Open education emerged when the state had an active role in shaping and financing post-secondary education. In the twenty-first century, two pressures influence the way openness is conceived. The first is the compounding of neo-liberal economics with austerity following the financial crash of 2008. The second is the consolidation of networked and digital technologies at an institutional and infrastructural level, illustrated by massive open online courses (MOOCs). This article examines the place of open education in this emerging climate of economic constraint and technological possibilities. The article argues that openness is not a property or feature of a technology but that such properties can result in affordances. This understanding informs a review of openness in The Open University (UK), in relation to MOOCs and in the open educational resources movement. A relational view of affordance suggests that openness depends in significant ways on the character of broad social processes and that if they change then the affordances of technologies for openness change with them. The current marketisation of higher education, the reduction in public finances and continuing economic uncertainty lead to contradictory and conflicting pressures. Arguing in favour of education as a public good, the article criticises calls for a ‘business model’.




Monday, June 15, 2015

Common sense about feedback

Exam feedback

This paper takes a common sense approach to fixing "the feedback problem" - the stuff we all know is broken. Clue: It's not rocket science. Fixes needed include:
  • Aligning expectations (of staff & students, and between teams of markers)
  • Identifying all feedback available
  • Develop the student’s ability to self-assess
  • Draft-plus-rework - instead of the student simply producing a ‘finished’ product
  • Improve the linkage of assessment strategies across programmes and between modules/units
  • Ensure feedback is timely
  • Consider the role of marks – they obscure feedback
  • Reduce overemphasis on written feedback – oral can be more effective

All simple, logical and entirely correct. So here's the problem - we all know this, so why don't we do it? What is the structural problem that prevents the fix being implemented?


A scholarly approach to solving the feedback dilemma in practice. Assessment & Evaluation in Higher Education 14 Jun 2015 doi: 10.1080/02602938.2015.1052774
It is clear from the literature that feedback is potentially the most powerful and potent part of the assessment cycle when it comes to improving further student learning. However, for some time, there has been a growing amount of research evidence that much feedback practice does not fulfil this potential to influence future student learning because it fails in a host of different ways. This dilemma of the disjuncture between theory and practice has been increasingly highlighted by the UK National Student Survey results. This paper uses a model of the assessment process cycle to frame understandings drawn from the literature, and argues that the problem with much current practice resides largely in a failure to effectively engage students with feedback. The paper goes on to explore how best to effectively engage students with assessment feedback, with evidenced examples of feedback strategies that have successfully overcome this problem.








Friday, June 12, 2015

A fundamental misalignment: undergraduate science research projects

Darwin The Problem: An industrialized scale higher education system where "research projects" concentrate on the product rather than the process.

Proposed Solution:
"Our analysis suggests that even highly experienced academics involved in the provision of undergraduate research projects experience a range of contradictions around their assessment and evaluation. At the base of these contradictions lies the need for contextual judgement in the assessment of desired higher order, process-related learning outcomes and the misalignment between assessment and desired learning. It is perhaps not surprising, given the role of the journal article as the currency of scientific worth, that formal reports have been adopted as the dominant mode of assessment. However, the misalignment between this mode and the intended learning could be substantially avoided if assessment practices were to be redesigned. While we do not suggest abandoning formal reports, we do advocate the development of ways to make complex learning and higher order thinking visible as an additional and substantial component of assessment, and the concomitant development of appropriate grading criteria. In their study of unassessed, not-for-credit research experiences in science, Laursen et al. (2010) outlined a range of informal ‘markers for growth’ used by project supervisors in judging their students’ progress. They included cognitive, affective and behavioural markers: signs that students were engaging in or developing critical thinking, an understanding of the conceptual framework their project was part of, engagement in research, learning through problem-solving, dealing with risk and uncertainty, and developing independence. They have much in common with the intended learning outcomes described here. As well as confirming and expanding on the importance of these markers, our findings provide more concrete illustrations of the actions and thinking in which a student is expected to engage. We suggest that such (predictive) descriptions could form the basis for formal as well as informal judgements of student progress and achievement. In addition, their very richness and contextual specificity provides a valuable basis for students to form an understanding of how to progress and develop."


Unassessable or simply unassessed?


A fundamental misalignment: intended learning and assessment practices in undergraduate science research projects. (2015) Assessment & Evaluation in Higher Education doi: 10.1080/02602938.2015.1048505
Authentic experiences of research are seen as valuable elements of undergraduate science, providing motivation for students and linking the research and teaching activities of academics. But as such experiences are made available to increasing numbers of students as formal, graded parts of the curriculum, important questions are raised about their pedagogical function and the ways in which they are assessed. This article draws on interviews with academics involved in the provision of such experiences to ask: what do academics intend that their students learn, and do conventional approaches to project assessment relate clearly and effectively to these intended outcomes? We describe four categories of intended learning and suggest that conventional approaches to assessment are fundamentally misaligned with most of these outcomes. We argue that this is due to the focus of these approaches on the products, rather than the processes and experiences, of research, a focus that partly arises from a sense of discomfort with assessment based on context-dependent judgements informed by subconscious expertise. We further suggest that alternative approaches to assessment could build on academics’ own descriptions of the experiences and behaviours they value in students.









Thursday, June 04, 2015

Flipping lectures: a reflective diary

Jeremy Pritchard has shared his experience of flipped lectures in the form of a reflective diary via SpeakerDeck. (Anyone else used SpeakerDeck? Looks interesting - no audio narration option though?) I like this format more in many ways than the traditional academic paper for this sort of discussion and it's very helpful to have experience shared in this way. Well worth a read:





Thursday, May 14, 2015

Social media in education again already

Bluebottle I've just finished writing yet another invited review article on ... yawn ... social media in education. This paper could have been written 10 years ago. In fact, I think I did. Part of me wants to ask Why is this taking so long? 
And yet, the early days Internet analogy to the printing press answers that question. Wake me up when we've moved on.


Josefsson, P., Hrastinski, S., Pargman, D., & Pargman, T.C. (2015) The student, the private and the professional role: Students’ social media use. Education and Information Technologies, 1-12
Research has shown that students perceive a distinct divide between educational and private use of social media. The present study explores this divide by focusing on master students’ perception of roles when using social media in a higher education context. A qualitative method has been used, mainly comprising of analyses of home exams and interviews, which were conducted with students enrolled in the master’s course “Social media technologies”. Results support previous research stating that students perceived a distinct divide between educational and private use of social media, and furthermore provide a more detailed understanding of this divide. The results from the study also indicate that there is yet another type of use: social media as a tool for career-building purposes, or what is labeled as professional use. Implications of social media for use in higher education are described through the analysis of three roles as performed by the individual: the student role in educational settings, the professional role for career-building, and the private role.



Tuesday, April 21, 2015

Dangerous words - critical perspectives on methodology in pedagogic research

Neoliberalism I'm off to give a workshop on pedagogic research this afternoon. Individual career trajectories aside, you might think that's a comparatively neutral activity. It turns out that "academics must develop a counter discourse to that of neoliberalism if higher education's wider societal responsibilities are to be served". This paper argues that:
"both the Structure of Learning Outcomes (SOLO) taxonomy and constructive alignment downplay the distinctive characteristics of different forms of knowledge. Learning conceived according to such a model can easily become co-opted into the production of employable subjects. Our analysis suggests that significant gaps remain in the basis for one of the most widely adopted perspectives on learning and teaching in use today."

Tin hat on then.


Critical perspectives on methodology in pedagogic research. (2015) Teaching in Higher Education 20 (4): 442-454. doi: 10.1080/13562517.2015.1023286
The emancipatory dimension to higher education represents one of the sector's most compelling characteristics, but it remains important to develop understanding of the sources of determination that shape practice. Drawing on critical realist perspectives, we explore generative mechanisms by which methodology in pedagogic research affects the sector's emancipatory potential. In this, we critique the research that led to the Structure of Learning Outcomes taxonomy. Our analysis here enables us to offer a revised version of the taxonomy that is sensitive to horizontal knowledge structures. We further consider a set of studies employing approaches to research that were sensitive to variation in knowledge across disciplines, social relations, reflexivity, corporate agency and other considerations, enabling us to illuminate the stratified basis for our explanatory critique. There is potential for our analysis to assist in developing approaches that are distinctive to research into higher education.




Thursday, April 16, 2015

Teaching and communicating science in a digital age

Drosera capensis Teaching and communicating science in a digital age
This F1000Research channel brings together papers developed from presentations made at Teaching and Communicating Science in a Digital Age, a Society for Experimental Biology symposium involving Higher Education Professionals from across the globe to reflect upon the impact that digital technologies have and will have upon aspects of the communication of science:

  • Tweets from the forest: using Twitter to increase student engagement in an undergraduate field biology course: "Although students did not think they would use Twitter after the course was over, 77% of the students still felt it was a good learning tool, and 67% of students felt Twitter had a positive impact on how they engaged with course content."
  • Challenges and opportunities for early-career Teaching-Focussed academics in the biosciences: "We identify that there is a need for the learned societies to come together and pool their expertise in this area. The fragmented nature of the Teaching-Focussed academic community means that clear sources of national support are needed in order to best enable the next generation of bioscience educators to reach their full potential."
  • Digital collaborative learning: identifying what students value: "Here we present findings generated on PlantingScience, an online community where scientists from more than 14 scientific societies have mentored over 14,000 secondary school students as they design and think through their own team investigations on plant biology."
  • Interactive lectures: Clickers or personal devices? "We find that students prefer interactive lectures generally, but those that used their own device preferred those lectures over lectures using clickers. However, device users were more likely to report using their devices for other purposes (checking email, social media etc.) when they were available to answer polling questions."
  • Digital teaching tools and global learning communities: "We report on our ongoing efforts to develop a global learning community that encourages discussion and resource sharing."
  • Why do we bother? Exploring biologists' motivations to share the details of their teaching practice: "In this paper I explore the motivations of these individuals to disseminate the detail of their teaching practice. I reflect upon my own experience and my observations of the experiences of others and in doing so I explore common enablers/disablers to engagement with SoTL."




Wednesday, April 15, 2015

Sustainable assessment revisited

Arse "Sustainability in education may be interpreted as a feature of educational systems. It is not just about sustainability of the physical environment, but also about the sustainability of educational practices, some of which may be too resource-intensive to survive in a constrained financial environment. That is, promoting teaching, learning and assessment practices that involve less face-to-face but perhaps more effective contact between teachers and students. However, such a view of education is too narrow and provision-centred. What is more important for the longer term is to look at the notion of sustainability from the perspective of learning. What educational practices are needed now in order to form and sustain learners who will be able to operate effectively in a complex society?
From such a viewpoint, sustainability becomes transformed into a question of whether educational provision equips learners effectively, not just for immediate educational requirements, such as what they need to be able to do in a course, but also for whether it prepares them for what might be required in the future whether that be in educational institutions or beyond. That is, in higher education do educational activities equip learners for the multiplicity of challenges they will face after graduation? From this perspective, the consumption of educational resources is judged in terms of their effect in producing students who go on to become self-managing persons who, in association with others, can draw on whatever they need to continue learning effectively beyond the end of the course and be able to make judgements about their own learning outcomes. Sustainable learning is thus a function of what students gain from education, not what inputs are put into the process.
This paper focuses on the particular role of assessment in sustainability debates within education. It considers what sustainable assessment means and what is involved in building such ideas into courses to support learning in the longer term. Teachers may well be teaching with the longer term in mind, but unless this work is actively supported through assessment practices, their good intentions can be inhibited. This paper positions sustainable assessment as a way of rethinking outcomes, curriculum and pedagogy away from a focus on disciplinary knowledge to what students can do in the world. It reviews literature that has taken up the idea of sustainable assessment and its implementation. While it is judged to be a successful intervention in thinking about assessment, it suggests that the implications of sustainable assessment have yet to be fully embraced. This paper considers where the emphasis for further development should be and what related ideas might also be considered. It concludes by identifying directions for embedding sustainable assessment in courses and it discusses some of the key issues to be considered, with a particular stress on the role of assessment design."


Sustainable assessment revisited. Assessment & Evaluation in Higher Education, 09 Mar 2015 doi: 10.1080/02602938.2015.1018133
Sustainable assessment has been proposed as an idea that focused on the contribution of assessment to learning beyond the timescale of a given course. It was identified as an assessment that meets the needs of the present in terms of the demands of formative and summative assessment, but which also prepares students to meet their own future learning needs. This paper reviews the value of such a notion for assessment; how it has been taken up over the past 15 years in higher education and why it might still be needed. It identifies how it has been a successful intervention in assessment discourse. It explores what more is needed to locate assessment as an intervention to focus on learning for the longer term. It shows how sustainable assessment can help bridge the gap between assessment and learning, and link to ideas such as self-regulation, students’ making judgements about their own work and course-wide assessment.





Thursday, March 26, 2015

Gameplay Engagement and Learning in Game-Based Learning

Purple Earlier this week I listened to student presentations about the use of educational games, and very good they were too. But I'm a sceptic when it comes to games in education, believing that there's usually a better in answer. In spite of that, games won't go away, so this systematic review is about the best there is at telling you everything you need to know.


Gameplay Engagement and Learning in Game-Based Learning: A Systematic Review. Review of Educational Research March 25, 2015, doi: 10.3102/0034654315577210
In this review, we investigated game design features that promote engagement and learning in game-based learning (GBL) settings. The aim was to address the lack of empirical evidence on the impact of game design on learning outcomes, identify how the design of game-based activities may affect learning and engagement, and develop a set of general recommendations for GBL instructional design. The findings illustrate the impact of key gaming features in GBL at both cognitive and emotional levels. We also identified gaming trends and several key drivers of engagement created by the gaming features embedded within GBL, as well as external factors that may have influences on engagement and learning.