Thinking outside the penalty box

21/12/2017

Sam Ashworth-Hayes

Thinking outside the penalty box

This blog post is a bit different. A lot of the articles we write try to show you how we change people’s behaviour; the principles we rely on, our successful case studies, our ideas for the future. In every case we’ve emphasised the importance of proving cause and effect; that the course of action we take leads to the outcome we want. This post focuses on this problem – known in the economic literature as ‘identification’. How can we show that one thing affects another?

Our case study is a paper by The Behaviouralist’s co-founder Robert Metcalfe on education. A lot of time, money, and thought is put into getting the best out of our schools and students. During an election campaign, the airwaves are filled with politicians promising smaller class sizes and better facilities, while teachers up and down the country drill the importance of self-study into their students.

It’s safe to say that we want to know what makes students ‘tick’. What factors go into making our output – knowledge, as proxied by exam performance – and what can we do to improve its production? A vast array of academic studies have focused on various bits and pieces; class sizes, IQ, teacher quality, parental education, and so on. Something that’s harder to measure is the student’s own effort; the hours spent chained to their desk reading the course notes.

What’s not immediately clear is just how important this effort is; it comes tangled up with a host of other variables, including a student’s aptitude for a subject, their home life, their intelligence. As a current postgraduate, I can – anecdotally and ruefully – confirm that my effort certainly seems to correlate with my performance. Then again, that might just be because I spend more time on subjects I enjoy and have a knack for.

The way to show the effect of effort on attainment is simple; you just vary effort while holding absolutely everything else constant, and bang; instant identification. The problem – the three-foot prehistoric dragonfly in the ointment – is doing this. You can’t have an individual student take the same exam a few thousand times, varying their effort with each run through. You will have difficulty altering incentives within schools in such a way that everything else is held constant.

Instead, you have to find a natural experiment; some situation that creates variation in effort, but that doesn’t directly affect education. The challenge is to find the right set of circumstances – thinking outside of the box. Or, in this case, the penalty box.

The world’s most watched football tournaments are the World Cup and the European Championships (‘Euros’). Both take place in four year cycles of qualification and finals, and match up in such a way that one of these tournaments is held every other year. The 2016 Euros are followed by the 2018 World Cup. Despite the perennial underperformance of the national football team, England is a nation of obsessive football watchers – and students take their exams as these tournaments reach their crescendo.

What this means is that students in tournament years have an incentive to put off work and watch the football – adding their study time to the litany of casualties left by England’s latest sporting misadventure. Better yet, whether a student takes their high school exams – GCSE’s – in a year with a tournament is completely random, and totally unrelated to other factors affecting their education.

So what did the results show? In tournament years, the average grade dropped for the exams held in the weeks when football was played. Unsurprisingly, the effects were particularly large for boys. So yes, working hard does matter. As Sir Ian McKellen wisely puts it…. Seen in this light, England’s early exits could be seen as a public spirited attempt to improve the education of young Brits.

The point of this post isn’t to recommend that the government moves exams away from the periods when the football is on tv – although that could potentially be a very popular move. It’s to show that there are often ways to answer very complicated questions with enough ingenuity. This sort of lateral thinking is often applied in our consultancy work; we are often lucky enough to have the luxury of running our own experiments in the form of randomised control trials – the gold standard for identification. Thinking up a way to structure an experiment, however, is a non-trivial task. Hopefully, this post will give a bit of an insight into the sort of thing we worry about. To find out more about our work, click here.