Evidently not enough: The limits of evidence-based policymaking

'There is a wave of enthusiasm for good quality evidence, passing through all corners of government at the moment,' says Ben Goldacre in Building Evidence into Education - and this, his paper for the Education Endowment Foundation, can be seen as the latest manifestation.

Moreover, the EEF, with the National Institute for Health and Care Excellence (NICE) serving as the model, is one of four new institutions announced at the beginning of March by the Cabinet Office Behavioural Insights Team - or the 'Nudge unit', as it is better known - all of which are dedicated to assembling and making available high-quality evidence.

This is all much to be applauded. It would be a huge step forward if policymakers, particularly politicians, were just - please - to ask as matter of course, the obvious, first question every time an intervention is proposed: what are the facts that make me think this will work? It is shaming that not only ministers but all of us like to rely on our hunches about what is effective, on our tastes for what we would like to be effective, on our friends, the pub chatter, the dinner party, the press, our values - or on any number of other random sources as inspiration and justification for important decisions.

Goldacre's paper is another, powerful, exercise in persuasion that evidence matters. And that simple message may be what we should exclusively concentrate on now, rather than getting bogged down in methodological issues such as whether randomised control trials should really be the gold standard of evidence, or what other ways we have of establishing relevant facts.

But even when the Nudge unit's initiative is fully operational, and we have four excellent warehouses full of well-authenticated policy products, there will still be hard questions to be answered about how social policy decisions are to be made, which cannot be resolved just by getting ever more and better evidence. Indeed, Goldacre is well aware of this, as he makes clear in the final paragraph of his paper:

'We recognise that being a good doctor, or teacher, or manager, isn't about robotically following the numerical output of randomised trials: nor is it about ignoring the evidence, and following your hunches and personal experience instead. We do best by using the right combination of skills to get the best job done.'

The first step down this road may be to see that there is a lot hidden, or muddled, in phrases such as 'social policy decisions'. When a particular doctor or child welfare officer is deciding what action to take to help a particular patient or child, they are not making a 'social policy decision' - or a policy decision at all. They are making an individual decision. For that, it may be a great help to know that there is strong evidence showing that on average a certain intervention will produce a good result. But however strong that evidence might be, an experienced practitioner is always going to also exercise their professional judgment. Even if it is wholly reliable, even if it comes from the best of authorities, certification of particular remedies is not all that doctors and teachers feed into their decisions.

Indeed, it is a commonplace that these decisions are about particular people in particular circumstances - in a context. And when we use the term 'personalised services', this is what we should mean: someone looks at your problem and helps to see what is best for you.

Crucially, this is not the same as the JobCentre officer getting the right facts about you to work out how the benefits code applies. Only in the most impoverished sense could such an approach be described as contextual. But from a public policy standpoint, we do need to understand what doctors and teachers think they are doing when they have a child or a pupil in front of them and they take an action, when that action goes beyond simply collating the individual facts and applying the general rules.

The extreme view of the importance of context is Friedrich Hayek's, as set out in 'The Use of Knowledge in Society'. It is only the man on the spot who can see the ever-changing detail that must be respected if he is to 'get the best job done'. Only he has 'the right combination of skills' to which Goldacre refers in the quoted passage above.

At the other end of the spectrum is the notion of 'managerialism', as applied in recent years to public services. Here, not only objectives but also the means to achieve them can be specified in advance, by general rules or statements of 'best practice'. Success is then measured by audit, to determine whether conformity with these rules has been achieved.

If the Hayekian view is right, this managerial approach misses a central truth about what skills are needed to make good decisions. So the next step is to put some flesh on the bones of Goldacre's concluding paragraph.

To be clear, it is not that evidence-based policy and the creation of institutions dedicated to disseminating the best available evidence are bad things: on the contrary, they are an essential aid to help practitioners make better individual decisions. Nobody thinks that doctors should be ignorant of developments in medical science, however experienced they may be.

Nevertheless, such information is just part of what the teacher or doctor uses when they exercise their professional judgment in deciding what to do in a particular situation. What we need, then, is much harder to uncover than more and better rules about what constitutes evidence. We need to work out, without bluffly asserting simply that we should 'trust the experts', what we mean by 'exercising professional judgment'.

There is clearly more to this than merely respecting the facts, weighing them carefully and applying them judiciously. We all know from our everyday lives that people sometimes make bad decisions about where to go on holiday, for instance, just by not bothering to find out what the hotel or resort will really be like. But we also know people who spend hours doing their research on the internet and still end up having a rubbish trip. They didn't know how to use the facts.

An understanding of how individual professionals make individual decisions matters because if talk of decentralisation and building trust in the professionals - in healthcare or elsewhere - is to make any sense then it has to amount to more than vulgar hostility towards managerialism, box-ticking and targets. It also has to go beyond pointing fingers at the hierarchies of evidence which put expert opinion lowest on the list of 'what counts'. We have to have a story, as compelling ideologically as the managerial story, about how it is that professional judgment works best, in ways that managerialist processes cannot mimic.

Evidence-Based Policy: A Practical Guide to Doing It Better, by Jeremy Hardie and Nancy Cartwright, was published in September 2012 by Oxford University Press