The end of the party: How we could be heading for a post-democratic era

I have been researching the issue of political leadership in recent months. Here is a piece I wrote with Professor Michael Kenny of London's Queen Mary University on the subject for the summer edition of the New Statesman:

The rise of anti-politics populism in Europe has coincided with the appearance of new, charismatic leader figures whose appeal rests on their unwillingness to accept the compromises and responsibilities that governing in a representative democracy requires. In the UK, we think of Nigel Farage telling it as it is down the pub; or Russell Brand, bathing in the platitudes of anti-politics and the afterglow of 1960s radicalism. In France, Marine Le Pen has guided an older style of neo-fascism towards a modern, media-friendly form of aggressive populist-nationalism. And in Italy, the comedian Beppe Grillo fronts the Five Star Movement, which eschews the most basic kind of political responsibility, standing for elections yet celebrating its refusal to take power under any circumstances.

All of these figures speak to the inchoate desire of electorates for leaders who understand their concerns and satiate their anxieties. They also reflect deeper trends that pose a profound challenge to the culture and legitimacy of representative democracy – the growing unwillingness of citizens to identify with the kinds of compromises and tough choices that ruling requires. Responsiveness without responsibility, the political zeitgeist of the populist era, is undercutting the values – tolerance, patience, playing by the rules of the game – on which representative government was founded.

One response to what is happening is for mainstream leaders to jump aboard the populist bandwagon and harness it for their own purposes. In recent years both George Galloway and Nick Clegg have, in different ways, done exactly this, with mixed results. In Italy, the election of Matteo Renzi has led some to wonder if a new version of the kind of retail politics and energetic centrism which was a hallmark of the 'Third Way' politics of Clinton and Blair might provide the solution to the European left's problems today. But the political centre ground, too, has been churned over in the past decade by the rise of anti-politics, deep-rooted anxieties over identity and belonging, and growing discontent at the inequalities and inequities generated by the economic system.

However, it is important to appreciate that western voters do not possess a developed preference to be governed by irresponsible populists. Rather, lending temporary support to those such as Farage and Le Pen is a way of signalling disaffection with the mainstream. This growing disenchantment stems from a paradoxical set of expectations of leaders – and the depth of the sentiment is an underexplored aspect of the anti-politics mood. Leaders matter a lot to citizens, as plenty of political science research into 'valence' voting (that is, voting on the basis of who is best to deliver widely shared goals, rather than specific policy positions) shows. And it is through the prism of judgements about the character and capacity of individual leaders that voters develop a sense of identification with parties and their programmes.

Yet nearly all the mechanisms for socialising the party allegiances of citizens have ceased to work. Family, church, trade union and community are far less likely to serve as conduits directing the young towards one party or another – or, indeed, towards any party at all. In the 1950s, the Conservative Party had three million members and the Labour Party over a million. Today, neither has 200,000 members.

Instead, mass politics seldom takes the form of participation in a single organisation or democratic process, but increasingly amounts to the passive observation of the unyielding, endless probing of the character, decisions and position of the party leader or prime minister. This trend has been played out in technicolour in British politics of late. Despite, or perhaps because of, his resolve to take on some serious economic reform issues and go against the grain, Ed Miliband has been subjected to sustained and personal criticism. David Cameron enjoys much better individual ratings, as well as an enviable televisual technique, but he, too, endures unending critical commentary about his judgement, work rate and apparent lack of core conviction. Few leaders escape the ritualised humiliations of modern politics.

In response, politicians and parties retreat ever more assiduously to the ground they know best and where they feel safest – donning the protective armour of the professional politician and employing the kinds of ancillary professionals (spin doctors, advisers and party managers) who offer the hope of managing the media-driven storms that are such a feature of political life.

Much the same pattern is being played out across western democracies, leaving some to wonder, in a dim echo of the 1930s, if political leadership can now only be exercised properly in undemocratic or authoritarian structures of government, such as China or Singapore.

In Europe there is an intensifying debate about what political parties and politicians should do in the face of the populist signal, yet little thought has been given to what it is saying about our political leaders. This is an issue that needs much more extensive consideration and creative thinking.

In his latest book, The Myth of the Strong Leader, Archie Brown identifies a deep yearning for the transformational leader – the one-off who emerges at a time of crisis, gathers in his (and only occasionally her) hands the reins of power and puts the country on a new path. This yearning underpins the continuing appeal of transformational figures of yesteryear, such as Lloyd George, Churchill and Attlee. It also explains the way in which the shadows of Tony Blair and Margaret Thatcher still fall across the two main parties in British politics.

Populism feeds off this kind of nostalgic yearning but also creates the demand for it in the first place. But if beating populists at their own game offers diminishing returns for leaders, attempts to identify themselves as heirs to Thatcher and Blair bring the danger of unflattering comparisons, and diminish the prospect of developing and projecting a political personality that is fitting for the present time. Simply laying claim to being a 'conviction politician' is insufficient to establishing leadership credentials, as Gordon Brown discovered early on in his premiership.

The late Peter Mair was one of the leading European political scientists of his generation, and in his final works he argued that electorates were losing their commitment to representative politics, refusing to participate, losing the habit of voting and lapsing into sullen indifference. In response, the parties and their leaders were also retreating – into the institutions of the state and away from the cultures and values of those they were meant to represent. Instead of representing the people to the state, they increasingly represented the state to the people.

Mair noted the specific importance of the dissolution of political alignments associated with the class structures that prevailed for most of the 19th and 20th centuries. This shift has recently entered a second, disorientating phase, following the rapid and bumpy transition from a manufacturing economy to a post-industrial one.

This has had many social consequences, not least the emergence of a much more segmented set of audiences for politics, as the idea of a unified public gives way to varied groups that are more fragmented in their cultural backgrounds and values. The social solidity and civic cultures that accompanied the development of socially unequal, but stable, industrial societies have been displaced by ties based on locality, religion, ethnicity and nationalism, or the more fluid loyalties of urban cosmopolitan and digitally networked populations.

Leaders are increasingly caught between contrasting social expectations. They are expected to dominate their parties, take all the critical decisions and express the popular will through decisive and effective action. They are also supposed to connect with, reassure and speak to the mood of varying social constituencies.

No single individual is in possession of the skillset demanded by these diverse expectations. As leaders fail to match up to them, our frustration and disillusion with them and with politics grow. And, in a context where coalition governments are increasingly likely, and where important social problems, such as an ageing population or climate change, require the kinds of alliance-building, resource distribution and institutional innovation that can only be conducted with a reasonably secure democratic mandate, the leadership paradox has become ever more intense.

How can political leadership respond? The task ahead is no less than to reinvent leadership in democratic societies. This means a whole lot more than lamenting the emergence of the professional politician and identifying mechanisms to broaden the talent pool within politics through such measures as primaries. It means something more fundamental, too: eschewing a leadership model that defines 'strength' in highly masculine terms as defined by Westminster tribalism, or dreaming of an Asian pivot towards technocratic or authoritarian elitism.

It also means not seeking to avoid the issue altogether, as do those – such as the Occupy movement or advocates of community organising – who want to dissolve leadership into networks and turn leaders into radical campaigners. Such moves denude politics of its governing capabilities and evade the problems associated with the increasing focus on the political leader in our culture.

Instead, successful leaders will need to understand and internalise two challenges, above all. First, they need to find a new balance between, on the one hand, the responsive dimension of leadership (which allows them to grasp and respond to shifts in public anxieties and preferences) and, on the other hand, the exercise of responsibility in governance. Getting this balance right is fraught with difficulties, given the constraints – fiscal, legal and geopolitical – that now delimit the terrain of public action for national governments. It requires political skill to articulate popular concerns while drawing realist parameters around the available courses of action.

This challenge can be approached by showing leadership on issues that are in the long-term national interest but where the public mood is hostile (such as membership of the European Union), or by taking on policy solutions that are of long-term significance but that have little immediate pay-off, such as care of the elderly.

This means learning how to recombine the visionary and the pragmatic in office – rather than doing the vision thing when campaigning and then letting pragmatism hold sway in power, as Barack Obama did in his first presidential term. It also means finding a more resonant and less distant language in which to talk to the different publics that make up the electorate.

Second, it means accepting that electorates are far more diverse in their identities and outlooks. Leadership in the era after mass democracy will require a new understanding of the particular concerns of different groups and of the common interests and ties that still bind people together. A sense of statecraft is vital here. Instead of 'delivering' bundles of policy pledges to discrete demographic groups, thereby reinforcing existing divisions, political leaders will need to be in the business of constructing institutions in which public life can be housed, allowing allegiances to be formed, professional practices to be nurtured and traditions to take shape.

Some of these will be institutions of public service, such as the NHS and children's centres, while others will be economic or democratic, such as industrial partnerships in core sectors, or city region authorities that build on the success of London and Greater Manchester. In each case, political leadership will be geared towards a more patient, long-term task of building new institutional frameworks, and away from the short-term gestures of policy announcements or target drives.

These are big challenges and no single politician can meet all of them. That is why transformative political leadership can no longer consist of heroic individuals bending politics to the will: the leader is too transitory, the change too vulnerable to repeal. Even if political leaders cannot be moral visionaries, there are institutional reforms and kinds of democratic training that can be developed which may well make political leadership of the kind we need in the 21st century more, rather than less, likely.

Power can be dispersed more widely across the state, so that cities and counties once more become significant sites of political leadership, a critical issue for England. This is one overlooked way of broadening the talent pool of future leaders. Leadership cadres also need to be recruited using a wider range of routes, with priority given to working-class, ethnic-minority and female points of entry into candidate selection and party hierarchies. And the Madisonian task of constructing stronger firewalls between money and politics ought to become the main focus of those committed to political and constitutional reform.

If we do not address these issues, there is a danger of a further drift towards what some commentators have called a 'post-democracy'. This will bring a slide towards the further hollowing-out of the democratic process, the terminal decline of mass parties and the disappearance of politicians governing in proactive and far-sighted ways. Without a reinvention of the leader, the political party itself may soon be over.

Michael Kenny and I will be returning to explore this issue further in an upcoming edition of Juncture, IPPR's journal of politics and ideas.