As Jeremy Corbyn takes the reins, Tony Wright calls on Labour to engage in a serious act of self-investigation and redefinition. The next general election is not lost already, he says, but much work must be done to win the battle of ideas that comes before it.

‘Now that the dust has settled, it is possible to examine the landscape left by the earthquake.’ That was how RH Tawney began his famous article on ‘The Choice before the Labour Party’, written soon after the debacle of 1931, when a Labour government had been overwhelmed by financial crisis and suffered political disintegration from which it took many years to recover. Today, as the party negotiates another ‘earthquake’, Tawney’s analysis of its earlier crisis still provides a useful perspective on the choices facing the party.

The party’s fundamental problem in Tawney’s view was its ‘lack of a creed’, from which all its other difficulties flowed. By this view, a creed is not a rigid doctrine but ‘a common conception of the ends of political action, and of a means of achieving them, based on a common view of the life proper to human beings, and of the steps required at any moment more nearly to attain it’. Without a firm intellectual and moral foundation of this kind, the party would necessarily falter in office and lose its way. Its function was not to ‘offer the largest possible number of carrots to the largest possible number of donkeys’ but to focus on its core purpose of ensuring that everyone enjoyed ‘the essentials of a civilised existence’.

Why is Tawney’s analysis relevant now? It seems to me that the lack of a ‘creed’ (in Tawney’s sense) is fundamental to the party’s current difficulties. Blathering on about ‘our values’ just underlines the vacuity of ideas that was evident in the leadership election. Jeremy Corbyn’s brand of old-style socialist religion was able to flourish precisely because it seemed to offer something that people could believe in, even if it had no chance of winning an election. The more his opponents talked about the imperative of ‘winning in 2020’ the more they could seem (however unfairly) like a bunch of office-seeking politicians without an inspiring new idea between them.

Ideas matter – ‘the world is ruled by little else’, said Keynes. They provide the materials from which a political ‘narrative’ can be constructed that shapes programmes and perceptions. They explain how the world is and how it might be changed. They define ends and identify means. In other words, they are not an optional extra – something to be left safely to ‘the intellectuals’ while practical politicians get on with real business – but are absolutely intrinsic to how politics works. The Thatcherites certainly understood this. ‘Never, never underestimate the importance or the power of the tide of ideas,’ said Nigel Lawson in 1987, adding that: ‘No British government has ever been defeated unless and until the tide of ideas has turned against it.’

It was New Labour that turned the tide of ideas then. At least in its genesis it assembled a set of ideas that altered the terms of political trade, about the relationship of state and market and individual and community. This ‘third way’ was much-mocked, but it was no mere sub-Thatcherism – it was the basis on which a progressive consensus could be constructed that would command the political landscape for an extended period. Those aspects of New Labour that came to discredit it, not least its style of political management, have obscured the extent to which it had succeeded in winning the battle of ideas. Its success on this front was indispensable to its wider political success.

A similar project is required now, but the task is much more difficult. In the 1980s, Labour had got itself into such an unelectable state that it was pretty clear what had to be done to get back in the game. That is what the New Labour reinvention did. It was helped by Conservative disarray, an attractive leader and a benign economy. Its promise of economic dynamism combined with social justice was a winning combination, buoyed by the revenues from a thriving financial sector. That world has now gone. This does not mean that New Labour has nothing to offer us, but it does mean that a further project of reinvention is now required.

This demands ideas – and ideas that have purchase on the world as it is today. There was something absurd in Labour’s leadership election about proposing policies for an election in five years’ time without first establishing a secure intellectual foundation for any set of policies. A sensible leader would announce that the next 18 months will be spent on some serious thinking about what the party is all about, before the fruits of this process are converted into policies and programmes. At the moment the perception is of a party that does not really know what it is about, limited to nibbling at the edges of an agenda set by the Conservatives (a bit less austerity, a bit less welfare cutting), and without a distinctive and confident voice of its own on issues ranging from immigration to the EU.

Labour lost the general election for a variety of reasons, with trust (or lack of it) in managing the economy as a fundamental feature. But behind and beyond particular factors was an uncertainty about its general message and identity. Its leader declared that the party was no longer New Labour, but what had it become? To whom was its appeal now directed? The post-election inquests identified the problem, but finding a solution was much more difficult. Unless this central issue is tackled, which involves the party finding its organising idea, individual policies will lack a coherent intellectual foundation and the next election is likely to be no more successful than the last.

It is not that people have fallen in love with the Conservatives (far from it), but the Tories are making the running in terms of ideas. Whether on the public finances or welfare (and much else) they have established a new common sense – ‘you have to live within your means’ – and Labour has no competing narrative. It may well be true that state-shrinking is an ideological project taking place under the cover of deficit reduction, but Labour has no alternative vision of the state – or of the deficit – to offer. Even when it talks about ‘equality’ the word now just sounds like a leaden abstraction without content or traction. Unless and until it can turn the tide of ideas, the outlook for the party is bleak.

Some think this bleakness extends to social democracy as a whole, not just to its British version. Certainly it is in trouble almost everywhere, as new populisms on both left and right squeeze it out, and harder times undercut a traditional politics of public spending. Perhaps social democracy was a creature of the 20th century, with its base in an organised working class – a creature now marooned as that base disappears within a society of fragmented classes and complex identities, and as a spirit of individualism erodes collectivism. Perhaps New Labour was the last hurrah of a political tradition in historic decline.

Even if that conclusion is resisted, it would be foolish to understate the scale of the contemporary challenge. Social democracy defined itself as the left alternative to Marxism. Its intellectual confidence found classic expression in Anthony Crosland’s The Future of Socialism, published in 1956 (reissued Constable, 2006), in which it was announced that capitalism had been so domesticated that ownership was no longer an issue and that equality would be secured by progressive taxation and public spending in a growing economy. Our distance from Crosland is measured by his introductory remark that he would only write about Britain and ignore the wider world, something impossible in the globalised capitalism of today.

Yet that is not the only distance. The assumption that capitalism had solved its problems now looks eccentric. More than that, though, his description of the social democratic policy armoury now needs revision. It seems to have become a political axiom that any general rise in income taxes is unacceptable, whether progressive or not, and that redistribution dare not speak its name (hence the unlovely arrival of ‘predistribution’). Not only has social democracy experienced the erosion of its political base, but it seems to have lost much of its traditional policy armoury as well.

This identifies the scale of the problem that Labour now faces, and why any political reinvention has to be based on intellectual reconstruction. Without this, its role will be confined to negative resistance or mere office-seeking. It needs to be able to say, clearly and convincingly, what it is now for, and by what means its objectives are to be pursued. For Tawney the answer was ‘simplicity itself’:

‘The fundamental question, as always, is, Who is to be master? Is the reality behind the decorous drapery of political democracy to continue to be the economic power wielded by a few thousand – or, if that be preferred, a few hundred thousand – bankers, industrialists and landowners? Or shall a serious effort be made – as serious, for example, as was made, for other purposes, during the war – to create organs through which the nation can control, in cooperation with other nations, its economic destinies; plan its business as it deems most conducive to the general wellbeing; override, for the sake of economic efficiency, the obstruction of vested interests; and distribute the product of its labours in accordance with some generally recognised principles of justice?’

This sounds like strong stuff today – although it was pretty much what was being widely said at the time of the financial crash – but its point about the issue of power at the heart of democracy and the need to assert a public interest in relation to powerful private interests remains as relevant as ever. It is not ‘anti-business’ to want a kind of capitalism that is not just a race to the bottom, or so short-termist that it damages wider interests, or to seek ways in which the state and market can cooperate to invest in skills and innovation. The same applies to nourishing new forms of enterprise, and encouraging worker participation.

The attack on vested interests, in all their forms, should be natural Labour territory. The abuse of power, whether state or corporate, should be its enemy. It should conscript Adam Smith for an assault on market behaviour that damages consumers and the pluralists for an assault on centralised state power that disempowers citizens. Its mission should be to empower people against the powerful, the little people against the big people. If this has an insurgent quality to it, so much the better, as that is what a party of the left should be and what the temper of the times demands.

It is not as if there is a poverty of material for a party of the left to work on. Evidence accumulates on the acceleration of inequality and its damaging effects. Social cohesion is weakened if a class of the super-rich detaches itself from everybody else while a class at the bottom suffers a similar detachment. A party that convincingly offers a combination of security and opportunity can speak for the majority. It strikes a chord when it talks about life-chances, and the need to remedy the gross disparities (including of life itself). Nor is this only a message for a minority or a denial of ‘aspiration’, as the evidence on the domination of the professions by the privately educated shows. In this sense the ‘little people’ are the overwhelming majority.

So Labour should have a story to tell, and one capable of finding a wide resonance. It needs to be underpinned by policy commissions that think freshly and boldly about key areas, from economic and industrial policy to a new welfare settlement, and from life chances to democratic reform. Timidity should be resisted. The party needs to show that it understands the policy challenges facing the country and is prepared to meet them with honesty and intelligence. The national health service is a case in point. Too often in the past Labour has been content to be the defender of the NHS and to reap electoral benefits as a result. This is no longer adequate. If a publicly funded health service is to be sustained – something which people clearly want – then it has to meet the costs of an ageing population and new drugs and treatments. If this is to be done, without other public services being run down to pay for it, then a secure funding stream will need to be found. The party that had the courage to set up the NHS should have the courage to propose how its future should be safeguarded.

There are similar choices and challenges on almost every front. It requires a realistic radicalism to confront them. The realism involves an understanding of how the world is, with all its complexities and limitations, including the need to keep public finances in good order, and to be able to construct a majority appeal that has something to say to both the comfortable and the uncomfortable. The radicalism involves being prepared to offer bold solutions to the problems of the day and to abandon politician-speak for some passion and conviction based on clear principles. Labour’s leadership election was dogged by a false opposition between realism and radicalism, when both are going to be required for what Tawney’s contemporary, GDH Cole, once described as a politics of ‘sensible extremism’.

There is plenty of work for a party of the centre-left to do, and no reason to believe that the social democratic tradition is in terminal decline. But it does require some urgent intellectual and political reinvention. Parties need regularly to reinvent themselves in response to changed circumstances (as the Conservatives have frequently done and as New Labour did), while politics retains an endless capacity to surprise. When predictions are most apocalyptic, that is when an injection of historical perspective is most needed.

Labour is certainly in a mess. It seemed determined to replace a doubtfully electable leader with a wholly unelectable one, a political death-wish dressed up as an embrace of unspun authenticity. Yet the party has been in a mess before. Those of us who lived through the 1980s got used to asking ‘whither Labour?’ (and whether it should be whither or wither) and if the party could ‘ever win again’. Then after the 1992 defeat there were terminal predictions about the party’s electoral future and much expert commentary to the effect that permanent Conservative rule had been established. Of course, what actually followed was the 1997 New Labour landslide. With an electorate that is ever more fragmented and dealigned, rapid reversals of political fortunes are increasingly probable. However unlikely it might seem at present, the next election is not already lost. There is still all to play for, but only if Labour can reinvent itself and get back into the game. And that, above all else, means winning the battle of ideas.

Tony Wright is professorial fellow in politics at Birkbeck, University of London, and a board member of the Centre for the Study of British Politics and Public Life.

This article appears in edition 22.2 of Juncture, IPPR's quarterly journal of politics and ideas, published by Wiley.