‘In politics if you don’t intuit the future, it is all too easy to be eaten by your past.’ In order to remain relevant, argue Gavin Kelly and Nick Pearce, those on the political left must not only apprehend the forces that are shaping the political landscape of the next decade but also determine a story – the ‘new tune’ – that will resonate with tomorrow’s voters.

The idea that electorally successful parties ‘own the future’ is a familiar trope of modern politics. Today, however, our leaders struggle to raise their sights beyond the present – and with little wonder. Rarely in recent decades has the world confronted such a potent mix of economic and political turbulence: instability in global markets, the implosion of the Middle East, the rise of a raw American populism, and the fracturing of the European project. As if this weren’t enough, the UK has chosen this moment to hold a referendum on its membership of the European Union. What price stargazing into the 2020s?

Yet parties that have suffered repeated electoral defeats, and are seeking a path back to power, have no choice but to consider the terrain on which future political battles will be fought. That is particularly true for the Labour party, which has both the luxury of broad unity on the immediate European question and the burden of a long stretch in opposition.

Despite this, and for all the speculation about Westminster power-plays and leadership plots, very few on the left have spent time defining the challenges that any renewed social democratic project needs to meet and master in the 2020s. Politics is about changing the world. But to have any hope of achieving this you first need to have sense of where it is headed: the economic, social and cultural currents upon which any political vessel sets sail. Opening this conversation is the purpose of this essay.

At various points in the past, Labour has come up with a fresh analysis of the forces remaking Britain: the incoming Attlee, Wilson and Blair governments all did so. Each united a critique of the existing economic and political order with a governing agenda that sought to liberate the energies of the country and usher in a better future. Nothing remotely similar is being developed today. Too much of our contemporary debate is predicated on a set of assumptions formed 20 years ago, some of which need ripping up. Social democrats are walking backwards into the future.

Age trumps class?

Politics is being reshaped by a set of demographic, economic and cultural shifts. Understanding each – the change as well as the continuity – is a precondition for the coherent rethinking that is needed. To jump straight to new policy prescriptions, or focus only on organisational challenges, is to miss a vital component of that renewal.

One hallmark of our political age is that European social democrats are losing elections and shedding support (with vote shares at 70-year lows), in part because they have struggled to adjust to an ageing electorate. Polarisation in turnout between young and old gives disproportionate political weight to security- and stability-minded older voters, whose numbers are swelling with increased longevity. Age rivals social class as a determinant of political behaviour in Britain in way that isn’t true of the United States, where youthful Latino migration and the mobilisation of African-American voters have forged a new progressive electoral coalition. Older voters kept Scotland in the UK, elected a majority Conservative government last May, and will likely determine whether Britain remains a member of the European Union. And their political clout will only increase through the 2020s: the over-60s population will rise by a remarkable 5 million by 2030.

Over the same period, a thwarted millennial generation will come of age. The buoyant labour market has generally given them work, but at wage levels nearly a fifth lower than their counterparts enjoyed before the crash. They’ve clung tight to existing jobs rather than move to new roles that might boost their future prospects. As they raise children, they can expect to receive greatly diminished support from a state that is simultaneously shrinking and greying. They carry large debts and pay high rents; those lucky enough to have mortgages expect to be paying them off for the duration of their working lives.

To date, this generation’s discontent has disrupted politics but otherwise left the structures of power untouched. But as it approaches middle age, its demands will feed through more aggressively into mainstream British politics: the combination of a dysfunctional housing market, stunted earnings growth and shrunken working-age welfare may reinforce the sense that the free market and small state are unlikely to solve their problems. Although they haven’t inherited tribal political loyalties – and any reheating of traditional collectivist recipes will leave them cold – their relatively weak position in society means they may be receptive to an active government that takes on interests on their behalf.

The main leftish currents of thought on generational politics are blind alleys, however. Dismiss any notion that class unity might trump demography: immigration has driven a wedge into the social democratic older vote, detaching large swathes of low- and middle-income older voters. Discount the politics of open generational conflict: any politics defined against the interests of older voters will be a losing one. And ditch the defeatist notion that an agenda that simply prioritises existing social entitlements for the elderly will suffice. This doesn’t just fall short morally – safeguarding one generation while spurning others – but it fails intellectually too. It ducks the task of crafting a politics that speaks to a key dynamic of this demographic challenge: a generation trying to find a new settlement between working longer and more flexibly, family care and a modicum of leisure. As yet, no party has properly embodied this shift and articulated its consequences and potential.

Nor has any party made the modal transition from a 20th-century catch-all cross-class electoral alliance to a 21st-century cross-generational one. Common interests will have to be forged between the generations, spanning issues from the investment in productivity growth needed to pay for decent pensions, to a properly funded NHS, intergenerational family support and affordable childcare. There are plenty of issues where interests can align, and where family love and loyalties will trump brute economic interest. But where they diverge, as they clearly do on housing and paying for social care, for instance, local leaders will increasingly have to take up the baton.

National politics still has an absolutely essential role to play, not least in ensuring that the powers and funding are in place to make local leadership a realistic option rather than a convenient cop-out. But housebuilding is an inevitably local issue, and with funding for social care and older-age benefits increasingly devolved, and more fiscal powers for local government on the cards, much of the hard graft of intergenerational compromise will be brokered by local or city leaders. They should embrace this challenge.

Economic shifts: dark skies and unlikely radicals

These demographic trends are part of a wider economic shift. The whole way we think about globalisation has changed since the Third Way years. Back in the 1990s there was optimism about ever-increasing economic openness. Emerging economies with fast-growing working-age populations were poised for long boom-times, dot-com boosterism was rife, and skills and flexible jobs markets were viewed as a sure route to rising living standards.

Now, however, the global skies look darker. Trade and capital flows have stagnated, global economic demand is weak, and austerity has shrunk the resources available to compensate those who have lost out from global change. Underlying imbalances between surplus and deficit countries persist and mercantilism is on the rise, as exporting countries with ageing populations, like Germany and Japan, build up their investment positions.

It is against this harsher backdrop that emerging challenges need to be assessed. The fiscal and monetary aftershocks of the financial crisis will continue to reverberate into the 2020s; technological challenges to the nature of work – sometimes over-hyped in the recent past – are likely to be more keenly felt; and the immediate outlook for inequality will worsen.

Macroeconomic debates – which were at the centre of ideological cleavages in the 1970s and 1980s before subsiding during the more benign 1990s – made a comeback during the crisis which has not abated since. Settled assumptions about the nature of the ‘economic problem’ – as well as the role of monetary and fiscal policy in securing steady growth, full employment and stable inflation – are in flux. Deflation is as much of a risk as inflation. Productivity has slumped and many doubt it will revert to its long-term trend. The relationship between wages and prices has confounded all predictions. Economists are scratching their heads and long-held views are being jettisoned. Erstwhile establishment figures of the late 1990s, such as Larry Summers and Adair Turner, have become the new radicals.

One consequence of these changes is fiscal weakness: as things stand, it is likely that the deficit will still be with us at the end of the decade. And, if the past is any guide, we are closer to the next recession than the last, which would mean fiscal policy taking far more strain before much of the damage done by the last recession has been repaired. Even if a downturn is some way off, our potential growth rate may be lower than previously thought. If so, the Croslandite conviction that undergirded the whole Blair–Brown era – that steady growth will fund rising social entitlements – is more questionable now than ever before.

This prospect, together with the downward shift in the long-term outlook for interest rates, and the shoddy condition of much of Britain’s public realm, necessitates new thinking. One element should be a transformation of our ambitions on investment, both public and private. There is no ‘correct’ amount of investment, but a formidable spectrum of institutions and economic commentators have all highlighted the UK’s inadequate levels of public investment; while the EU has a wider aspiration for overall R&D spending to reach 3 per cent of GDP by 2020, the UK is currently at just 1.7 per cent. Social democrats have, of course, generally been predisposed towards higher investment but, as in 2010 and 2015, have often avoided making an explicit case for it, for fear of appearing fiscally loose. That position has to change. There is no plausible route to tackling our productivity problem that doesn’t involve rising investment. Add low borrowing costs, weaker global demand and the need to build a low-carbon infrastructure into the policy mix, and it becomes utter folly to persist with fiscal rules – targeting an overall surplus – that strangle public investment. To do so is to condemn future generations to lower living standards, not less debt.

Fiscal resilience provides another challenge. As we look across the 2020s, the rising spending pressures associated with ageing, together with some erosion of the tax base, will intensify, forcing new thinking on how to widen the fiscal base (unless public services are to be allowed to decline irreversibly). Add in the possibility that the trend rate of productivity growth is declining and our fiscal future is bleaker still. Replenishing the tax-base would involve big choices: higher rates of VAT (accompanied by offsetting transfers to poorer households), serious moves towards land or wealth taxes, and reversing the trend towards an ever narrower income tax base. It may also necessitate a shift towards the use of hypothecated taxes, despite their drawbacks. Generations of social democrats have instinctively approached issues of tax through the prism of progressivity; the lens of tax resilience will have to become just as important.

The fiscal outlook also demands a steely clarity about spending choices. The familiar ruse of seeking to sound ‘credible’ via seemingly stern fiscal rules while dodging the difficult implications for current spending (by simply highlighting the difference with the political right) has been tried and has failed. Social democrats should be upfront about both their radicalism in relation to investment and their responsibility in relation to current expenditure.

Perhaps the hardest challenge of all will be projecting the right style of economic leadership for an era in which unconventional policy responses are becoming the new orthodoxy. This is a demanding task – harder now than in the more benign mid-1990s. For many on the wider left, this tilt towards the necessity of radical thinking feels pregnant with possibility. Yet while there is some truth in this, it is perhaps more obviously the case that a volatile global economic context places a huge premium on reassuring economic leadership. Self-professed radicals are likely to be less trusted to undertake risky, even if necessary, economic policy departures.

These demographic and fiscal shifts will be occurring alongside an evolution in the nature of work. Today, there is a strident debate about the outlook for employment. There is the breathless optimism of those seeking to reinvent the left around a post-capitalist sharing economy (with talk of ‘full unemployment’), alongside the techno-pessimism of those who foresee little beyond the spread of mass precariousness. Both look misguided.

The UK jobs market over recent years has been both remarkably jobs-rich and depressingly wage-poor. Throughout the downturn a formidable number of high-skill, high-pay roles were created, a trend which shows few signs of slowing. There was a similar surge in low-skill, low-pay work and ever-sharper forms of insecurity at the lower end of the jobs market, as new ways were found to shift risk from employer to worker (which, among other things, has reinforced the case for in-work support). Amidst these changes there has, however, also been important – if largely unremarked – continuity: typical job tenure is almost exactly what it was a generation ago. Talk of relentless change and the ‘end of the world of work as we know it’ will pass many people by.

As we look ahead then, we know the legal wage floor will climb steeply by 2020. This will create a great wage compression, as differentials get squeezed in the lower-pay half of working Britain. Pay progression will have a political salience like never before, and the rise of ‘one-wage towns’ in poorer areas will pose a cultural as well as economic challenge. More broadly, the incentive to automate is likely to rise, transforming (though perhaps not eliminating) a greater number of task-based roles. Tech-based occupational risk will transcend white and blue collar roles.

These challenges to securing decent work in a high employment society are set to coexist with the continued erosion of traditional collective workplace organisation. Trade unions are already invisible across large swathes of the mass-employing private service sector. Among low-paid workers, membership rates are just 9 per cent (and lower still in the private sector), in contrast to 39 per cent for the higher paid. Despite this, the left has for a generation side-stepped the question of the institutional innovation needed to reach and support the most exposed parts of the workforce. The contrast with the growing vibrancy of this debate in the US, where ‘alt union’ and other forms of labour activism are multiplying, is telling.

Looking to the 2020s, a critical component of any progressive employment agenda must be pro-labour institutional innovation, whether in the traditional service sector, among the growing ranks of the self-employed, or in the small but rising online economy workforce. A new, more innovative politics of the workplace is urgently needed.

The return of rising inequality

The politics of the 2020s are also likely to have to reckon with a resurgence of inequality – an issue that has thrown US politics into a tailspin. In the UK, income inequality, which under the main measure has been high but broadly flat since the early 1990s, is likely to rise by 2020, as prolonged benefit cuts bite and the earnings of the better-off recover ground. On child poverty, projections show the clock being turned back to where it was before Tony Blair took office.

Beyond this, the future outlook for inequality becomes much harder to divine. Shifts in global demographics – namely the levelling off of the working-age population – lead optimists to believe that there will be a slow but inevitable rebalancing of power between labour and capital, resulting in a dip in inequality. The effects of China’s demography could yet be dramatic: its working-age population fell by nearly 5 million in 2015 alone. Against this are pitted the many forces that sustain inequality – returns to technology, assortative mating, educational inequalities, the influence of big money on politics – which many think are so cemented that it would take an extraordinary disruption, something akin to war, to shift them.

The politics of rising inequality presents a sharp dilemma for the left. It will certainly give currency to the popular view that attacking inequality should be the left’s central rallying cry: it is a message that unites many shades of progressive opinion, and it points to new issues around which to mobilise, such as tackling rising political inequality and the need to build stronger firewalls between wealth and power. Yet there are few grounds – at least in the UK – for believing the sentiment that an overarching argument about the corrosive impact of inequality on society provides a winning platform. The exploits of the ‘1 per cent’ can provoke widespread outrage. But that is very different thing to vote-switching.

Historically, the left has always been most successful when it has translated its concern with equality into popular causes that command wide appeal and speak to peoples’ every day hopes for a better life: the struggle for decent wages, good housing, free education and the NHS. A renewed case for a more equal and less class-bound Britain needs to learn from this rather than lapse into abstract and economistic arguments. That means offering an optimistic account of how to meet urgent challenges, whether they be housing the young, providing universal childcare, or building a new system of social security for the self-employed. It is through shared institutions like these that pro-equality causes can put down roots and the egalitarian ethos upon which they are built endure. To paraphrase Margaret Thatcher, this achieves the real prize: changing how we think.

Identity politics in a disunited Kingdom

In the UK, these demographic and economic trends intersect with important shifts in identity that have major implications for how we govern ourselves. For now, the left is strongest in Britain’s cities, which have emerged as the key sources of economic and political dynamism in advanced economies. This metropolitan dynamic may well accelerate in the 2020s, reinforced by continued population growth, the migration of 20- and 30-somethings out of towns and smaller cities to the main urban centres, more powerful civic leadership, greater ethnic mix, and the capacity of leading cities to attract talent to help deal with social problems like under-performing schools (witness the urban Teach First phenomenon).

We should, therefore, be prepared for a widening in the gap between a small number of relatively vibrant, diverse metropolitan areas – not just London – and many other smaller towns and communities that feel bypassed. One of the many consequences will be a further stretching of the worn elastic that has held a broad-based centre-left coalition together. Economic policymaking must focus much more resolutely on post-industrial towns and their sources of future growth than it has done over the last 30 years, strengthening the case for industrial strategies with teeth and for devolution that spreads power and funding to towns and counties beyond the urban heartlands. At the same time, the left needs to use its leadership of cities to demonstrate that it is capable of effective, popular and pragmatic government on a wider scale. It must also nurture and promote its cadre of talented local politicians to create a new generation of national leaders.

Just as English devolution is likely to take hold, there is an inexorable dynamic towards a looser, messy federalism in the UK as a whole. Each phase of devolution to Scotland will create new arbitrary boundaries (the current welfare settlement being a case in point) generating further calls for a fuller transfer. The rise of a politicised English identity is likely to continue, albeit that it will run in tandem with deeper but fragmented devolution within England, creating a complex interplay of civic and national loyalties.

National political leadership will be more complicated for all parties, but especially for Labour, in the 2020s compared to the 1990s. It will mean navigating the reconfiguration of England’s two unions: a quasi-federal United Kingdom and a new external settlement of looser membership of the EU – or indeed, Brexit. These factors – alongside the decline of the two-party system and the fracturing of the UK’s political geography – will make efforts to build a political project based on ‘national renewal’ both more vital and far harder to achieve than during the 20th century.

Immigration, perhaps more than any other issue, deepens the challenge. It would be wishful thinking to pretend that the political cleavage it represents will diminish any time soon. This is in part because the recent migration surge will be carried into the population mix of the 2020s, but also because the demographic and fiscal pressures we’ve highlighted already reinforce the case for continued in-flows. A class politics of raising wages and tackling exploitation at work, combined with judicious industrial strategy and public housing investment, may help to mitigate the political challenge. But immigration is a cultural as much as an economic issue, and there is no plausible future in a revanchist white working-class communitarianism. Electoral reality points to patient strategies of forging common economic interests, and to new local and national patriotic identities in which new arrivals and the so-called ‘left behind’ can each find recognition. It will be a hard graft. Post-war history suggests that the left recovers ground after waves of migration have gently receded – when questions of how we live together in more diverse societies, rather than movements of people, become the focus of political attention.

Speaking to the new electorate

Opposition parties and movements need to spend their time assembling a politics that speaks to the electorate that is emerging, rather than to that which is passing. Some important elements of this are already in view, and tend to provide a degree of comfort to a leftish world-view: increased working poverty, growing cross-class concern at a diminished public realm, an economy skewed towards the accumulation of housing wealth not productive capital, ever more diverse and socially liberal cities, the rising ranks of the insecure self-employed, the shift to renting and fall in homeownership. All of these are big issues, and the left feels it can speak to them. Too often, though, these are all it wants to talk about.

The surest sign that a party is ill-prepared for the future is when it is content conjuring up images of the electorate that are in sync with its existing postures and prejudices. The reality is that the Britain of the next decade is likely to be characterised by a rise in highly skilled workers in a jobs market that is steadily rebalancing towards private-sector service employment. Around 700,000 fewer people are expected to rely directly on the state for their salary at the end of the decade than at the start. Over the same period, the number of working families receiving some form of tax credit support will fall dramatically from around 5 million to 2.8 million, meaning arguments about the risk of ‘losing’ support in the future may have less traction. There will be a steadily growing share of older voters – over-55s are likely to be in the majority by 2020 – who will be enjoying lower levels of poverty, record employment and high levels of outright homeownership. Prosperous, greying suburbia will share elements of social conservatism with a left-behind, economically disadvantaged electorate, creating new geographical and cultural chasms that the metropolitan left will struggle to bridge. On these and many other crucial trends that define our times the left is unsure of itself and has little to say. It has yet to begin the real work of seeking an alignment between the new electoral coalition it needs to build and its own character and policy programme.

Finding a new tune

Perhaps the hardest question posed by all these electoral shifts concerns the nature of the political parties capable of pulling together these disparate threads. Despite the Corbyn surge and the emergence of challenger parties in the UK and Europe, there are precious few signs of the successful and durable reinvention of the mass political party in the 21st century. Political parties are smaller, more state-centric and professionalised than in the post-war period, and there are no grounds for thinking trend this will be reversed.

At the same time, new social movements are in rude health, and modes of informal democratic participation have multiplied – and not just on the political fringes. Social democrats and broader progressive forces need to work out how better to combine professional, election-winning party politics with these more open, digitally enabled forms of political and civic organisation. How they practise politics matters almost as much as the policy choices they make.

There is, of course, far more to the art of opposition than developing a sense of the underlying forces that will shape tomorrow’s politics. Above all, the harsh facts of political life must never be forgotten: without a compelling candidate all else is irrelevant; absent a credible economic story, narrated by a trusted voice, no other policy stance matters.

Yet the scale of the task facing social democrats extends far beyond these bare essentials. ‘Don’t stop thinking about tomorrow’ was the background music hummed by those undertaking the ideological renewal in the early and mid-1990s. Their counterparts today need a new tune. They mustn’t let political short-termism and the huge uncertainties that always weigh against future thinking deflect them.

As social democrats cast forward, there are grounds for optimism as well as some obvious sources of pessimism. All need to be reckoned with: intellectual effort, as much as organisational graft, underpins the process of renewal. In politics if you don’t intuit the future, it is all too easy to be eaten by your past.

Gavin Kelly is a former deputy chief of staff at 10 Downing St and writes here in a personal capacity.

Nick Pearce is professor of public policy and director of the Institute for Policy Research at the University of Bath.