Has the age of political giants passed?

When David Cameron reshuffled his cabinet this summer it reinforced the widely held impression that, compared to previous generations, we live in an age of political pygmies. What has happened to all the politicians of real stature – the ones we can admire even as we disagree with them? The last of the 'big beasts', Ken Clarke, has finally been shuffled off the stage. William Hague, an adroit orator and serious man of letters, is also creeping towards the exit. The most intellectually ambitious member of the government, Michael Gove, finds himself shunted to the sidelines. In their place come some shiny new members of the class of 2010, of whom almost no one outside of professional politics has ever heard, and some long-standing party loyalists, who almost no one has heard of either. The optics are better – a few more women, some media-friendly faces, younger, fresher, more family-oriented. But no one could mistake Cameron's new cabinet for the triumph of political substance over style.

The idea that the current generation of politicians lacks stature is not just a problem for the Tory benches: it may be even worse on the other side. When Harold Wilson resigned as prime minster and Labour leader in 1976, the candidates who lined up to replace him were Jim Callaghan, Michael Foot, Denis Healey, Roy Jenkins, Tony Benn and Tony Crosland – all politicians of genuine substance and experience, with an average age of 58. They included at least three major writers and intellectuals (Jenkins, Foot and Crosland; four if you count Benn and his diaries), and all of them had a significant 'hinterland' outside of politics, a phrase popularised by Healey. By contrast, when Gordon Brown resigned as Labour leader in 2010 the candidates who lined up to replace him were Ed Miliband, David Miliband, Ed Balls, Andy Burnham and Diane Abbott – all career politicians with little experience of professional life outside of politics, and with an average age of 45. It is a more diverse list than that of '76, but only because of Abbott, the candidate who came last. Without her, the average age of the four white, male, middle-class rivals was just 42. They are all serious men and serious politicians, of course, but it seems fair to say that John Campbell, the recent biographer of Roy Jenkins, is not going to be writing a life of Andy Burnham any time soon.

Are our politicians really getting smaller, or is it that politics is not as big as it used to be? We need to be careful about assuming that the present era is unique in its view that the big beasts have all gone missing. During the 1920s and 1930s it was also commonplace to complain that politics had been taken over by placemen and apparatchiks at the expense of the politicians of true substance. After all, theirs was only a generation away from the age of Gladstone and Salisbury: who was Stanley Baldwin to compare with that? The dominant political figure of the time, Lloyd George, had been frozen out by men he would once have had for breakfast. One of the running jokes in Evelyn Waugh's Vile Bodies, published in 1930, is that the bright young things can no longer remember who is prime minister because the various candidates are so hard to tell apart. Is it Sir James Brown or is it the Rt Hon Mr Outrage? And does it really matter when both men are such glaring mediocrities? As Miss Runcible says when she wakes up one morning in 10 Downing Street, having no idea where she is, 'It really all is too bogus.'

Mavericks, managers and party machines

Yet the 1930s were hardly an age of small political issues, any more than ours is. Like now, it was a time of economic austerity at home and proliferating crises abroad. It was also an era of coalition politics, which tends to blur the lines between the political parties while reinforcing the impression of a divide between the entire political class and the rest of us. (In Vile Bodies, the ex-king of Ruritania bemoans the fact that whenever he comes to England, 'Always there is a different Prime Minister and no one knows which is which.' He is told, 'Oh, sir, that's because of the Liberal Party.') Coalition politics and shifting political allegiances facilitate the rise of the party managers – the politicians who know how to cobble a deal together and make it stick. It also creates space for mavericks and outsiders to rail against the entire system of compromise and fudge. Then as now, some of the best-known political figures were on the fringes of the main parties, carving out a distinctive space for themselves with their disdain for the political operators. Where the 1930s had Winston Churchill and Oswald Mosley, we have Boris Johnson and Nigel Farage. All subsequent contrarians have probably imagined themselves as Churchill: the naysayer who turned out to be the ultimate politician of substance. But Churchill is very much the exception, not the rule. Doubtless Johnson would like to envisage himself as Churchill to Farage's Mosley, but Mosley represents the far likelier model for both: noise over substance, and a flash-in-the-pan rather than a game-changer. Waiting for Churchill is a futile political pastime.

Johnson illustrates another feature of eras of managerial politics: a smaller setting often provides the best backdrop for projecting a big political personality. Like Ken Livingstone before him, Johnson has used the London mayoralty, notwithstanding its very limited powers, as a platform for what passes as thinking outside the box. It is easier to convey the impression of a fresh vision when you are railing against political constraints than when you are trying to operate within them. Regardless of the outcome on 18 September, Alex Salmond has proved himself the master of this particular game. He stands out from the crowd of his fellow British politicians because he offers the possibility of a new kind of politics, and indeed polity, even if it may not turn out to be workable and popular in practice (if it does then he will truly have proved himself the master politician of our time). The largest political figures of the current age are often the ones operating where the real power isn't, which gives them the room they need to flex their muscles. Sometimes, when the pictures get smaller, the stars do get bigger.

The most significant difference between the present and the 1930s, however, relates not to its political personalities but to the institutions that underpin them. What has unquestionably shrunk in recent decades is the size, scope and reach of political parties. Baldwin might have been a relatively unassuming man but no one could have mistaken him for a political lightweight. He was the boss of a political party that took a huge amount of managing: the Tory party in the 1930s was itself a large, fractious and powerful beast. Equally, to master the House of Commons, as Baldwin did, was no small task. It took enormous political skill and experience, because the Commons had a clear sense of its own power to make and break governments. This was the compact that Max Weber identified as lying at the heart of modern democratic politics, if it was to work. The task of mastering mass political parties and powerful representative institutions was the guarantor of substantive political leadership. Political parties were formidable machines, which meant that no one could control them without possessing an equivalently formidable political skillset. Weber believed that it was impossible to rise to the top of a political system such as Britain's without having the leadership qualities needed to transcend it. Without those qualities, the system would swallow you up.

It is this compact that has now been broken. Political parties are no longer the formidable machines they once were: they are thin, fragile, hollowed-out institutions, lacking in members (they are no longer in any meaningful sense 'mass' parties) and often lacking a sense of purpose. They are shells of their former selves. While the same cannot exactly be said of the House of Commons, it too lacks much of its past heft. Consequently, the skillset needed to manage these institutions is not what it once was, either: it too is narrower. Our managerial politics lacks the element of institutional mastery that was a requirement of the previous age, because the institutions themselves are less substantial. This is what breeds an ever narrower political class, since connections within that class are now more important than the ability to coordinate large and often conflicting political interests beyond it. Politicians start younger and rise quicker now that small networks carry more weight than broader coalitions. The incentive to acquire wider experience of both politics and the world is absent because it is no longer so necessary. Experience can be trumped by insider knowledge, which is easier to acquire.

'Not just a politician': a retreat from the hinterlands

What has also been broken is the link between a career in politics and many of the professions and institutions that traditionally provided a training ground for political life. Trade unions, newspapers and the armed forces all once produced a steady supply of political recruits. Weber thought that these were among the best ways to learn something about politics before doing politics. He believed that varied experience of extra-political struggles was crucial in bringing substance to the political process. But these institutions too are now greatly diminished in scale and scope. Public relations and breakfast television might have filled some of the gap, but it's not the same. Traditional newspapers, like mass political parties, used to have to try to cover the waterfront; struggling to become a presenter on GMTV can be just as cut-throat, but it doesn't offer the same variety of perspective. The one profession that still supplies a steady stream of politicians is the law, which has always had the greatest overlap with politics. For that reason it does not really answer the charge that the outlook of the political class has narrowed: 'He's not just a politician, he's a lawyer too' is not the most resounding defence of any politician's breadth of vision. None of the Labour leadership candidates in '76 was a lawyer.

Just as politicians start younger, so they also seem to get out earlier. What appears to have gone missing are the politicians who move in and out of prominence in the context of a full-time political career, waiting for their chance to seize the moment. Clarke, to his credit, is hanging around because he wants to continue the fight over Europe into the next parliament and any possible referendum. But Hague will be gone, like Michael Portillo before him, seemingly burned out by having done politics since he was a teenager. A lifetime of cosy BBC documentaries awaits. Labour's two most experienced and popular politicians of the moment are Alistair Darling and Alan Johnson, both of whom have revealed an extensive hinterland since taking a step back from high office. Nevertheless, it's hard to envisage a route back to the top for either of them. One of the reasons why no current politician can possibly be another Churchill is that Churchill only revealed what he was made of near the end of an epically long political life, after he had done everything else and failed at quite a lot of it. He did not seize his moment at the first opportunity (he failed at that too). The other means by which Weber believed it was possible to learn how to do politics was to fail at it and to come back for more. Within a narrowed political class, with fewer entry- and exit-points, failure becomes less of an enabler and more like drink for an alcoholic: one is too many and a hundred is not enough.

These are some of the reasons why our politicians seem to lack stature when compared to those who went before. Yet it seems absurd to say that they are therefore bound to be lesser human beings. They still have to be smart, ruthless, adaptable and adept at a wide variety of tasks. The skillset has narrowed in many ways, but in others it has broadened. Present-day politicians require the stamina to survive a remorseless 24-hour news cycle. They don't have to control mass-membership political parties, but they do have to master the full panoply of modern communications technologies, most of which are primed to trip them up. Modern news management is not as appealing to outsiders as grand oratory, but it is just as demanding for the people who have to practice it. Politicians need to be as tough as they have ever been, and in some respects even tougher. One reason the political class has narrowed is that professional politics is less fun than it used to be. It's a lot more like hard work.

Falling standards or outdated expectations?

The present round of complaints about falling political standards is a bit like the regular lament about declining educational standards. Around this time of year it has now become traditional for employers to complain that whatever the story of endlessly rising exam results (something that Michael Gove, for all his efforts, has barely put a dent in), highly qualified school-leavers and graduates are often barely competent in many basic tasks. They lack the breadth and the vision of earlier generations, who had been educated in the round. In fact, most of the evidence suggests that young people are smarter, broader-minded and better informed than they have ever been. With the informational resources available to them, it is hard to imagine how they could be otherwise. But, as their world has expanded, the tests we subject them to have narrowed. Exams are more formulaic and more prosaic than they used to be, which makes them easier to pass for anyone who wants to put in the effort required.

Politics is the same. Our politicians have a wider and better informed view of the world than any previous generation. With the resources available to them, how could it be otherwise? But the tests we require of them – electoral, presentational, managerial – are increasingly narrow. The reason we remain so attached to those tests is that we fear that without them we will lose what few reliable standards of accountability and authority remain. As in education, the challenge in politics is to find ways to test for aptitudes that better reflect the variety of 21st century experience without appearing to allow the candidates to set their own standards. No one has worked out how to do it yet.

At the same time, we need to be careful what we wish for. It is worth remembering that the stellar cast-list of Labour leadership candidates in 1976 contained many remarkable men, but no ultimately successful politicians. The parliamentary Labour party chose perhaps the least stellar of them – Callaghan – to be its leader, and it almost certainly chose wisely. He had the most limited hinterland, but he was the shrewdest politician and he made a pretty successful prime minster, at least for a time. But even he was seen off by Margaret Thatcher, a politician who famously had almost no hinterland at all (that was what provoked Healey to coin the phrase). Intellectual breadth and literary abilities are no guarantor of anything in politics – Obama is proof of that. There are alternative role models out there – it would be hard, for instance, to accuse Putin or Modi of lacking political heft. In a time of managerial politics, it can be tempting to look with envy at the decisiveness of the political strongmen as they throw their weight around. But the experience of the 1930s should be sufficient to warn against succumbing to that temptation.

In this respect, our age may be no different than any other. The most successful democratic politicians are rarely the most impressive human beings. They are often simply the ones with the greatest staying power, or the ones who happened to find themselves in the right place at the right time. It's true that Roy Jenkins was a very impressive home secretary in the late 1960s, and Ken Clarke was likewise a strikingly adept chancellor in the mid-1990s, but these may have had as much to do with propitious circumstances as with any great political skills: it is possible that many different politicians could have made a success of those jobs at those times. The two most successful leaders in contemporary Western politics are Angela Merkel and Stephen Harper, neither of whom is famous for having an extensive hinterland (Merkel likes football, Harper likes ice hockey). Both have been subject to widespread disdain for their perceived lack of broader political vision, but both have ridden out the mockery. As Baldwin knew, if you are still standing when the others have fallen away, you are more than halfway to winning the battle of ideas. On that basis it would be rash to claim that there are no politicians of substance among the current generation. Who knows which of them will still be standing in 10 or 15 years' time – it could be George Osborne, it could be Ed Miliband, though it's unlikely to be both. (It is also unlikely to be Cameron, if only because he appears to lack that kind of staying power.) Whether it's Osborne or Miliband, neither of them could then be said to lack real substance. They would have passed the one political test that always matters.

David Runciman is professor of politics and a fellow of Trinity College, University of Cambridge. He is the author of The Confidence Trap: A History of Democracy in Crisis from World War I to the Present, published by Princeton University Press.

This article appears in edition 21.2 of Juncture, IPPR's quarterly journal of politics and ideas.