Julian Corner surveys a wide and complex network of voluntary organisations and societal actors that has backed itself into an unsustainable position, and calls for a radical reassessment of their fundamental structures and objectives.

This essay is based on the Involve Yorkshire and Humber Annual Lecture delivered on 18 November 2014.

How do we scale our solutions so that they match the scale and nature of the problems we face as a society? The current austerity has made this challenge both socially and economically urgent and more complex than ever. The complexity is such that anyone who presumes to offer 'an answer', especially an answer that might operate at scale, should probably not be trusted. The idea that solutions to deep social dysfunctions can simply be designed and rolled out now seems at best naive and at worst intellectually corrupt.

I'm afraid that philanthropy, as well as government, is highly culpable in encouraging our search for the scalable solution. It has instilled a belief that in the field of social care there can be such a thing as 'expertise', which seems to be a paradigm we have imported from the physical into the social sciences without very much basis.

Crucially, our obsession with solutions and expertise has distracted us from the most plausible response to the challenges of scale and complexity, namely collaboration. How else can we problem solve, bring our collective resources to bear, and inject diverse challenge into our thinking, if not through collaboration?

Collaborative leadership then is how I want to frame the response to the challenge of scale and complexity in a time of austerity. In exploring a concept as apparently benign as collaboration, I will suggest that its implications for the voluntary sector in particular are far from comfortable or straightforward.

Judy Robinson, in a recent essay, captures this well:

'The voluntary sector itself and its leadership will have to face up to whether we can articulate better some sense of being one sector with common roots and values linked to social solidarity.'

'Small is beautiful and it needs more recognition and support', in Slocock C (ed) Making Good: The Future of the Voluntary Sector, Civil Exchange [PDF]

As a shorthand for the enormity of the challenge we face, I am persuaded by the parallels that are drawn regularly between our social and our economic and environmental models. Just as the latter two have both proven themselves in recent years to be unsustainable and in need of fundamental change, so the changes required in our social model seem no less radical or urgent.

I don't mean that our society is broken. I mean that the model we have created to respond to social challenges has run out of road. The most eloquent testimony to this recently was the BBC Panorama programme on the death of Peter Connelly in Haringey. Few watching that documentary could fail to conclude that we have got ourselves into a position that is unsustainable.

I was particularly struck by Sharon Shoesmith's recollection of standing on the fifth floor of the children's department, looking out over the rooftops of Haringey. She describes herself as hoping that somewhere out there her social workers were making the right judgments that day. I found that an extremely poignant image of the isolated and almost impotent public official holding an intolerable burden.

Throughout the documentary, we were repeatedly reminded that of the children who die at the hands of a parent each year, only one in 10 is known to social services. So nine in 10 of these children are not spotted by the systems most of us agree are already overstretched. What this tells us, surely, is that the expectations we place on statutory services to solve entrenched social harms are wholly unrealistic. Unrealistic because these harms run so deep, and unrealistic because the reach of statutory services is in truth so limited. And it is worth remembering that Peter Connelly died well before the austerity started to eat away at services' capacity.

Returning to the image of the isolated public servant, I wonder what we would like Sharon Shoesmith to have been thinking as she looked out over the Haringey rooftops. I would like her to have taken more comfort from a sense of a common purpose between her staff, those of other departments, voluntary agencies and above all the community itself.

Of course, she would still have needed her staff to have made good judgments, but as part of a collaborative endeavour, building on the kind of community solidarity that Judy Robinson describes. I would like her to have felt more confidence that child protection was the collective responsibility of a strong and supportive community, within which her staff played a vital but not unrealistic role. I would like her to have felt that she wasn't carrying this for everyone else.

The Panorama documentary painted the most devastating picture of public vitriol directed at Haringey social workers, of a media witch-hunt, of our politicians scrambling to apportion blame, and of partner agencies leaving social services to shoulder responsibility entirely alone. It became a nightmarish example of what happens when a society loses sight of the values that hold it together: mutuality, tolerance, reciprocity, kinship. And it occurred to me that nowhere in the programme did we hear from or about the Haringey community itself, from the civic society that surrounded Peter Connelly. It was an almost dystopian vision of a society in which there are only individuals, a state and a media.

The role of the voluntary sector

The role that the voluntary sector needs to play in helping to build a more collaborative society has never been more important. The erosion of respect for our political, economic and cultural institutions has left a nation that needs desperately to unify around a set of values which form a positive expression of who we are.

These values cannot be handed down to us from our national leaders, especially if they are cloaked in patriotism. These are values that need to be negotiated and enacted in our civic society, as part of a diverse and localised democratic process.

Voluntary organisations can help create to spaces and platforms where this participatory process can play out. And the beauty of voluntary organisations is that they allow people to come together and collaborate around values without necessarily having to articulate them.

I want to argue, however, that there are tendencies and traits within the voluntary sector that militate against this crucial role, traits that we need to watch for and guard against. If voluntary organisations are going to be the agents of change we desperately need them to be, we cannot take off the table that some of them may need to change.

The organisational challenge

I will start with perhaps the most challenging tendency or trait. A large proportion of voluntary organisations in the social sphere come into being in order to address an unmet need, often because there is a gap in existing service provision. The charitable organisation is then formally constituted around a commitment to fill that gap. The challenge here is that gaps in service delivery are not static. They tend to arise out of the relationship between a number of variables at play in a local area. These change over time, but organisational forms tend to be pretty rigid. This is because it takes huge effort to establish and maintain the legal, financial and human structures of an organisation.

These efforts inevitably pull the attention and energies of those involved inwards. In the meantime, the world around the organisation changes. Needs change, other organisations change; the financial climate, politics, legal structures – they all change. This is inevitable. And so considerable agility is required of the organisation as its task becomes increasingly complex and onerous.

It is often said in public discourse that small voluntary organisations are fleet of foot and can turn on a six-pence. That may be true of some, but I'm not sure there is much evidence for this in the majority of cases. Many small voluntary organisations really struggle to evolve and adapt. This is not their fault, but it is certainly a weakness of the organisational model that many choose to adopt.

It is not only legal and financial structures that create this rigidity. Voluntary-sector leaders find themselves on deep cultural tramlines which dictate that a successful organisation is one that survives and even grows. We talk continually about organisational sustainability as if this was the overriding objective, and we measure people's careers on the growth they have achieved in organisational turnover.

These are pull factors that can be very hard to resist. But it might just be time to question whether the organisational form that we have come to expect of many voluntary organisations always suits the needs of our fast-moving and complex society.

I have been struck recently by the example of MAC-UK, which Charlie Howard set up seven years ago to promote a mental wellbeing response to gang culture and violence. Their work is award-winning and widely celebrated, but what is really striking is that MAC-UK has only three years to go until it closes in its current form. Howard set it up with a 10-year timeframe because she didn't want the organisation's needs to distract from its mission, values and purpose. This has caused MAC-UK to work in an entirely different way, collaborating with statutory services to embed its ideas into their core practices – and they in turn have had to adapt themselves to accommodate very different ways of working.

Imagine the alternative. MAC-UK develops and runs its own practice; it wins contracts from local commissioners who are only too willing to transfer risk out to the voluntary sector; the environment changes and the commissioners start to add further demands into the contracts, demands that aren't in line with MAC-UK's own practice and learning; some pots of money dry up, so they have to start compromising on quality and staffing. And before you know it, MAC-UK is expending most of its energy keeping the show on the road, with waning passion for the ideas and beliefs that founded it.

It takes courageous leadership to impose a shelf-life on the organisation you have created and run. In the case of MAC-UK, Charlie Howard has already stepped aside from the day-to-day running to develop a radically different organisational form that can carry forward the next iteration of the journey. Notably, her response to the challenge of scale is to try to build a movement around MAC-UK's ideas. For me, her example shows that mission and values need not be bound by organisations: they can be portable and transferred to new organisational forms as the ideas and the environment evolve.

Our default response to a gap in our social fabric is to create an organisation. This might be considered ironic, given that unmet needs and gaps in services often arise out of organisations failing to evolve and adapt. In other words, we respond to the problem with the same logic that created it in the first place: the organisational form. But this way, what is to stop the new organisation becoming the problem that needs to be solved in five to 10 years' time?

Perhaps this reflects an inevitable process of evolution, in which organisations emerge in response to the limitations found in others. But in that case, we cannot maintain the line that everything must survive. We surely need greater fluidity than that.

Movements versus organisations

One question this suggests for collaborative leaders is whether they want to lead organisations or movements. LankellyChase is supporting Civic Systems Lab to develop new platforms which can seed, develop and host citizen-led and highly localised projects. A tech parallel might be the way smartphones created a platform that gave rise to a dizzying array of apps. Civic Systems Lab is collaborating with local authorities and others to work out how systems can be reframed so that they encourage mass participation in the local economy and community, enabling people to develop and prototype micro-ideas whose collective impact could be huge.

Part of what this model is trying to overcome is the inherent exclusiveness of the constituted organisational form that can stop people collaborating and participating on their own terms. The organisation is reinterpreted as an open system that draws in and helps to build the capabilities of local citizens. It focuses on strengthening the community, so that the community can start to overcome some of its own challenges.

I appreciate entirely that our society needs voluntary organisations that are trusted and hold deep community learning. LankellyChase supports WomenCentre in Calderdale and Barca in Leeds, both of which are multipurpose organisations that start with people and community and then build their service offer around them. A warning light always starts flashing in my head when voluntary organisations talk about 'going into' communities. WomenCentre and Barca are of the community, they come from the community, and this gives them the courage to change and adapt their organisation as the needs of their community evolve. They are good examples of 'form follows function' organisations, able to model and therefore influence the changes needed more widely.

Symptoms versus structural causes

This brings me to another trait or tendency. Almost inevitably the passion of voluntary organisations for addressing gaps in support draws them towards urgent social problems such as drug misuse, homelessness, violence and mental distress. Stepping back from these problems, it is rarely noted that each is strongly associated with the same profile of structural root causes, most obviously intergenerational poverty and inequality.

Yes, the symptoms of poverty and inequality need to be addressed, but the risk is that each symptom competes for attention and resources, resulting in a clamour for sustainable organisational models that can cover all of these bases. What this distracts from is an appreciation of the deeper structural problems that generated these symptoms in the first place.

Symptom-led responses make it very hard to join up the dots. To take two random examples, drug treatment charities and child abuse charities have organised themselves into entirely separate subsectors, with scarcely any connection, even though both of these problems share many of the same structural root causes.

There is no doubt – because it is human nature – that vested interest exists across all sectors responding to social need. I visited a local authority recently within which the prosperous but mixed city had a large number of established voluntary organisations doing vital work, all looking to the council to sustain them. Meanwhile, the considerably poorer town elsewhere in the county had close to zero voluntary organisations working in it. Notably, the council was facing remarkably little pressure from the voluntary organisations to spread funding to benefit the population of the less-well-served town.

In effect, the demands of the incumbent organisations were playing their part in maintaining an iniquitous funding allocation. In a similar way, the focus of many voluntary organisations on individual symptoms surely helps to structure the political and delivery siloes that make it so hard to meet the needs of whole people, whole families and whole communities.

Building collaborative analysis

We have an unhealthy tendency in the voluntary sector to load responsibility and blame for these challenges on service commissioners, who have the unenviable role of squaring their understanding of the needs of a local community with the claims of the organisations who want to keep working there. There is no doubt that many commissioners need to become a great deal more collaborative and participatory in determining what those needs are. But we also have to recognise that it must be hard for them to collaborate with organisations which find it very difficult to see or think beyond the immediate symptom that defines their organisational model.

We desperately need spaces where diverse players, including commissioners, voluntary organisations, faith communities and local people, can come together, without their organisational hats, and collaborate around a first-principles analysis of what the community actually needs. This would require an equality of exchange that might only be possible if there is proper investment in independent infrastructure to support it.

This equality of exchange would also require a different quality of leadership. By this, I mean leadership that is capable of clearing organisational ego and imperatives out of the way, that is able to live with a much higher level of uncertainty, and is willing to enter challenging discussions without already knowing the answer.

At the heart of this, we have to be able to challenge the assumption implicit in our blanket term 'the voluntary sector' that all voluntary-sector activities are ideologically compatible. In fact, voluntary organisations hold quite different visions of the society they are trying to build. I would cite here the work we support of Leeds Gate on asset-based community development and of St Mary's Community Centre in Sheffield on appreciative enquiry and time-banking. These have a common focus on people's capabilities and skills, a focus that represents a real challenge to prevailing deficit-based and problem-oriented approaches. Yet in our current system, all of these approaches fight for the same money, on the same terms, as if they were equally valid.

We do not yet have the safe, equal and collaborative spaces where the tensions and differences between our various visions for society can be worked through and reconciled. And this might partly explain why we have known about asset-based approaches for so long and yet they struggle to take a meaningful hold on the way we think about services and support.

The value of systems thinking

An alternative methodology, and equally collaborative, is provided by 'systems thinking'. Typically, this mode has been used by statutory agencies to help them understand where the demand for their services comes from. But there is no reason why the voluntary sector shouldn't also lead such exercises.

We are currently supporting Advice UK to use systems thinking in Bradford to understand the nature of demand for advice services, which have been especially squeezed by the austerity. The idea is to use advice services – but it could be any other type of crisis service – to collect and analyse data on where glitches and gaps elsewhere in the system have led people to need advice. This is to understand what the systems thinker John Seddon memorably calls 'failure demand'. The point is to feed live data and analysis out into the rest of the system so that the gaps and glitches can be routinely addressed.

Imagine a complaints team in a department store which saw its role simply and only as successfully resolving complaints, and which didn't analyse complaints data and feed it back into the rest of the store. You would think they were missing a very obvious trick. And then imagine that the complaints team is actually a separate organisation whose business model depends on these complaints, and you start to see why organisations find it so difficult to forge genuine collaborations.

The complaints team must make itself quite unpopular, shining a light on the shortcomings of its colleagues, and of course the rest of the store might be only too happy for complaints to be channelled elsewhere. But if the business is going to run effectively, everyone has to see themselves as a part of one system of interdependent functions that are not competing, not blaming one another. They have to be united by a desire to serve. And everyone has to be clear that the ultimate goal is for the work of the complaints team to dry up completely.

And I would ask openly, where in the public and voluntary sector do we ever genuinely contemplate such a goal for our crisis services? It is, of course, only an ideal. Crises will always occur, just as people will always complain. But we are a long way from viewing demand for these services as a litmus test for the success of the rest of our strategy in other areas. The component parts are viewed as unrelated and they are funded as such.

The challenge of scale

This is probably the right moment to tackle the issue of scale, as much of what I've said implies there might be virtue in scaling back rather than scaling up. It is, of course, true that there are ideas and models that deserve to reach a much wider population. I'm not suggesting that everything can be achieved purely out of localised collaboration.

Yet for voluntary organisations, scale holds many pitfalls that we don't seem to be getting much better at avoiding. The chief pitfall is that voluntary organisations which push themselves to operate at scale are repeatedly captured by the very system they were set up to help improve. They run up against limitations such as siloed delivery, rigid eligibility thresholds, stigmatising targeting of unpopular groups, limiting access to services through dehumanising assessments, cherry-picking beneficiaries and gaming outcome figures. All of these problems have been transferred wholesale into the voluntary sector.

And there is another equally familiar problem with scale, namely that it kills the goose that laid the golden egg. There is an idea that practice can be codified and even commodified, such that it can be transplanted to new environments and reliably produce the same results. The arguments against this idea are well-rehearsed: most obviously, that it discounts the passion, talents and commitment of the original team who cannot themselves be 'scaled up'. And it discounts the history of how the idea evolved, how it sits alongside other services locally, how it has been supported by a number of informal and formal systems. In other words, it discounts people and context.

When we think about scale, we usually think about replicating the product, not reproducing the process. But if we want to scale something that is entirely determined by people and context then we need to focus our attention on the conditions of success that produced and supported the original model or idea in the first place. Models that have trusted relationships at their core cannot simply be sold in a transactional market. The process of scaling has itself to be a collaborative exercise between multiple players – providers, commissioners, local people – all of whom need to work together to reproduce the conditions of success.

Voluntary organisations often gravitate towards product – the 'evidence-based service' – and this certainly is important. But we also have to be custodians of process. What gives us this role is our freedom to be led by values in ways that simply aren't open to public sector organisations. This doesn't mean that public-sector staff are any less values-driven. It simply means that independent organisations do not face the same constraints, and therefore have the opportunity to follow through the radical logic of their values.

Voluntary organisations that compromise on their values give away their power. They lose the opportunity to find out what kind of a society their values might help to build. They become handy accomplices rather than exciting collaborators. On the contrary, we need people who can champion a vision of a society in which we live by our desire to do good by each other, to contribute, to be generous, to be neighbourly. In the short term, championing these values may well require sacrifice. But in the longer term, we have to believe that these values can prevail.

Building trust

At the heart of collaborative leadership is the challenge of scaling values, not organisations. The more we succumb to market-driven competition, the harder this will be. There is no right or wrong here, but there is a clear choice. If we want services that are highly relational, in which people are freed up to be really great with each other, then we are going to have to build a lot of trust.

As things stand, the fingerprints of the state are all over the support that we provide to each other. We have got ourselves into a place socially where it has become incredibly complicated for one human being to help another. We give these two people different names, we legislate for their respective roles and rights, we pay money to one and not the other, we praise or blame one depending on what the other does, we place conditions and time limits around their relationship.

I've argued that this isn't sustainable, but to get ourselves out of this bind will require a voluntary sector in which at least the majority of us can agree that our core role is to help build a more trusting society. If we want to operate beyond the limitations of statutory services then we need to collaborate around the goal of creating communities in which we trust ourselves, trust each other, and in turn are trusted to help one another.

To help build a new social model, the voluntary sector needs to be really clear that it is embodying and living the values that it wants to see in that social model, even where it costs organisations revenue, scale and perhaps even their existence. Many organisations already push themselves incredibly hard to do this, but I have argued that there are tendencies and traits within voluntary organisations that are not going to serve us well in the future if we are to be the agents of change that we need to be. And so I would characterise the core challenge for collaborative leadership in a time of austerity as needing the courage to be led radically by mission, values and purpose, and not by money.

I should finish by referencing the responsibility of my own part of the sector, trusts and foundations, which surely have a crucial role to play in backing those who show most courage, in funding process as well as product, in helping to build movements as well as organisations, and in funding the spaces where there can be equality of exchange.

All of which suggests that independent funders should look beyond the immediate pull of scalable service models and instead help to create headspace for collaborative leaders to think boldly at a time when financial pressures are in danger of paralysing them. Just as there has never been a greater need for voluntary organisations to bring about social change, so there has never been a greater need for independent funders to step up and take a lot more risks in pursuit of a shared vision.

Julian Corner is chief executive of the LankellyChase Foundation.