Seth Newman talks more about our recent review [opens in new window] of the Welsh Government’s Supporting People Programme (the Programme). He looks at the difficulties in trying to evaluate the Programme and other preventative programmes like it.

Supporting People [opens in new window], introduced in 2003, helps a wide range of vulnerable people by equipping them with the necessary skills and confidence required to live independently and thereby reduce demand on other public services, such as hospitals or residential care homes. The Welsh Government currently spends about £125 million per year on the Programme.

The last comprehensive review of the Programme in 2010 [opens in new window] concluded there was an urgent need for a more rigorous approach to evaluating Supporting People services. In response, in 2012 the Welsh Government introduced a new ‘outcomes framework’ to try to measure the improved levels of individual well-being brought about by the Programme. We highlighted in our report A Picture of Public Services [opens in new window] in 2015 how there is a growing interest in measuring ‘personal outcomes’, and illustrated how personal outcomes can be integrated within a national outcomes reporting framework.

However, our recent report on Supporting People also highlighted a number of limitations with the Welsh Government’s approach to evaluating Supporting People, which make it difficult to form a comprehensive judgement of the success of the Programme. The Welsh Government is planning to revise the outcomes framework and has recently completed a consultation [opens in new window] on its proposals to address some of the current concerns.

The Welsh Government also commissioned research [opens in new window] published in 2016 to explore the feasibility of using a method known as data linking [opens in new window] to evaluate the Programme. There is some emerging evidence from this work that people receiving Supporting People interventions show a greater decline in their use of health services when compared with a control group. However, the 2016 feasibility report also emphasised the need for a broader and more robust data set to support any definitive judgements about the impact of the Programme on demand for health services. The Welsh Government has subsequently commissioned further research over the next few years to improve data quality and look for evidence of the impact of Supporting People interventions on other areas of public services, such as social services

In addition, some local authorities have produced case study [opens in new window] examples that highlight potential cost savings to other statutory services because of Supporting People interventions. The case studies outline the various ways in which the Programme has helped people with mental health, alcohol and substance misuse problems, those experiencing domestic abuse and young people leaving care.

On its publication, our report generated a passionate response on social media from a diverse range of people affected by, or involved with, services that are funded by the Programme. Overwhelmingly, the online conversations demonstrated the belief that the Programme provides vital support services to vulnerable people.

Reviews by us and others have found that the Welsh Government has experienced similar challenges in demonstrating the impacts of some of its other preventative programmes. We carried out a value for money review [opens in new window] of the Communities First programme back in 2009. Despite finding clear examples of local benefits, the review concluded that the then Assembly Government could not demonstrate the overall impact of the programme in meeting its objectives. More recently, a process review [opens in new window] published in 2015 and a lessons learnt report [opens in new window], produced by the National Assembly for Wales’ Equality, Local Government and Communities Committee in July 2017, highlighted remaining weaknesses in the performance management and evaluation arrangements for Communities First.

The Welsh Government has also commissioned ‘Evaluability assessments’ [opens in new window] of its Flying Start and Families First programmes which have identified similar issues. The November 2016 [opens in new window] and May 2017 [opens in new window] assessments concluded that to date there is insufficient information to assess the effects of the two programmes fully, despite them operating for a number of years. The assessments recommended ways to evaluate the programmes more robustly in future.

However, we do not underestimate the inherent difficulties of evidencing the impacts of such preventative programmes on both individual lives and other public services, particularly the Supporting People programme, which is incredibly diverse in the range of people it helps and the services it provides. Preventative programmes, and indeed other publicly funded programmes, face the challenge of attributing any positive changes seen in individual outcomes to the influence of the programme, as often the individuals being supported are also be receiving help from a range of other public services. One approach to attributing causality is to carry out in-depth longitudinal and qualitative research, and the Welsh Government is not planning any such evaluation.

It is also notoriously difficult to prove the counterfactual. The case studies referred to above highlight the potential savings to other public services brought about as the result of a Supporting people intervention. However, in practice it is difficult to confirm the extent to which these costs would have been incurred. One approach to identifying what might have happened in the absence of the Programme is to establish a control group and test outcomes for programme users against outcomes for the control group; an approach which the Welsh Government is using in the data linking work described above. Work which has over time the potential to enable accost/benefit analysis of the Programme.

In response to these challenges, the Wales Audit Office is planning an event on this topic in Spring 2018 as part of its Good Practice Exchange programme. The exact remit of the event has yet to be determined but please keep an eye out for more details [opens in new window].

About the author 

Seth Newman joined the Wales Audit Office from the South Wales Police and Crime Commissioner’s office 18 months ago. He has previously worked for a number of local authorities in South Wales as well as the Welsh Local Government Association and Local Government Data Unit. His background is in Community Safety and Criminology.