“NYAS commissoined OOC to develop a bespoke outcomes framework that would evidence the positive outcomes of advocacy intervention for children and young people.
OOC ensured that professionals, staff and young people were directly involved in designing the framework and led us through the complex process of validation. OOC has been instrumental in helping us to collectively capture data and information highlighting the issues for children and young people in care. As a result, we have evidenced the difference we make and have been able to present these findings to Commissioners and Governments, helping to influence change to protect and promote children’s rights”
Sharon Lovell, Director, National Youth Advocacy Service
The challenge
Public services must demonstrate the outcomes of commissioned services in children’s social care. To meet this evaluation need, NYAS required an evidence-based framework, including measurement tools, in order to:
· Prove that NYAS is focused on the difference it makes to people’s lives
· Provide evidence for improving services
· Keep NYAS relevant in an increasingly competitive marketplace
Our solution
The development of an evidence-led outcome framework involved several different stages; the initial stage consisted of a scoping review of evidence associated with evaluating the impact of advocacy and the identification of ‘constructs associated with advocacy’. The evidence review sought to define advocacy, as in line with NYAS and the wider sector, as well as clarifying the term in regard to the law, and demonstrating its impact within social care. The review sought to evidence advocacy in regard to qualitative and quantitative data, in order to form an evidence base for the outcome framework.
The next stage of the project involved the development of a logic model, based on the service that NYAS provides, which was further underpinned by existing evidence. Upon completion of the logic model, impact measures, based on the evidence review and the service NYAS provides, were designed. This involved the creation of domains relevant to the service and a questionnaire, which is to be utilised as part of the impact measure.
Finally, initial validation of the measure was undertaken as the framework was piloted to five of the NYAS regional offices. The data from the pilot exercise were analysed in order to test the quality of the questionnaires to ensure that they pass all the standard tests of reliability, validity and sensitivity.
The results
Analysis carried out, based on the pilot exercise, was used to identify change within the three core domains of the framework; knowledge, participation and empowerment, in order to evaluate the impact of advocacy within the service that NYAS provides. The finished measures have been assessed as having both face and content validity. Statistical analysis has confirmed that both questionnaires are reliable with an acceptable Cronbach Alpha score for the dichotomous questionnaire and a good score for the Likert questionnaire. However, the sample size is not adequate to have a high confidence level attached to these scores. Although the questionnaires require further validation, they have still undergone a more rigorous development process than many other questionnaires used in social care. It provides a solid foundation on which to build on and develop; as explained at the beginning of the development work, questionnaires and outcome frameworks in general are an iterative process. Until sufficient robust evaluations have taken place with sufficient sample sizes, outcome frameworks and associated questionnaires are always a work in progress and require periodic review.
Results from the pilot indicate that there were statistically significant improvements for each domain post-advocacy. The largest positive differences were evidenced in the domains of knowledge and participation, arguably the most important domains in terms of the core remit of NYAS. Although children and young people had an improved understanding of their rights, and feel that the opportunities for participation have increased, their belief in their own empowerment has not improved to the same degree. However, it must be noted that empowering children requires a whole system change for vulnerable children and young people and whilst NYAS have a key role in this, other organisations that work with these children and young people must also evaluate how inclusive and child centred their processes are and improve where necessary. It is hoped that by NYAS taking the lead in capturing this information, where concerns about knowledge, participation and empowerment are identified, organisations that contribute towards this will also seek to improve their services. Finally, the pilot exercise evidenced that children in both age groups were overwhelmingly happy about the service they had received from NYAS.