Gauging Long-Term Impact in the Social Sector

Businesses readily measure long-term performance using any number of indicators, including, commonly, return on investment. For nonprofits, in contrast, there are no standard yardsticks, and long-term results are far more difficult to assess. Yet an accurate understanding of delivered long-term impact is becoming increasingly vital in this arena. Donors, seeking greater transparency, want to know precisely how their contributions are making a difference. Organizations operating on the ground want a better understanding of what works best and most efficiently so that they can optimize their efforts and program portfolios. Governments, communities, and other stakeholders—including targeted program participants—are also showing ever-increasing interest in long-term results.

Despite the growing premium on demonstrating long-term impact, though, few large international nongovernmental organizations (INGOs) today have an effective means of measuring it. Most organizations measure short-term indicators—such as the number of program participants or the number of interventions undertaken—which are relatively easy to track. Some organizations also measure their efforts’ direct short-term effects, such as improved grades in school. Most organizations, however, lack a rigorous, systematic approach to gauging long-term impact, whose measurement poses far greater challenges.

Such was the case until recently for SOS Children’s Villages, a leading INGO focused on improving the situation of children who are at risk of losing, or who have already lost, parental care. The organization reached more than 1 million children, young people, and adults ¹ 1 International Annual Report 2014, SOS Children’s Villages International, 2015. Notes: 1 International Annual Report 2014, SOS Children’s Villages International, 2015. across 134 countries in 2014 through the provision of care, education, health, and emergency relief services. At the core of this work is the empowerment of children and young people, as well as their families and communities, through family-strengthening programs and the provision of family-like care.

Although SOS Children’s Villages attaches considerable significance to the long-term impact of its efforts, it has been challenged historically to quantify precisely that impact. Working in partnership with The Boston Consulting Group (SOS Children’s Villages and BCG have worked together for several years, exchanging knowledge across the social sector), however, the organization recently developed a highly effective approach to impact assessment, one that comprehensively measures the long-term effects of its efforts and provides valuable insight into how SOS Children’s Villages might further improve the impact of its programs. Critically, much of this approach is transferable to other organizations operating in the social sector, including the ones focused on other target groups and issues.

Measuring Impact

SOS Children’s Villages strives to do more than simply maximize the number of lives it touches. It aims to ensure that the long-term impact of its work with children, their families, and their communities is as significant and sustainable as possible. The organization also seeks to continually improve its effectiveness.  

SOS Children’s Villages recognized that it needed a clear understanding of the long-term results of its efforts to achieve those goals. With this in mind, it sought to develop a robust approach and methodology for measuring long-term impact, focusing on its core work in family strengthening and family-like care. SOS Children’s Villages’ family-strengthening efforts are aimed at empowering families and communities to effectively care for and protect their children, thereby preventing the separation of children from their families. Through the provision of family-like care, the organization strives to provide quality care in a stable family setting for children who cannot live with their biological families.

The assessment methodology that SOS Children’s Villages developed jointly with BCG evaluates two elements of the programs’ long-term impact: the nonfinancial (or all-around) impact on the individual program participants and the community and the financial impact on society. The determination of the long-term impact on individual participants is based largely on information gleaned from interviews of former program participants by external researchers. This is supplemented by qualitative research conducted through focus group discussions with former child participants and their caregivers. The outcomes ² 2 Ratings are based on an adjusted version of the U.S. Agency for International Development’s Child Status Index. Notes: 2 Ratings are based on an adjusted version of the U.S. Agency for International Development’s Child Status Index. are assessed along the following eight dimensions:

• Livelihood. What is the former participant’s employment and income status, and does he or she have sufficient resources to assure survival and development?
• Care. Does the former participant have strong, stable relationships that provide ongoing social, emotional, physical, and material support?
• Food. Does the former participant have sufficient food?
• Shelter. Does the former participant live in adequate conditions in a stable, secure home?
• Education and Skills. Was the former participant well prepared for future employment or is he or she being well prepared? Preparation requires, for example, the availability of relevant education or training and the ability to regularly attend classes.
• Protection. Is the former participant safe from abuse, exploitation, and discrimination?
• Physical Health. Is the former participant in good health on the basis of age?
• Social and Emotional Well-Being. Is the former participant happy, and does he or she have a positive outlook on life?

Each dimension is tracked using a number of related indicators.

The programs’ long-term impact on the local community is similarly assessed using a variety of indicators. These include, for instance, the following:

• Community Awareness. Are local stakeholders aware that certain children are at risk, and do stakeholders have a clear understanding of how to improve conditions for these children?
• Community-Based Support Systems. Is the community putting in place mechanisms to support vulnerable children and address violations of children’s rights?
• Progress Toward Sustainability. Would activities related to the programs continue if SOS Children’s Villages withdrew from day-to-day involvement in the community?

This information is gathered through interviews and focus group discussions with critical stakeholders in the community and is supplemented by case studies and desktop research.

The programs’ long-term financial impact on society is gauged by the programs’ social return on investment (SROI), a comparison of the programs’ total costs and benefits to society. (See Exhibit 1.) The calculation of societal benefits is based on easily quantifiable elements ³ 3 The impact is quantified only for former program participants who are doing well in the livelihood and education and skills dimensions. Costs, however, are taken into account for all assessed former program participants. This leads to a conservative SROI figure. Notes: 3 The impact is quantified only for former program participants who are doing well in the livelihood and education and skills dimensions. Costs, however, are taken into account for all assessed former program participants. This leads to a conservative SROI figure. . The most important of these elements are the present value of former program participants’ projected lifetime incomes; the present value of projected long-term financial benefits to the children of former program participants; the programs’ direct financial benefits to the community, including the local spending of SOS Children’s Villages; and the savings in government social spending resulting from former program participants’ ability to support themselves and their families financially. The calculation is supplemented by a number of sensitivity tests to ensure that the picture that emerges is realistic.

Testing the Approach on the Ground

To date, SOS Children’s Villages has conducted and completed pilot projects in Ethiopia and Swaziland to test its impact-assessment approach. The results were highly encouraging in both cases. The approach proved feasible in the field. It also, despite the pilots’ limited sample size ⁴ 4 Each pilot program studied outcomes for 40 former program participants. Notes: 4 Each pilot program studied outcomes for 40 former program participants. , provided critical insight into the programs’ performance and long-term impact—essentially answering all of the questions that the organization had posed.

For SOS Children’s Villages, the news on its programs’ long-term impact was highly encouraging. Among the most salient findings was that a convincing majority of former program participants—almost 90 percent—were doing well on at least six of the eight dimensions considered. Moreover, local stakeholders indicated that they highly valued the organization’s work in their communities.

Further, the programs’ SROI was compelling, with a €1 investment yielding benefits to society of €4 in Ethiopia and €6 in Swaziland using conservative assumptions. (See Exhibit 2.) In fact, the programs’ true impact is even higher than these figures indicate, since the SROI calculation only includes easily quantifiable elements (and excludes such things as the multiplier effects of program participants’ increased incomes on society); valuable but indirect societal benefits, such as corresponding reductions in health care costs and crime rates, are not considered.

These findings galvanized SOS Children’s Villages, with management viewing the results as both motivation and a mandate for carrying its work forward. The findings also identified potential areas of future improvement for the organization in the two pilot countries. In Swaziland, for example, a higher-than-expected number of former program participants encountered difficulty securing jobs, despite the fact that they were well educated. In response, SOS Children’s Villages plans to focus more on vocational training to meet the requirements of the local labor market and increase participants’ employability.    Encouraged by the way its methodology performed in the pilots, SOS Children’s Villages plans to apply it to other locations in which the organization’s programs are operating. Said Richard Pichler, former CEO of SOS Children’s Villages, “We are certain that as we go further with this world-class approach, we will obtain learnings that will help us create a lasting impact for more and more children across the world.” The organization is currently conducting assessments in five additional locations—Cote d’Ivoire, Senegal, Togo, Tanzania, and Nepal—that are scheduled to be completed in 2016. The rollout to programs operating in other locations will be ongoing. The organization says it will use the results from these assessments to continue to increase its programs’ effectiveness and further optimize its program portfolio.

SOS Children’s Villages’ impact-assessment methodology gives the organization a unique and powerful lens for understanding the long-term effects of its programs. Although the methodology has elements tailored specifically to SOS Children’s Villages, its core elements are broadly applicable to other organizations operating in the social sector and could prove highly valuable to those organizations. Both SOS Children’s Villages and BCG seek to share our learnings in designing, implementing, and leveraging the approach with other organizations.