Participant feedback is generally pegged as the “softer” leg of nonprofit program measurement compared to quantitative approaches like randomized controlled trials (RCTs). Organizations can view input from users of social products and services more as a “suggestion box” than a critical measure of effectiveness. Meanwhile, the field has long considered third-party evaluations that relegate participants to subjects of a study a gold standard for developing evidence of program outcomes.
However, several new research initiatives aim to show funders and nonprofits that participant feedback has empirical links to hard outcomes. The early returns confirm that organizations that gather feedback from direct participants and their communities to continuously improve their programs and policies are finding that surveys, interviews, and focus groups can do more than surface new ways to interpret quantitative findings and explain the why and how. They can also highlight causal links to past outcomes and, remarkably, provide a proxy for future outcomes.
Two Case Studies
The Center for Employment Opportunities (CEO), a criminal justice organization that aims to move paroled men and women into livelihoods with higher job retention and to lower recidivism, offers an example. During its first four decades, CEO (founded in the 1970s as the Vera Institute of Justice) built a reputation for achieving hard measures of success; the data it gathered about program participants showed that the organization meaningfully reduced recidivism for people recently released from prison compared to formerly paroled people who did not participate in its programs.
Are you enjoying this article? Read more like this, plus SSIR's full archive of content, when you subscribe.
But in 2016, after encounters with David Bonbright of Keystone Accountability, a proponent of constituent voice, and a feedback initiative grant from Fund for Shared Insight, CEO shifted its approach from only gathering data about participants to also gathering data from them. As a result of participant input, it formed partnerships with external agencies to expand training opportunities, offering OSHA construction certification and commercial driver licensing. It also shifted certain administrative tasks, such as signing up for road crew shifts, from participants to staff so that participants could focus on attending interviews for more-permanent jobs.
After seeing how feedback sparked program improvements, CEO began to include participants at higher levels of decision-making: For example, it appointed people who completed the program to advisory committees across the United States in 2019, and to its board of directors in 2020. In 2021, with a research grant from the Fund for Shared Insight, it began investigating how participant feedback on program quality connected to hard outcomes like obtaining permanent jobs. Among other findings, CEO learned that participants who responded to requests for feedback during the first four weeks of the program were more likely to meet the organization’s job search and placement goals three months and six months later. The act of giving feedback itself became a predictor of outcomes.
Another example is the nonprofit Pace Center for Girls (Pace), an education and social service initiative founded in 1985 for girls likely to have experienced risk factors associated with delinquency and adverse childhood experiences. The organization uses a set of gender-responsive, trauma-informed, and strength-based prevention and early intervention practices to improve girls' self-efficacy and advocacy skills, as well as their relationships with peers, family members, and adults in their communities.
Before incorporating feedback into its measurement system, Pace used standard data like grades and school attendance to help girls identify their needs and track their progress. But while this enabled the organization to gauge academic outcomes it didn’t clearly show things like how program relationships and experiences influenced participants’ outcomes; how teacher turnover impacted girls’ experiences of services; or how that impact could ripple out and affect their grades, attendance, and engagement. In 2016 Pace decided to shift away from a compliance-driven approach and focus on building more-holistic program evidence. To do that it began gathering participant feedback, interrogating it with program participants, and letting them know what changes their input sparked.
Through surveys, focus groups, and participatory action research, Pace participants began to reveal how different groups (segmented by race, age, or length of time in the program) experienced the program. In 2016, surveys and use of the Net Promoter Score System (which assesses how likely Pace girls are to recommend the program to others), revealed that girls of different races experienced the program differently and had differing levels of satisfaction with services. The organization responded to these girls’ suggestions for enhancing processes like trauma counseling and committed to ongoing listening. Five years later, girls of all races gave Pace virtually the same feedback, showing high likelihood that they would recommend Pace to others and bringing the experience of the least satisfied level with those of the most.
Blended Measures
Even evaluation shops with reputations for large-sample, experimental, or quasi-experimental evaluations are beginning to take a more-holistic approach and incorporate feedback methods. For example, Jennifer Laird, program director at RTI International's Center for Research, Evaluation and Equity in Education, uses a measurement approach that dignifies participants and delivers quick, actionable insights by blending participant feedback with empirical studies. Her work constitutes a valuable model for funders and nonprofits looking to build more equitable evidence.
Laird used the approach recently for an impact assessment of a Californian social enterprise. The enterprise hired previously incarcerated people to train and work in its shops before entering the competitive job market. The idea was to see whether feedback data could be an early proxy for subsequent outcomes from an RCT, as a less-costly option for organizations that couldn’t afford an RCT.
Laird introduced a feedback component as a simple survey to the broader empirical study, and then tested correlations to see if the data on participants’ perceptions a third of the way through the program could predict whether they found a livelihood 18 months after starting. After doing factor analysis, she found associations between early participant feedback and exiting the program for a competitive job or school. Two associations rose to the top. One was a confidence measure: If participants were fearful about leaving their supportive work environment, they did less well. The second showed that having a strong connection to a staff member boosted successful job placement. (Indeed, RTI International’s results ultimately revealed the strongest feedback predictor for the longer-term outcome was participants feeling that staff treated them with respect.)
Thanks to participants’ early feedback, the social enterprise didn’t have to wait for 18 months to make improvements. To address the confidence gap, it brought back alumni to speak to participants’ fears and explain how their training had prepared them to move on. To strengthen connections to staff, it looked for ways to increase staff retention and foster strong interactions with job coaches on staff. “There was total value in undertaking both the RCT and the perceptual feedback,” says Laird. “I don’t consider an RCT the gold standard now, because there are things you can’t learn from them. Participatory evaluation is really important, and you can merge the two.”
Supporting the Use of Feedback
Nonprofits like CEO and Pace, and evaluation shops like RTI International, have discovered that participant feedback can point to quick wins for improving services and build more-equitable evidence. In particular, it can help organizations see how participants’ experiences and the organizational processes that shape them influence outcomes—insights they might miss by focusing entirely on monitoring and compliance. Following are some practices that can help organizations effectively use participant feedback, with examples from Pace.
- Involve program participants and other stakeholders in research and evaluation to ensure that you are asking the right questions. In 2018, for example, Pace collaborated with the Broward County Girls Coordinating Council in Florida to conduct participatory action research that addressed girls’ involvement in the juvenile justice system. The council, composed of Pace girls and adults from both Pace staff and the surrounding community, identified “failure to appear in court” (FTA) as a major cause for girls’ detention in Broward. The council gathered information to understand the causes of FTA; championed youth-friendly solutions, including help lines and transit tips to help girls make their court dates; and created community awareness around the issue. The following year, Broward County witnessed a 27 percent reduction in the number of girls detained for failure to appear in court.
- Blend qualitative and quantitative feedback to strengthen your evidence base and make better decisions. Pace continues to collect quantitative data on grades and attendance, but now it also conducts surveys and focus groups with girls, families, and team members to parse meaning. The organization uses survey data to identify the “what”—the specific issues impacting participants’ experience—and to guide focus groups and interviews that lead to a deeper understanding of the “why.” For example, survey information suggested a link between staff engagement and a participant’s positive program experiences. Focus groups and interviews that followed helped Pace identify specific actions and mindsets staff could take to nurture positive program experiences, and to modify the organization’s approach to influencing girls’ behavior. The results show in hard measures like grades and attendance.
- Adopt a systemic approach to understand organizational performance and culture. Applying a blended approach across an organization allows it to see patterns and relationships between participants' reported experience and organizational processes and functions like training or mentoring. It can also help determine how to reinforce positive cultural attributes, such as openness and transparency. Pace correlates participants’ feedback, gathered by program leadership, with employee satisfaction surveys, which allows it to interpolate multiple perspectives into decisions that impact organizational culture. For example, participant feedback highlighting the importance of positive and lasting teacher relationships informed both program measurement and a human resources compensation assessment that led to enhanced teacher benefits.
- Establish operating strategies that include listening to and acting on participant feedback so that all decision-making—including hiring and investing in technologies—benefits from participant views and empowers participants themselves. For example, Pace has Girls Leadership Councils at each of its sites comprised of girls who want to work on their leadership and advocacy skills. The council provides a safe space where girls can discuss issues they care about, give feedback on the organization’s approaches to influencing girls’ behavior, and suggest activities to promote retention in the program.
Valuing Qualitative Feedback
Connecting participant feedback to program outcomes is part of a nascent social sector movement. In the past few years, funders such as the 10 foundations co-invested in Fund for Shared Insight have given grants to help several nonprofits, including CEO and Pace, conduct research to that end. In addition, equity-focused evaluation intermediaries that care about participant voices have sprung up. For example, We All Count is committed to bringing equity to data science, the Equitable Evaluation Initiative now offers a framework for shifting practice, and Project Evident is helping nonprofits derive lessons from questions their participants and social sector leaders are asking.
What’s certain is that any nonprofit leader must understand the conditions that make their intervention work. Seeking participants’ insights at every step of community research, evaluation design, and delivery is the best way to build that evidence equitably, and create timely means to predict, aid, and interpret outcomes.
Support SSIR’s coverage of cross-sector solutions to global challenges.
Help us further the reach of innovative ideas. Donate today.
Read more stories by Lymari Benitez, Yessica Cancel, Mary Marx & Katie Smith Milway.