Closing the Evaluation Gap

(Illustration by Gracia Lam) 

Evaluation is an essential capacity for nonprofit organizational effectiveness. “Well-run organizations and effective programs are those that can demonstrate the achievement of results,” states a GuideStar article on the importance of evaluation. “Results are derived from good management. Good management is based on good decision-making. Good decision-making depends on good information. Good information requires good data and careful analysis of the data. These are all critical elements of evaluation.” This is why many foundations and individual donors require evaluations from grant applicants.

Yet, not everyone has access to high-quality evaluation. In particular, smaller nonprofits with annual budgets of less than $500,000, often community based, are far less likely to have the “culture, expertise, and resources to continually engage in evaluation,” according to a 2016 report by the Innovation Network. Only 16 percent of small organizations conduct evaluations, and of this group, 2 percent have staff dedicated to evaluation and 14 percent hire external evaluators. By contrast, 69 percent of large nonprofits (those with budgets of $5 million or more) conduct evaluations, 20 percent have dedicated staff, and nearly half (49 percent) hire external evaluators.

This discrepancy is particularly troubling because small nonprofits represent the vast majority of those registered as 501(c)(3) organizations in the United States. Slightly more than two-thirds of all such nonprofits had less than $500,000 in expenses in 2015, and 88 percent had budgets under $3 million, according to a report by Brice McKeever for the Urban Institute. Smaller organizations cannot afford evaluations, and yet without them, they will struggle to obtain funding. 

Nonprofits know the importance of evaluation and desire support for it. In a survey conducted by The Bridgespan Group for JPMorgan Chase, more than 200 nonprofits serving low- and middle-income communities identified evaluation as one of the top areas where they need the most help, right behind fundraising and communications—both of which partly depend on evaluation to demonstrate results to funders and the public.

But the fact remains that evaluation can be prohibitively expensive. It often requires high-priced external consultants or full-time professional staff with advanced degrees. Conventional wisdom suggests that nonprofit organizations should spend about 10 percent of their budget on evaluation. So an evaluation for a $1 million program should cost approximately $100,000. This is a massive expenditure for any small organization.

This problem demands a solution. The gap between those organizations that have access to high-quality evaluation and those that do not mirrors and exacerbates the inequities already experienced by the communities that smaller nonprofits serve. In order to address issues of equity, every nonprofit, regardless of size, must have access to evaluation and be able to use data for ongoing learning and improvement, advocacy for their community needs, and communication about promising practices and outcomes.

Cohort-Based Evaluation Capacity Building

Given the ever-widening gap between the “haves” and the “have-nots,” the sector must develop solutions quickly to avoid further weakening nonprofit organizations and the communities they serve. Two groundbreaking strategies have shown promise: cohort-based evaluation capacity building and shared measurement platforms.

Leslie Fierro, assistant clinical professor of evaluation at the Claremont Graduate University, defines evaluation capacity building (ECB) as “implementing one or more interventions to build individual and organizational capacities to engage in and sustain the act of evaluation—including, but not limited to, commissioning, planning, implementing, and using findings from evaluations.”

ECB has proven to be an effective approach for meeting the increasing demands by funders for evaluation and for strengthening the long-term capacity and sustainability of nonprofit organizations. The benefits are many and include improved attitudes, behaviors, and knowledge among individual staffers and better organizational practices, leadership, culture, and efficiency, according to a 2012 article in the American Journal of Evaluation by consultant Susan Labin and her coauthors.

Cohort-based ECB provides such benefits to multiple organizations through training in evaluation via workshops and one-on-one coaching. Staff learn evaluation basics and also implement their own studies with the support of an external evaluator. Staff within participating organizations learn important evaluation skills, use data to improve program quality, and can, under certain circumstances, build sustainable evaluation capacity. The cost per evaluation is often less than 5 percent of a program’s budget—a fraction of the proposed standard of 10 percent.

I learned about ECB cohorts firsthand in 2002 when I worked with The Robert Bowne Foundation and The Brunner Foundation to develop an 18-month ECB course to help after-school programs. At that time, such programs were just beginning to assess the relationship between their work and formal education outcomes as part of the No Child Left Behind Act of 2001. The entire sector scrambled to modify their program models and figure out how to evaluate their impact on school grades, attendance, and test scores. Funders and nonprofit program officers would come to us and say, “I don’t have very much money but I need to conduct an evaluation,” and I would explain the cohort model and ask if they had colleagues who might be interested. To my surprise, they were always able to recruit between 5 and 10 organizations together to share the cost. Over the years, I have had the privilege of engaging staff from hundreds of after-school and youth-development programs in ECB initiatives.

Despite such promise, the cohort-based ECB solution has its limits. In recent years, the growing demand for research-based tools, data-driven decision-making, and rigorous analytics has begun to strain nonprofit staff, who will need to take graduate-level statistics courses or hire high-priced consultants to create rigorously tested tools and perform complicated statistical analyses. So, nonprofit organizations still need to partner with outside experts and/or create full- or half-time positions internally to meet the growing demand. 

Shared Measurement Platforms

Fortunately, there is another solution that promises to revolutionize the practice of evaluation and the sector’s access to it. There are now a growing number of research-based validated tools on the market and several innovative web-based solutions that allow thousands of nonprofits to easily administer statistically valid measures and access real-time reports that include high-quality analyses of outcome growth, program quality, and benchmarks. These shared measurement platforms offer significant benefits, including lower cost, greater efficiency, expert guidance, improved credibility and consistency in reporting, cross-site learning and collaboration, and an increased ability for funders to make informed choices.

“The level of research that goes into the development of these systems, the timeliness and consistency in data reporting that they permit, and the standardization that they enable across multiple grantees all contribute to a significantly better quality of data about grantee performance,” says a 2009 FSG report titled “Breakthroughs in Shared Measurement and Social Impact.” “At the same time,” the report continues, “these systems offer grantees a degree of credibility that individual organizations’ idiosyncratic evaluation approaches often lack.”

Since the report, artificial intelligence (AI) and machine-learning techniques have boosted the power of these platforms, leveraging larger data sets so that real-time insights can be provided about what works, for whom, and in which contexts. These analytics inform front-line staff what has worked for clients similar to the ones they are serving so that they can tailor their work accordingly and readily move from reports to action without having to become data experts.

Organizations can become members of these types of platforms for less than 0.05 percent of a program’s budget and access rigorous analytics that are on par with randomized controlled studies that can cost hundreds of thousands of dollars. As more organizations join these platforms, the data sets become rich and diverse and the platforms smarter and more effective. Shared measurement platforms that value ongoing feedback, continuous improvement, and inclusivity can serve as massive joint experiments that enrich the sector’s knowledge and benefit every organization that participates.

A shared measurement platform is not a silver bullet. Staff often need help moving from recommendation to practice, and so nonprofits may need to hire consultants to help them integrate shared measurement tools into their work and understand the research and theories behind them. This step often entails working with a program consultant who can help write new curricula and establish new professional development opportunities.

Nonetheless, even with the added expense of technical assistance, shared measurement platforms greatly reduce the cost of both evaluation and professional development, because they make evaluation targeted and readily transferable. They enable everyone to train on a common set of practices and contribute to a collective body of training materials, grant language, program models, and curricula. Of course, each organization will make these shared measures their own, but they start with a solid research-based foundation. 

Applying technology and cutting-edge analytics to evaluation can not only reduce cost but also increase the quality and usability of the data collected for organizational effectiveness and field learning. The new shared measurement platforms promote a better understanding of communities often left out, simply because they have not had the resources to collect their data and study their impact. When information is gathered from and valued by all stakeholders, organizations can unearth new insights, understandings, and solutions that will produce sustained, informed, and equitable social change.

Read more stories by Kim Flores.