Next Generation Evidence: Strategies for More Equitable Social Impact
Kelly Fitzsimmons (Editor)
500 pages, Brookings Institution Press, 2023
Evidence is powerful: It helps us understand the needs of communities, make decisions in times of complexity, and generate stronger and more equitable outcomes. However, practitioners face a number of structural and practical hurdles to building and using it. Traditional evaluation and research methods are often not timely, affordable, meaningful, or inclusive for helping practitioners make decisions to increase their impact for students, families, and communities. Too often and for too long, evaluation was a thing done to practitioners and the communities they serve, relegating them to a passive role when they should be regarded as leaders of this work. And for too long, evaluation practice has resisted broader, more modern methods, including data science, AI, and R&D approaches, that are relevant for users but perhaps not for researchers.
Next Generation Evidence: Strategies for More Equitable Social Impact features innovative thinking from leaders across policy, government, technology, philanthropy, research, community, and practice. (Get the free ebook.) Together, these leaders lay out a vision for a stronger, more equitable data and evidence ecosystem that uses actionable approaches and centers on the voices of people and communities most directly impacted by the problems we seek to solve.
In this excerpt, AmeriCorps’ CEO Michael D. Smith reflects on his experience championing the role of evidence in the past decade and calls for a broader and more inclusive definition of evidence. It is a powerful and personal story of a policy maker evolving his approach to measuring impact and making evidence-based decisions, and one that I believe many leaders in the social sector (myself included) can relate to.—Kelly Fitzsimmons
* * *
Systems Must Change: Dismantling, Disrupting, and Reimagining Evidence
As I was preparing to write this essay, I sat down to watch a talk I gave at TEDxMidAtlantic in 2014. At the time I delivered the talk, I was director of the Social Innovation Fund, a program of the Obama White House and the Corporation for National and Community Service. At the Social Innovation Fund, we sought to combine public and private resources to prove, improve, and scale promising interventions in low-income communities. It is never easy to watch yourself talk, especially after so much time has passed. But watching this one, I found myself wincing even more than usual. Because while my passion about investing in what works has stayed the same, my feelings about how we got here and what we need to do about it have evolved greatly since that talk. In the intervening years, I have spent more time working closely with organizations doing the hard work of building safety nets and springboards on top of what can feel like a bottomless cavern of neglect, institutional racism, and lack of investment where it is needed the most. What I have come to realize is that we need to radically reimagine our approach to evidence in the social sector.
During those remarks almost ten years ago, I discussed why we need to invest in evidence-based solutions if we want to transform the nonprofit sector, find a bigger impact, and get more results. Americans are obsessed with data, rating, and reviewing, but for some reason, that obsession does not apply to the nonprofit sector. Back then, we were spending $300 billion a year on more than 1.5 million nonprofit organizations, but one in eight nonprofits that year spent zero dollars on evaluation, and more than half did not have a theory of change or a logic model. And while these problems seemed to be getting bigger and more complex, we were making decisions on which nonprofits to invest in based on anecdotal stories of success and numbers served. I concluded that I believed the best way to ensure that our limited dollars find their way to the most deserving nonprofits was to follow the evidence of impact, and I cited the Social Innovation Fund as an example. At that time, the fund had invested more than half a billion state and federal dollars to more than 200 organizations that were testing about eighty-six different models with some evidence of impact.
This work matters deeply to me. I grew up in a poor Black neighborhood in a small city in New England. My mom and dad were single teenage parents; with what we knew from data on kids like me, I should have been an unfortunate statistic. My saving grace? The neighborhood Boys and Girls Club. But as I got older and spent my career working in the nonprofit sector and philanthropy, I was forced to wonder why, if that youth center was so transformative for me, did so many of my childhood friends end up struggling in so many different ways? In fact, in a less-than-five-mile radius in my little neighborhood, at least a half dozen organizations, all separate 501(c)(3)s, were doing very similar work, but with diminishing results. In spite of these well-intentioned organizations, my city struggled with poverty, teen pregnancy, and low graduation rates. Too many youths who looked like me were victims of homicide—including my younger brother, who was killed at the age of twenty-seven.
In my talk, I pondered aloud why, despite the fact that my family’s story is shared by countless others in this country, we keep doing the same things over and over again, expecting different results. Then I shared some thoughts on what we should do. First, I suggested that individuals, who are responsible for 80 percent of giving to all nonprofits, stop giving dollars to any organization that cannot articulate impact. I defined impact not just as how many people were served, or as isolated success stories but, rather, as how many kids went to college, and stayed in college; how many got jobs, and how many kept them. Second, I shared that we’ve got to know when to walk away. The nonprofit sector does not face the same market forces that drive dollars away from ineffective solutions in the business world. Sometimes when social sector approaches have been poked and prodded and they no longer work, we are going to have to say “No more money.” Finally, I shared that the philanthropic sector would have to push for more mergers or acquisitions to build stronger organizations instead of the “behemoths and masses of ineffective organizations that are out there.” (Cringe, I know.) And, we should then bet our money on the winners. Some nonprofits might shut their doors, I thought at the time. Some might go away, and many need to. That might sound harsh. But, perhaps, this focus could put an end to the “Hunger Games” that we’ve created where nonprofits that aren’t growing are fighting each other for scarce dollars.
Now, you can probably tell why I am cringing. So arrogant—so pompous. Somewhat out of touch. I put all the burden on the organizations doing the hardest work; trying to keep their doors open in and out of recessions and near-depressions. And organizations that so many families will turn to in times of greatest need, as we have just seen during the pandemic. Here is what changed my mind.
What Influenced Me?
At the Social Innovation Fund, I worked with hundreds of nonprofits of all shapes and sizes that were struggling to meet basic needs of their constituencies while also building evidence that would pass the scrutiny of funders. Imagine if the organizations could focus on their core missions without constantly trying to prove themselves? In the aftermath of the tragic killing of Trayvon Martin and the shocking trial where his murderer was acquitted, I helped design President Obama’s My Brother’s Keeper initiative (MBK), and in 2014, I left SIF to lead MBK. We aimed to address the persistent opportunity gaps facing boys and young men of color and ensure all youth could reach their full potential. And that is what the president talked about when he launched the program; he addressed the nation about the urgency of making sure every kid in this nation, no matter their background or neighborhood, knows that their country cares about them, values them, and is willing to invest in them. He also spoke about the urgent need to focus on evidence, data, and results, or, more simply, investing in what works and building on what works. “We don’t have enough money or time or resources to invest in things that don’t work, so we’ve got to be pretty hard-headed about saying if something is not working, let’s stop doing it. Let’s do things that work. And we shouldn’t care whether it was a Democratic program or a Republican program, or a faith-based program or—if it works, we should support it. If it doesn’t, we shouldn’t.”1
In 2018, the MBK Alliance announced the winners of our inaugural national competition to identify and invest in communities making steady progress to substantially improve the lives of boys and young men of color. The critical importance of the work of these extraordinary organizations and countless nonprofits like them became even more clear in 2021. In the wake of the disproportionate effect that COVID-19 and ongoing racial injustice was having on under-resourced Black and brown communities, these high-performing organizations continued to meet their core operational goals to reduce barriers and expand opportunity for boys and young men of color and their families—but they did not stop there. In the face of so much uncertainty and overwhelming obstacles, they took on even more. They began serving meals; delivering food; handing out personal protective equipment and literature; creating mutual aid networks; helping organize and support calls to action against police violence; and responding to increases in street violence. When I sat back to think about the kind of hoops organizations like these—and many we invested in through SIF—have to go through to prove their work without the resources needed, it convinced me even more of the need to rethink how we approach gathering and applying evidence.
What Have I Learned in These Past Eight Years?
First, I would challenge us to tackle the system, not the nonprofits struggling to hold together a society that was never built to support its most vulnerable citizens. When I gave that TEDx talk, I did not spend any time talking about the massive gap in funding evidence-based programs versus the need to invest in evidence-based policy reforms that seek to dismantle the inequitable systems that created the conditions we need to address in the first place. When we put all the pressure on nonprofits trying to address the base of the hierarchy of need, we give policymakers, business leaders, and everyday citizens a pass on investing in change at society’s roots. No amount of randomized control trials and evidence-based interventions will combat the legacy of redlining; Jim Crow; redistricting; under-funded, inequitable schools; and the prison industrial complex. But even when approaching systemic change, there is an opportunity to invest in evidence-based policy reforms that prove they work, such as eliminating external school suspension; increasing support for restorative justice, diversion, and other violence-prevention initiatives; increasing access to public spaces for young people; and facilitating opportunities for trained adults to mentor underserved youth. We also cannot forget that the road to macro reform is paved with lots of micro changes that aren’t splashy but are pivotal to transformation, such as government budgeting processes; procurement processes; community engagement; capacity building; and data systems. One example of this micro shift is equity budgeting, which suggests radical intentionality about the inclusion of vendors, contractors, and businesses that are led by people of color and organizations led by residents.
Second, I downplayed the importance of balancing statistics and storytelling, as well as what we consider to be acceptable evidence from the start. Now I see it is not only about better science. It also is about being proximate to the need. It is about letting people closest to the pain be closest to the power because they hold both the causes and solutions in their daily, lived experience. It is about providing the time for rumination and reflection. It is about mirroring the data with the emotion. It is about spending time in communities that don’t have the resources to build complex evaluation models but, for some reason, are outperforming the rest. To paraphrase Edgar Villanueva, author of Decolonizing Wealth, we have to resist our colonized mindset. We have to resist wanting our solutions tied up in neat, polished packages with the perfect prose from elite institutions and, instead, set our sights precisely on where change is happening.
Here is an example. Early in my career, I was part of philanthropic efforts to bridge the digital divide, distributing computers and internet access across the country. I had one grantee that was not as responsive as others. He submitted reports late, and they were incomplete. I decided to make a site visit to ensure our funds weren’t being swindled. I landed at the airport with my MapQuest directions printed out, picked up my rental car, and headed to Ferriday, Louisiana (population 3,312). I drove by shotgun shacks, abandoned homes, and kids playing with homemade toys. I pulled up to our community technology center, which was housed in one of those homes that looked like it was on its last legs. Kids were running in and out, talking, laughing, and learning, and using the computers in a room heated by a wood stove. I met the director, who shared with me that his late mother, who had cared for children in the neighborhood, had left him the house; he could think of no better way to honor her than to create an informal afterschool program. It was one of the only safe places in the area kids could go after school, get some homework help and mentorship, use a computer, and get a snack. And the leader was keeping it together out of his own pocket with some occasional grants and some help from the neighbors. No quasi-experimental design could have shown me what I saw with my own eyes and heard in the stories from that servant leader and the children and families I met that day.
We need data. We need evidence-based approaches. We need to trust but verify. But we also need to listen, look closely, create avenues for storytelling, and clear on-ramps to creating social impact for individuals without access to the resources that come with privilege. We need to recognize that the organizations serving the soup and handing out warm coats also may be the best advocates and engineers of revolutionary reforms to address hunger and homelessness. And, we need to reimagine our definition of evidence-based approaches so each of these iterations and innovations at every stage is part of the solution.
Third, if we are asking nonprofits to save lives and stay on top of the science that guides their service delivery and advocacy, we have to do more than talk about it. We cannot tell nonprofits to invest in what works without changing the way government and philanthropy fund operations, administrative costs, evaluation, research and development, data collection, and analysis. And, we need to help build the capacity of organizations of all sizes so they can own their data collection and evaluation instead of having to rely on outside firms with less cultural competency, less understanding of the community, and approaches that turn participants into subjects of a study, which can feel punitive and remote.
We also cannot ignore the fact that smaller organizations, and organizations led by people of color, find themselves constantly facing closed doors when it comes to the kind of resources they need to invest in infrastructure and growth. A recent study by the Bridgespan Group and Echoing Green showed that, in 2019, the revenues of Black-led organizations were 24 percent smaller than those of their white-led counterparts, and the unrestricted net assets of the Black-led organizations were 76 percent smaller than those led by whites. We also know that bias shows up in evidence and evaluation processes on all points on the spectrum. White researchers receive National Institutes of Health (NIH) grants at nearly twice the rate Black researchers do. Changing this paradigm starts with funding small organizations; investing in building a pipeline of more researchers of color from the communities undergoing evaluation; expanding the evidence toolkit to be hyper-inclusive; and leveraging practices such as government budgeting, procurement, and pay-for-performance (paying based on outcomes)—all of which can be tools for creating meaningful community engagement and more equitable structures.
I have long believed in the proverb, “If you don’t know where you’re going, you’ll end up somewhere else.” None of us can afford to spend our days tilting at windmills, hoping our work will transform lives. We have to demand that governments, businesses, nonprofits, and philanthropies do more to shift the massive amount of dollars to solutions that have measurable evidence of impact. But, we also have to expand our understanding of what constitutes evidence. We have to grow our tent so more diverse voices and perspectives fit under it and have a seat at the table. And we must evolve our concept of an evidence-based solution from a program that meets an immediate need to include policy reform that dismantles, disrupts, and reimagines the broken systems that have failed far too many. If I could give my talk over today, that is what I would say.