New Ways to Evaluate Impact

January 12 marked the fourth anniversary of Haiti’s catastrophic 2010 earthquake, and since the disaster, international aid and recovery programs have met with vastly mixed—and largely unknown—results. A year ago, our social innovation consultancy, Openbox, traveled to Haiti for #HaitiHack, a three-day hackathon designed to promote cross-cultural and cross-sector collaboration. This pioneering event kicked off a six-month investigation of community development initiatives in Haiti. Openbox collaborated closely with #HaitiHack’s core partners—Digital Democracy, KOFAVIV, and l’Ecole Supérieure d’Infotronique d’Haïti (ESIH)—to develop new ways of understanding how these organizations achieved social and economic impact.

At Openbox, we define impact as the ability to establish and maintain new and improved social and economic standards over time. Our research uncovered the power of human-centered design and systems thinking to improve data collection and analysis practices for defining and evaluating impact strategies.

Gathering data based on human-centered design helps supplement output-based quantitative assessment methods.

We made an early decision not to rely on quantitative output metrics alone for our evaluation, as they are often inconsistent—limited by timeframe, ease of gathering, reporting mechanism, funding constraints, and countless other considerations. Programs are often forced by funding restrictions to track specific metrics, which can lead to programs designed to produce selectively appealing numbers. A rigid, output-based quantitative assessment model easily validates such programs, without seeking to place those numbers in a larger context.

Instead, we used human-centered design (HCD) principles in our evaluation process to extract strategic insights about how Digital Democracy, a nonprofit that empowers marginalized communities to use technology to defend their rights, partnered with KOFAVIV, a grassroots group fighting gender-based violence in Haiti. HCD is generally used to bring traditionally ignored groups into the design process and reposition users as designers. We repurposed these practices to debunk modes of participation that dissociate evaluation subjects from their own storytelling process and reposition them as collaborators.

The Digital Democracy-KOFAVIV partnership focused on replacing KOFAVIV’s data reporting system after the loss of its records during the 2010 earthquake and adding tools such as a 24-hour national hotline. We used HCD-based exercises such as collaborative storytelling and visual mapping, allowing each organization to articulate their own understanding of the partnership and its value. We mapped the co-design process for the new delivery and reporting systems, explored how co-designing assisted adoption, and tracked how KOFAVIV’s new data sharing practices have influenced Haiti’s governmental systems.

Using “systems thinking” helps us understand impact through long-term strategy—not just short-term outcomes.

International development professionals are increasingly taking systemic factors into consideration when designing new programs and solutions. When it comes to impact evaluation, we believe a systems-based analytic framework that foregrounds scale and complexity better reflects how long-term strategy shifts systems toward impact. While we do appreciate short-term results, they’re only one small part of a bigger picture.

In our systems-based conceptual framework, we isolate a challenge that limits human development, identify a set of relevant conditions, and develop an internal logic to unlock pathways for systemic change. Each of these pathways manifests as a strategy developed by a group or individual. Highlighting the strategies devised to navigate and influence a system acknowledges collaborators as active participants in shaping impact, and can also map the resilience and adaptability of their thinking over time.

We quickly recognized that KOFAVIV’s multidimensional model was uniquely suited to navigating the notoriously fragile terrain of Haiti, so we set out to map the evolution of constituent behaviors over time to better understand how they operate. We found that KOFAVIV combats the entire spectrum of gender-based violence using a service model that follows logic comparable to socially contagious diseases. The organization’s network of agents help survivors transform into empowered human rights activists who can become agents themselves. Below is a visualization of this pattern showing how this grassroots organizing strategy helps KOFAVIV grow its network and impact over time.

Contagious Logic model. (Designed by Eli Rosenbloom)

Evaluation at the individual level helps us understand what leads to sustainable, scalable impact at the systems level.

Ultimately, we define impact as the ability to establish and maintain new and improved social and economic standards over time. Impact requires creative problem solvers who can devise strategies to weather the multidimensional challenges of complex systems. One of our considerations in evaluating impact is whether program participants become more empowered as individuals. Just as Digital Democracy ensures that KOFAVIV members own their own tools and KOFAVIV empowers survivors of gender-based violence, ESIH is building technologists equipped to operate in any technology environment.

Students from ESIH, a higher-education technology institute in Port au Prince, were participants in #HaitiHack one year ago. Today, ESIH’s students are using hackathon practices learned at #HaitiHack to complete school projects and participate in other hackathons, including NASA’s International Space Apps Challenge. ESIH students also run hackathons under their own homegrown label—Konbit Nasyonal—demonstrating how a small investment in individuals can jump-start a new level of innovation that promises long-term impact.

When it comes to evaluating impact within complex systems, we are just scratching the surface of what’s possible. We invite you to share your reactions and ideas in the comments.

Read more stories by Deepthi Welaratna & Garance Choko.