SPONSORED SUPPLEMENT TO SSIR
Advancing Evaluation Practices in Philanthropy
This special supplement includes six articles that address basic principles and practices that inform efforts to monitor performance, track progress, and assess the impact of foundation strategies, initiatives, and grants. The supplement was sponsored by the Aspen Institute Program on Philanthropy and Social Innovation and underwritten with a grant from the Ford Foundation.
When visitors step out of the elevator at the Omidyar Network (ON) office in Redwood City, Calif., the first thing they see is a quotation on the wall: “Every person has the power to make a difference.” A belief in the fundamental value and power of each individual shapes nearly every facet of ON, the philanthropic investment firm created and funded in 2004 by eBay founder Pierre Omidyar and his wife, Pam. At ON, we strive to create opportunities for individuals to unlock this power.
ON’s approach to measurement and evaluation is inseparable from our approach to philanthropy. We start with the notion that unlocking the potential of individuals is an indispensable way to improve the world. Given opportunities, people will tap their talents to improve their lives and consequently, the lives of our families and communities, which ultimately benefits society more broadly.
Omidyar Network strives to scale up innovation by applying many of the best practices from the venture capital industry to philanthropy. In rapidly changing markets, we believe the truly game-changing innovations will emerge from entrepreneurs who are empowered to identify changing situations and rapidly adapt their organizations. Our investment model is to invest in and then help scale up innovative nonprofit and for-profit organizations with the potential to create opportunity for hundreds of thousands or millions of people. Instead of funding others to implement programs related to our strategy, we seek to enable entrepreneurs to identify—and iterate on—their own strategies for creating social impact.
Measurement and Evaluation
ON has tried many approaches to measurement and evaluation over the last few years, and from that experience we are convinced that an organization’s approach to measurement and evaluation must flow from its investment model. ON’s approach is to conduct an up-front, detailed due diligence process to assess an organization’s potential and decide whether to invest. We then develop trusted partnerships with our investees to measure their progress in real time and provide additional, non-financial support as appropriate.
ON provides grants to nonprofit organizations and invests debt and equity capital in for-profit ventures. ON is an impact investor whose bottom line is social value. We take a systems view; we are driven by the problem that needs to be solved and have the flexibility to tailor the highest impact solution using the most appropriate type of capital.
Philanthropy through grants can benefit society in many ways. Grants can help nonprofits provide pure public goods. For example, the free legal tools offered by Creative Commons, which allow content creators to grant copyright permissions to creative work, increase the amount of content available to the public for sharing. Grants can help subsidize goods and services that produce positive societal outcomes but are undersupplied by the market. For example, Endeavor is helping to transform emerging markets by supporting entrepreneurs. Grants can help disadvantaged populations that are unable to participate in the market, as Landesa and other property rights organizations do. And grants can spur investments in high-risk ventures, as they did with the initial development of the microfinance industry.
Philanthropy through for-profit ventures, in contrast, can benefit society by leveraging the power of markets. When the primary motive is to generate profits, businesses will strive to deliver value in excess of costs and scale up. Customer willingness to pay for a product or service not only indicates value creation, but also serves as a valuable feedback mechanism.
ON has found that grants and for-profit investments can play complementary roles in delivering social impact, even, or maybe particularly, in difficult socioeconomic environments. Socially responsible grant capital can help potentially high-impact innovation gain market traction, and for-profit equity and debt capital can then help that idea scale up.
The phenomenal growth of microfinance is a good example of such complementary roles. Since 2004, ON has invested more than $100 million in 26 microfinance organizations: 15 nonprofits and 11 for-profits. In the 1980s and 1990s, most microfinance institutions were grant-funded NGOs. As microfinance’s impact and commercial viability became apparent, business investors—many with strong social motivations—invested heavily in commercial microfinance institutions, helping them grow rapidly. According to a 2009 Monitor Institute study, the percentage of the world’s top 50 microfinance institutions that were for-profit banks increased from 22 percent to 62 percent between 1998 and 2008. In other words, grants sparked and nurtured microfinance, and for-profit capital helped it scale up.
ON takes equity stakes in for-profit companies and typically provides negotiated general operating support to nonprofit organizations. This support gives organizations great flexibility to change their tactics and reallocate their resources in response to new information or changing market conditions. It also enables them to invest in critical overhead functions, such as developing management talent and information technology, which may otherwise not get funded because these functions are not directly related to delivering specific programs.
A grant or investment is only the beginning of ON’s relationship with our portfolio organizations. Although vital for growth, scale, and capacity, money alone cannot solve every problem or bring about the positive social impact we seek. Pierre and Pam Omidyar’s vision was to create an organization that augments the impact of its financial investments by deploying human capital and leveraging the knowledge, expertise, and innovation of all those involved in solving the world’s most challenging problems.
When requested by a portfolio organization, ON’s human capital team provides strategy, governance, and leadership assistance tailored to meet the grantee’s or investee’s needs. The team also creates opportunities for its investees to tap the power of the network by regularly organizing forums for them to exchange information and share best practices.
Selection Criteria and Due Diligence
ON’s investment selection criteria and process provide a foundation for measurement and evaluation of our investments. To make investment decisions, we consider factors such as innovation, scalability, and viability of business or revenue models. ON encourages nonprofit investees to develop earned-income streams whenever possible. Earned income is often a powerful tool for ensuring that a nonprofit focuses on understanding its constituents (customers); benefits from clear, market-based signals about the value of its product or service; and delivers products and services with values exceeding costs.
Like traditional venture capital firms, ON performs extensive due diligence before investing in an organization. Although traditional approaches to measurement and evaluation do not view due diligence as evaluation, organization-wide, up-front due diligence is central to our approach. In considering a funding investment, we carefully assess the organization’s management team, operational planning and governance, target market and competition, technology, product or service, and financials. We meet with the organization’s management team; interview customers, key channel partners, and board members; read industry and peer-reviewed reports; and analyze company financial and operating data.
The due diligence process is primarily about whether to invest, but it also affords an opportunity to explore strategically important issues that ultimately help ON support an organization. Detailed discussions with an organization’s management team help to ensure strategic alignment between ON and the investee. During this process the management team and ON agree on the metrics that will be used to evaluate the organization’s progress. Due diligence also helps ON determine how best to use its human capital to support an organization. Thus the diligence process has a direct bearing on all subsequent measurement and evaluation.
Reach and Engagement
ON, like all philanthropic funders, considers social impact the single most important investment criterion. Although our specific metrics vary across sectors and organizations, we have developed two overarching metrics that help us understand the social impact of portfolio organizations: reach and engagement. Reach is a measure of how many individuals are touched by a product or service. Engagement is a measure of the depth of that interaction.
Take for example the nonprofit Wikimedia Foundation, which operates Wikipedia. Wikipedia engages hundreds of millions of people in creating educational content, sharing information, and learning online. In 2010, a measure of Wikipedia’s reach was the roughly 400 million different people who visited its website. A measure of engagement was the 13 minutes that an average visitor spent visiting Wikipedia every month. Other measures of reach and engagement for Wikipedia include the number of active contributors of content and the level of their activity on the site.
Neither the reach nor engagement metric directly measures Wikipedia’s social impact, however. To determine impact we would need to understand the extent to which a product or service affects the end user. In this case, a potential impact metric for Wikipedia would be the improvement in students’ test scores after they use the site. Accurate estimation of Wikipedia’s impact would require an experimental or quasi-experimental approach, typically a randomized trial in the absence of a natural experiment, and might take months or even years to perform.
As an alternative to the time-consuming, costly, and complicated challenge of directly measuring impact, we often measure outputs— reach and engagement—as proxies for impact. Outputs are units of production, which often can be readily measured by an organization in the normal course of business.
Our use of a very limited number of easily collected metrics is a departure from some of our early evaluation frameworks in which we asked entrepreneurs to track as many as 18 different metrics. The earlier approaches not only were more costly to implement, but had less impact on investees’ success because neither they nor ON was able to prioritize appropriately.
We have learned that an organization’s social impact is best estimated using a combination of reach and engagement metrics tailored to the industry, the organization’s business model, and its growth stage. Some organizations, such as Wikimedia, have wide reach and relatively limited personal engagement. Other organizations, such as Endeavor, have modest reach and deep engagement. Selection of appropriate reach and engagement metrics can be difficult and nuanced.
Scalability and Social Impact
ON invests in highly scalable solutions with the potential to reach millions of people, and in which costs per unit generally decrease as the volume produced increases. Because organizations need a reliable revenue source to achieve scale, financial sustainability is a focus of our evaluation process—but there are some important differences in how we think about financial sustainability in for-profits and nonprofits.
For ON to consider investing in a for-profit company, the critical first filter is its potential for positive social impact, and the second is its ability to profitably and quickly scale up. We look at financial metrics that are standard for a profit and loss statement, such as revenue, operating margin, and net income. In particular, operating income is a leading indicator of impact because it demonstrates a company’s ability to deliver value to customers above cost.
In the for-profit organizations that ON funds, financial metrics are indicators of financial sustainability and social impact. A company’s revenue and margins are driven by the number of customers served and how highly those customers value the product, both of which provide a quantitative measure of the opportunities a company is creating.
To supplement standard profit-and-loss measures, ON uses metrics captured by the company—typically, key performance indicators that a for-profit company tracks to understand the underlying drivers of its financial performance. At Bridge International Academies, a Kenyan startup that operates a franchise-like network of low-cost, for-profit private schools, these metrics include the number of schools, the number of students and trained teachers, academic performance, and the turnover rate of students and teachers.
Revenue Streams
In nonprofit organizations, social impact is the most important evaluation criterion, and we spend considerable time with investees to define reach and engagement metrics that measure it. We also encourage and work with nonprofit investees to develop diversified sources of revenue as well as to generate earned income and other revenue sources to increase their financial sustainability.
Earned income is valuable to nonprofits not only because it provides revenue to cover operating costs, but also because it is a conduit for clear, rapid, and market-based feedback about whether the organization is providing a highly valued product or service. In this sense, not all revenue is equal. We view revenue tied to an organization’s products and services to be the highest value.
Similar to revenue and profit in for-profit organizations, market-based signals tied to products and services can be a powerful mechanism for getting nonprofit management teams to prioritize the activities of highest value to the end user. Market-based feedback also is likely to be more useful and valuable for both ON and investees because it informs real-time decisions without waiting for infrequent formal reviews.
For example, consider GuideStar, a nonprofit in which ON first invested in 2007. GuideStar offers information on 1.8 million US nonprofits and private foundations and had more than 10 million website visitors in 2010. Although 98 percent of GuideStar data uses are free, users pay a fee for more sophisticated tools and services. Fee revenue covered 90 percent of GuideStar’s operating costs in 2010. Charging fees for premium services has given GuideStar resources to grow quickly and enabled its management team to focus on delivering high-quality services to paying customers rather than raising money through donations. GuideStar now uses foundation grants primarily for special projects and opportunities.
Several of ON’s portfolio organizations have developed innovative donation models that are a twist on the earned-income model. For example, DonorsChoose.org matches donors with school classroom projects in need of support. When donors support a specific project, DonorsChoose.org invites them to contribute a small amount toward the costs of running the organization. In fiscal 2011, DonorsChoose.org covered a remarkable 102 percent of its operating expenses through this optional fee. Voluntary donor support of operating fees is a type of immediate feedback indicating the value the donor places on a product or service.
Cost-to-Serve
Cost-to-serve—the cost of providing a product or service to a single customer—is another metric that we typically track in for-profit and nonprofit investees. Cost-to-serve is an important measure of productivity— specifically, how efficiently an organization is delivering a particular product or service. In general, we find that despite its potential value for driving financially sustainable growth, few nonprofits focus on reducing cost-to-serve.
We calculate the cost-to-serve as the quotient of reach and the trailing 12 months of operating expenses. Cost-to-serve can vary significantly across organizations. For example, we estimate that in 2010, Wikimedia’s $0.02, whereas DonorsChoose.org’s cost-to-serve was $5.23, or roughly 250 times greater.
Cost-to-serve can offer some insight when comparing two businesses with similar models in the same sector, but is less useful when comparing across sectors. Typically, we look at an organization’s cost-to-serve to highlight opportunities for efficiency or asset productivity improvements. DonorsChoose.org, for example, is working on reducing its cost-to-serve by lowering new user acquisition costs and increasing the donor retention rate. Bridge International Academies—a new organization with relatively high startup-related overhead—has high costs relative to the number of students served. As Bridge opens more schools and benefits from the resulting economies of scale, we expect the cost-to-serve each student to drop dramatically.
Collaborative Evaluation
A successful relationship between ON and each organization is highly collaborative and based on trust. Collaboration starts with the due diligence process, through which ON evaluates the leadership team’s ability and commitment to practice exemplary governance, operational efficiency, transparency, and disciplined financial planning. This process builds relationships between ON personnel and the investee’s management team, establishing the basis for a positive, long-term relationship.
The spirit of collaboration and trust carries over to the measurement and evaluation process. Our investment professionals work closely with each investee to develop milestones jointly and then measure the investee’s progress against them. For-profit companies usually track the desired reach and engagement metrics during the normal course of business. For some consumer Internet companies, for example, the number of unique visitors to a site (reach) and number of page views per visit (engagement) are critical revenue drivers. The organization’s performance relative to established milestones is an important criterion for ON when we consider subsequent investments, but specific impact metrics typically are not captured in any agreement between ON and a for-profit company.
In nonprofits, ON first engages the leadership team in a dialogue about the organization’s potential social impact, and then works with the team to specify appropriate, related reach and engagement metrics that are codified in the grant agreement. These milestones provide a mechanism to both hold the organization accountable for achieving its agreed-upon objectives and identify when it might need additional human capital support from ON to be successful.
ON’s investment professionals regularly review each investee’s performance against these metrics. ON holds board seats or board observer rights in about 40 percent of our portfolio organizations, and metrics are usually reported by the management team at board meetings. If ON does not hold a board seat, the grant agreement typically requires the investee to provide quarterly reports of its progress against the metrics.
To supplement ongoing performance tracking and comprehensively evaluate our investees’ progress, ON holds an annual, internal portfolio review. The review covers the performance of each portfolio organization and the overall performance of all organizations by sector.
ON’s human capital contributions keep us in frequent contact with our portfolio organizations’ management teams. The close professional relationships with investees and the tight integration through these governance roles engender a sense of shared commitment between ON and each investee, building a foundation for effective, real-time evaluation and response. The frequent contact creates a continuous flow of information between ON and the investee, providing a substantial context for important decisions. Equally important, ON’s investment professionals have the added benefit of being able to look broadly across the entire market and competitive landscape, and can highlight specific issues that a more narrowly focused management team might miss.
What We’ve Learned
ON’s investment model fundamentally shapes how we approach measurement and evaluation. Developing and applying ON’s approach has taught us to:
Foster partnerships. Trust-based partnership is the key to establishing a productive approach to measurement and evaluation. ON fosters partnerships by listening carefully to our investees’ needs from the beginning of the due diligence process and investing heavily in our human capital function.
Exercise judgment. Data is rarely a substitute for good judgment. Only so much can be measured, much that matters cannot be measured, and any amount of data alone cannot provide all the answers. ON invests in talented management teams and works with them to strike the right balance between the robustness and complexity of a company’s approach to evaluation.
Be flexible. Markets, competitors, and companies can change rapidly. What is important to measure today may not be meaningful tomorrow, so we cannot rely on static measurements in a dynamic environment.
Embrace feedback. We manage what we measure. ON takes the time to understand what the data says about a company’s product or service. We then help our investees use that information to improve their product or service to deliver even greater value to customers, who will ultimately determine their success or failure.
At ON we continue to discuss and debate many topics related to how we approach measurement and evaluation, including the appropriate role for independent impact assessments, the optimal resource allocation across sectors and companies, the best approach to meaningful information and data sharing across funders, and whether to evaluate ON’s contribution to helping shift sectors.
Our view is that an organization’s approach to measurement and evaluation flows directly from its investment model. ON regularly re-evaluates whether its investees are having a positive impact on the sector and whether other investments are necessary to break bottlenecks. ON’s approach to measurement and evaluation thus remains a work in progress. We continue to iterate and innovate, jointly with our colleagues in the impact investing sector, with the objective of investing in and scaling organizations that have the ability to create opportunity for millions of people.
The authors wish to thank Tie Kim and Gowri Pai for research assistance; Jeff Bradach, Suzanne Lippert, Paula Goldman, Annis Steiner, and Greg Pershall for editorial comments; and Arjuna Costa, Chris Bishko, Amy Klement, Bill Barmeier, and Todor Tashev for valuable discussions.
See the complete evaluation supplement.
Support SSIR’s coverage of cross-sector solutions to global challenges.
Help us further the reach of innovative ideas. Donate today.
Read more stories by Eic Hallstein & Matt Bannick.