(Illustration by David Plunkert)
Impact measurement is a useful indicator of how philanthropy can make a difference. For an organization with a broad remit, such as The Hong Kong Jockey Club Charities Trust (“the Trust”), which serves Hong Kong’s changing needs, finding the right metrics to assess our collective social impact is challenging. We recently developed a new approach that measures the collective impact of individual funded projects, according to different funding themes, and across the entire organization. Sharing some lessons with our regional and global peers, we hope to contribute to the evolution of impact measurement and better serve our grantees and, through them, our beneficiaries.
The Evolution of Impact Measurement at HKJC Charities Trust
The Trust’s primary goal is to contribute to the betterment of society, which has meant different things at different moments in our century-long history. A key part of our evolution has been adapting both our funding goals and our accountability to address changing societal needs.
Since our first recorded charitable donation in 1915, contributions have continued to climb, especially starting in the 1950s, when Hong Kong’s population doubled in 20 years, from two million in 1951 to four million in 1971, mostly due to migration from mainland China.
With rapid population growth putting acute stress on government finances, the Trust came forward to collaborate with local authorities to fund projects such as building schools and hospitals, and then to expand this work by constructing arts facilities, sports grounds, public parks, and other infrastructure to meet the changing needs of a young, prospering society and its growing middle class.
(Click to enlarge)
In the early 2000s, the focus of our giving shifted to service programs to tackle emerging issues such as an aging population and environmental protection. Since each program had different goals and required different approaches, we promoted pledged outputs and clear targets for each project to ensure prudent funding allocation, accountability, and transparency. We also began developing tailored, rigorous impact measurement indicators for large-scale university-led programs. These measures began as the exception rather than the rule, but by the 2010s, our approach changed. We recognized that applying these rigorous measures across all programs could optimize impact.
This objective was not straightforward by any means. First, the local nonprofit sector had been unfamiliar with impact measurement and did not know how to implement it or where to find the required resources (funding, methodology, expertise, and manpower). Second, the breadth of our programmatic coverage made it challenging, from the Trust’s perspective, to identify a single common metric to assess impact.
After consulting grantees and external experts, we took a “lowest common denominator” approach by adopting a rudimentary measure for project impact based on changes in the knowledge, attitudes, and behaviors of beneficiaries. This simple and potentially universal framework could be applied to all projects and allowed grantees to adapt their execution according to their circumstances. We provided training and a basic template to help them produce indicators and survey questions.
The framework’s simplicity turned out to be a limitation. It lacked scientific rigor, and there was little evidence to show that shifts in knowledge and attitudes led to sustained behavioral changes.1 As the nonprofit sector became stronger and more professional, we believed that a more scientifically valid approach would better maximize value-added impact for every dollar invested and align our work with global trends.
This realization dovetailed with our organization’s evolving agenda. We wanted to understand the collective impact of funded projects within a specific field, not only the impact of each individual project. Two years ago, we launched six program areas dedicated to different issues: Positive Aging and Elderly Care; Youth Development and Poverty Alleviation; Healthy Community; Talent and Sector Development; Sports and Culture; and Sustainability. We are also testing a novel approach to assess impact that aggregates across all fields and levels of giving.
A Three-Tiered, Hierarchical Approach
Following a comprehensive literature review of studies and practices, our latest approach focuses on a hierarchical model that considers impact at the project, program, and organization (HKJC Charities Trust) levels. The aim is to produce timely, relevant, and rigorous evidence based on the following core principles:
- Clear objectives and hypotheses to define the purpose and scope of impact measurement;
- Rigorous evaluative designs to establish causality between interventions and outcomes;
- Validated measures to provide accurate and appropriate outcome data;
- Robust statistical methods to quantify the effects attributable to the interventions;
- Aggregation of impact to demonstrate social impact at multiple levels; and
- Regular, fit-for-purpose reporting.
(Click to enlarge. See glossary of all metrics in the notes at the end of this article.)
At the project level, each project should employ a robust evaluation design, such as a randomized controlled trial or longitudinal cohort study, supported by validated measurement indicators and appropriate analytic methods that aim to establish causality. This methodology helps us to monitor and evaluate projects and to provide useful feedback to funding applicants.
At the program level, thematic indicators are applied across multiple projects so that the combined impact on a particular problem or theme can be assessed. For instance, if multiple elderly care projects provide different services, the collective impact could be determined by using process measures, such as the total number of people served, and the end goals, such as health/disability burden outcomes. Both quantitative and qualitative outcomes are important considerations in creating a comprehensive view of impact that includes all beneficiaries. We believe that defining overarching program-level objectives and their corresponding metrics can help us articulate and report on social benefits generated across different but cognate projects.
At the Trust (organization) level, we assess whether and to what extent the Trust fulfills its underlying mission of bettering society. Project indicators can be transformed to dimensionless form (“effect size”) and aggregated using statistical techniques such as meta-analysis across five core domains in which projects strive to create positive change: Capacity and Capability; Health; Social Relationships; Financial Resources; and Environment.
It is still early days, and especially at the organization level, we continue to explore how to frame and validate our findings. We can look to our global peers, but we are also committed to finding evidence rooted in the East and in the context of communities at home.
Four Takeaways
Our journey toward measuring and validating our impact is a work in progress, and yet we have already learned several important lessons. We invite fellow philanthropists and practitioners to join us in this conversation.
1. There is no one-size-fits-all framework. Every project takes place in a particular context that must be considered when designing impact measurement. This context can shift, as the Trust experienced when our giving moved from building basic social infrastructure to focusing on service programs and related policy innovations. Additionally, impact measurement means something quite different to a large organization such as the Trust when compared with individual family offices, which may not enjoy as much experience or scale in strategic giving and impact measurement.
Government policy can also shape impact measurement. China’s 14th Five-Year Plan, for example, sees philanthropy as a key force for distributing wealth and achieving “common prosperity.”2 In India, a mandatory 2 percent of company net profits help fund corporate social responsibility activities to ensure that businesses adopt “inclusive growth.”3 The Muslim zakat tradition of donating a portion of wealth to charitable causes is another example. Policy enables the scaling up of available resources for doing good, which has implications for impact measurement approaches as stakeholders seek to ensure that their contribution is deployed in the best way possible.
These cases underscore how impact measurement must be fit for purpose, rather than one size fits all.
2. A user-centric mindset is essential. Users, especially grantees and beneficiaries, are the backbone of impact measurement, providing everything from beneficiary data collection to reporting to funders. Procedures must be designed in collaboration with users without becoming overly complex or duplicating data requirements. Users should not face an undue burden, and their feedback should be facilitated in the process.
But funders should also help users gain an awareness of the many impact measurement tools they might use. Funders should provide resources and training while maintaining dialogue with users to align expectations. Both funders and users need to learn and evolve together to drive sustainable impact.
3. Trust is paramount. How we measure impact can strengthen trust-based relationships between funders and grantees and foster a more collaborative and effective philanthropic partnership. Clear impact measurement and transparency will generate accountability in the grantmaking process and a sense of shared responsibility for creating impact. A two-way or multi-way dialogue ensures that grantees have a strong voice in shaping current and future initiatives. This participation can lead to more effective and enduring philanthropic endeavors.
4. Upholding integrity in impact measurement. The Trust’s funded projects often target society’s most vulnerable groups, whose agency is often ignored and whose trust must be earned. We work with grantees to ensure that beneficiaries choose to participate, that their privacy is respected, and that both data and impact measurement will be responsibly used for their benefit. As advocated by the Stanford Center on Philanthropy and Civil Society, organizations should manage data in line with their mission and with regard to privacy, permission, openness, and pluralism.
Leading by Adapting
How can funding organizations and their recipients know if they are achieving their objectives? This question must be considered in context. Rigorous, evidence-based impact measurement provides answers, but societal demands and circumstances also affect goals, methods, and applications. The funding organization’s goals matter, too. The Hong Kong Jockey Club Charities Trust now strives to articulate our impact at all levels of giving, including at the organization level, across all fields of philanthropic investment. This path has been less trodden, but we feel the stones along the way and continue to evolve and adapt toward greater impact for the betterment of the society we serve.
Read more stories by Gabriel M. Leung.