As a practitioner of evaluation on charitable projects in China, I do feel the “attraction” of having impact evaluations. However, there is very little discussion about “maybe we don’t need an impact evaluation” here. Love the concept of right-fit ME system and actionable/utilization-focused ME. It is just so challenging to pursuing, while the evaluations are hired and paid by NPOs, funders, etc. Will share this great article with my partners and will start reading the entire book The Goldilocks Challenge. Thanks!!
Great and very useful piece. This complements a piece Kate Ruff and I wrote here in SSIR about the role of infomediaries in improving the cost-effectiveness of efforts to solve social and environmental problems. We argue that harnessing the capital markets to solve social and environmental problems requires not only that organizations understand their impacts (and your guidance here is excellent about how that can be pragmatically approached), but also that a new profession of analysts emerge who can help make sense of the impact information reported by those organizations and help funders, policy makers, and the public at large understand where there is alignment with their own impact priorities.
The entire burden to understand and make sense of impacts cannot be shouldered by individual organizations or even them and their funders- just as Wall Street doesn’t and shouldn’t expect GE to be its own industry analyst.
Related to your point about the decreasing cost of collecting data, you might find relevant the Impact Tracker Technology (ITT) catalogue that features a compilation of low cost, information communications technology-based tools intended to help small to medium size organizations in the development sector collect data, communicate with their clients and measure their impact.
An excellent read, both for NGO practitioners and for students at the graduate level. I will recommend it to others. One comment: you pay less attention (except for one brief mention) to interventions whose nature may not justify investing in impact evaluations: i.e. non-service delivery interventions. Examples such as advocacy, campaigning and mobilization interventions. How do we right-fit evaluations for those type of interventions?
The article title might give an impression that we do not need social impact measurement, which is misleading. However, an author is later is giving the right way to collect data.
The real question is how we can collect impact data that goes a long way towards how much we are able to leverage such data for the benefit of our beneficiaries and other stakeholders.
We need to start at the enterprise level where nonprofits, social enterprises, and business have a direct access to data, progress, and outcome which can turn into powerful lean / feedback data to improve change/business approach. We wrote a perspective - “Finding a Needle in the Haystack: Bringing Impact Insights to Lean Data”.
“To create a right-fit evidence system, we need to consider not only when to measure impact, but when not to measure impact.”
The article title is a bit misleading. While the authors initially seem to propose the idea that impact evaluation is not always appropriate, the article only demarcates impact from operational analysis; it still has value as primer for transitioning from operational analysis to impact elucidation.
I would have also liked to see included references to conclusions on impact measurement (e.g. the lack of impact in microfinance programs), as I imagine the determination of impact on these programs is much more complicated and nuanced than could be stated in such a brief manner above.
Thank you for this article. I particularly like and relate with the alternatives offered. Impact evaluation is ‘trendy’ in the non-profit world, and soon we shall loose its actual value and power to inform programs that better people’s lives.
Recently, I attended a project partners meeting. The donor rep was insistent on us conducting an impact evaluation, and wasn’t prepared to take no for an answer. To my surprise, some implementing partners in attendance were open to adjust the program to support the evaluation.
Yet upon, additional discussion of the rationale for needing an impact evaluation and how results will be used, it was evident the donor rep wanted data just above activity/implementation monitoring! In fact, he would be satisfied with the monitoring data you have described in this article as: targeting, engagement and feedback. Imagine the waste of resources if the we didn’t do deep interrogation of the rationale, pros, cons and alternative to impact data.
Perhaps this article should have elucidated the need and use of annual monitoring of program outcomes to compliment routine performance monitoring, as alternative to some of the needs for impact evaluation. That’s to say, data on changes in behavior, adoption of Bette practices, etc.
Even if the concept and methodology for impact evaluation may be well understood among non profits including resident expert to do it well, other practical issues may impede implementation. One such issue is ethical obligation. For the same community in need, It’s a challenge to deny service to a group of people while serving others, even if doing so may lead to better future programs.
Again, thank you timely article that frees us from the ‘impact evaluation box’, giving us options-more creatively-about improving social interventions. I will rethink my own practice and approach to M&E systems strengthening, thinking along the CART principles.
This article is making do’s and don’ts points for impact measurement and impact reporting and they are practical helpful tips.
Here are some important points to be added to the impact report.
-Clear reasons for your impact journey
-Impact strategy aligned with appropriate indicators
-The key impact goals and stakeholders
-Inputs and outputs
-What actually happened? A positive and negative change
-Clear data insight from all the dimensions such as WHO, WHAT, HOW MUCH, CONTRIBUTION, and RISK
-What will be the next step? How are you planning to use your learning?
The below blog provides more insights about nonprofit impact reporting.
COMMENTS
BY MAYA JANKOVIC
ON May 17, 2018 05:38 PM
Great article. I enjoyed reading it. An eye opener.
BY Rui
ON May 24, 2018 09:52 PM
As a practitioner of evaluation on charitable projects in China, I do feel the “attraction” of having impact evaluations. However, there is very little discussion about “maybe we don’t need an impact evaluation” here. Love the concept of right-fit ME system and actionable/utilization-focused ME. It is just so challenging to pursuing, while the evaluations are hired and paid by NPOs, funders, etc. Will share this great article with my partners and will start reading the entire book The Goldilocks Challenge. Thanks!!
BY Sara Olsen
ON May 29, 2018 11:27 AM
Great and very useful piece. This complements a piece Kate Ruff and I wrote here in SSIR about the role of infomediaries in improving the cost-effectiveness of efforts to solve social and environmental problems. We argue that harnessing the capital markets to solve social and environmental problems requires not only that organizations understand their impacts (and your guidance here is excellent about how that can be pragmatically approached), but also that a new profession of analysts emerge who can help make sense of the impact information reported by those organizations and help funders, policy makers, and the public at large understand where there is alignment with their own impact priorities.
The entire burden to understand and make sense of impacts cannot be shouldered by individual organizations or even them and their funders- just as Wall Street doesn’t and shouldn’t expect GE to be its own industry analyst.
BY Toshihiro Nakamura
ON May 29, 2018 06:24 PM
Thanks for the insightful article!
Related to your point about the decreasing cost of collecting data, you might find relevant the Impact Tracker Technology (ITT) catalogue that features a compilation of low cost, information communications technology-based tools intended to help small to medium size organizations in the development sector collect data, communicate with their clients and measure their impact.
http://impacttrackertech.kopernik.info/
BY Tosca Bruno-van Vijfeijken
ON June 26, 2018 10:48 AM
An excellent read, both for NGO practitioners and for students at the graduate level. I will recommend it to others. One comment: you pay less attention (except for one brief mention) to interventions whose nature may not justify investing in impact evaluations: i.e. non-service delivery interventions. Examples such as advocacy, campaigning and mobilization interventions. How do we right-fit evaluations for those type of interventions?
BY UNMESH SHETH
ON July 26, 2018 03:48 PM
The article title might give an impression that we do not need social impact measurement, which is misleading. However, an author is later is giving the right way to collect data.
The real question is how we can collect impact data that goes a long way towards how much we are able to leverage such data for the benefit of our beneficiaries and other stakeholders.
We need to start at the enterprise level where nonprofits, social enterprises, and business have a direct access to data, progress, and outcome which can turn into powerful lean / feedback data to improve change/business approach. We wrote a perspective - “Finding a Needle in the Haystack: Bringing Impact Insights to Lean Data”.
https://www.sopact.com/perspectives/finding-the-needle-in-the-haystack-impact-insights-a-lean-data-approach
Thanks,
Unmesh
BY Daniel Gray
ON August 2, 2018 12:20 PM
“To create a right-fit evidence system, we need to consider not only when to measure impact, but when not to measure impact.”
The article title is a bit misleading. While the authors initially seem to propose the idea that impact evaluation is not always appropriate, the article only demarcates impact from operational analysis; it still has value as primer for transitioning from operational analysis to impact elucidation.
I would have also liked to see included references to conclusions on impact measurement (e.g. the lack of impact in microfinance programs), as I imagine the determination of impact on these programs is much more complicated and nuanced than could be stated in such a brief manner above.
BY Godfrey Senkaba
ON November 30, 2019 10:14 PM
Thank you for this article. I particularly like and relate with the alternatives offered. Impact evaluation is ‘trendy’ in the non-profit world, and soon we shall loose its actual value and power to inform programs that better people’s lives.
Recently, I attended a project partners meeting. The donor rep was insistent on us conducting an impact evaluation, and wasn’t prepared to take no for an answer. To my surprise, some implementing partners in attendance were open to adjust the program to support the evaluation.
Yet upon, additional discussion of the rationale for needing an impact evaluation and how results will be used, it was evident the donor rep wanted data just above activity/implementation monitoring! In fact, he would be satisfied with the monitoring data you have described in this article as: targeting, engagement and feedback. Imagine the waste of resources if the we didn’t do deep interrogation of the rationale, pros, cons and alternative to impact data.
Perhaps this article should have elucidated the need and use of annual monitoring of program outcomes to compliment routine performance monitoring, as alternative to some of the needs for impact evaluation. That’s to say, data on changes in behavior, adoption of Bette practices, etc.
Even if the concept and methodology for impact evaluation may be well understood among non profits including resident expert to do it well, other practical issues may impede implementation. One such issue is ethical obligation. For the same community in need, It’s a challenge to deny service to a group of people while serving others, even if doing so may lead to better future programs.
Again, thank you timely article that frees us from the ‘impact evaluation box’, giving us options-more creatively-about improving social interventions. I will rethink my own practice and approach to M&E systems strengthening, thinking along the CART principles.
BY Richa Kohli Pawar
ON December 8, 2021 02:02 AM
This article is making do’s and don’ts points for impact measurement and impact reporting and they are practical helpful tips.
Here are some important points to be added to the impact report.
-Clear reasons for your impact journey
-Impact strategy aligned with appropriate indicators
-The key impact goals and stakeholders
-Inputs and outputs
-What actually happened? A positive and negative change
-Clear data insight from all the dimensions such as WHO, WHAT, HOW MUCH, CONTRIBUTION, and RISK
-What will be the next step? How are you planning to use your learning?
The below blog provides more insights about nonprofit impact reporting.
https://www.sopact.com/perspectives/non-profit-social-impact-reporting
Regards,
Richa