(Illustration by iStock/gobyg)
Imagine this: At a gathering someone asks, “Has exposure to the internet been good for society or bad?” The ensuing debate reveals varied opinions, yet the conclusion suggests that everyone agrees the internet has been good for society. This conclusion makes you uncomfortable because you witnessed the whole debate. You saw that many around you nodded their heads for a specific answer because everyone at their table did. Many of those who said there were good aspects—like having their children do classes and homework on the internet—then went on to talk about how it could be bad, and often in ways that were completely different from others. Some worried about having their children exposed to inappropriate content. Others thought that playing games and being on social media could be addictive. All these statements started with “Good, but…” and as such those responses were counted as “good,” but you heard the debate and the sound of people’s voices: some emphatic, others contemplative, and some even uncomfortable about speaking up. The overall conclusion of “good” belied what you heard, which was much more complex.
This is not a hypothetical debate. This is exactly the type of question we asked the customers of one of our portfolio companies that provides broadband. At Omidyar Network India, we have come to realize that detecting this level of nuance is critical in our quest for the next frontier in positive social impact. When looking at impact measurement, it is vital to remember that people are at the center of the puzzle. To humanize the metrics that seek to inform us about impact, it is important to look at numbers and ask two crucial questions: “What do these numbers really tell us?” and “So what?” In other words, numbers independently do not hold a universal inherent value, and therefore do not tell us all we need to know. For precise and quantifiable impact data to be an effective analytical and decision-making tool, we need to be able to layer in qualitative insights that frame these numbers within the context in which they have been generated.
Thinking about impact measurement in a way that recognizes the value of both thick and thin data has implications for both portfolio companies and investors, especially those who strive to simultaneously work towards reaching their impact and strategy goals. In this regard, a crucial opportunity often goes unnoticed: the potential of impact data to inform consumer-centric strategies.
Most companies, even early-stage impact ventures, have some sort of ongoing customer satisfaction or Net Promoter Score (NPS) measurement in place. These traditional tools serve a critical purpose—they help measure whether customers are satisfied. When deployed thoughtfully, it can help companies improve their product-market fit. However, traditional NPS also faces limitations. It is unable to dig deeper to understand if, or how, a product or service is changing customers’ lives.
Omidyar Network India, in collaboration with our tech-led research partner Decodis, piloted a new approach to impact measurement centered around deep listening with large customer samples. We ran deep listening exercises with five ONI portfolio companies—one of them two years in a row—listening to 1170 customers in four languages, generating 4420 minutes (about three days) of audio. Working with this data with each company produced results that changed the way we looked at impact measurement.
We call this new way of looking at impact measurement “impact-market fit.” This involves listening deeply to large samples of customers to understand whether a company’s go-to-market strategy aligns with its intended impact. Impact-market fit requires the combination of what people say as well as how they say it. In case there emerges a lack of alignment, as in the scenario discussed at the top of this article, it lends itself to thinking about how to make relevant adjustments—on both strategy and impact goals. In other words, deep customer listening reveals how to tweak the customer value proposition and acquisition strategy to increase both impact and business value over time. Within a relevant framework and with a large sample size, impact measurement through deep listening can power a sensitivity analysis of which portfolio company product changes will yield the greatest improvement in impact. For example, later in this article we discuss an online learning platform for job seekers, Entri. Entri could choose whether to invest in functionality of their platform, the quality of their instructors, or the type of material they provide. We found that Entri should invest in the quality of their instructors, which will increase their impact the most.
For impact investors and their investees, this concept reframes the measurement of impact not as a judgment of a portfolio company but rather as a tool to help the portfolio company reach its full potential.
Using AI-Driven Text and Voice Analysis to Conduct ‘Deep Listening’
The key to determining the impact-market fit is deep listening—to allow customers to speak freely, in their own voice and their own language. Doing this, especially over hundreds and thousands of customers, creates a treasure trove of natural language. Analyzing those voices yields a nuanced level of customer understanding, which in turn creates a finely calibrated diagnosis of how a company can extend customer satisfaction to customer impact.
One of the critical requirements of deep listening is having a large sample, which enables segmentation analysis. However, such large amounts of language data collected through deep listening is impossible to analyze manually. We estimate that a deep listening exercise with a single company can generate 500+ pages of text, which would take a researcher roughly two months to analyze. With AI-driven text analysis, however, it takes two days. This text analysis provides key insights into what large samples of customers are saying beyond whether they are satisfied. To nuance this understanding further, we perform voice analysis on the responses to reveal how the customers say what they do. For instance, customers can use positive words to describe the impact a product or service has on their lives, but they can say those words either in a dull, uninterested voice or in an enthusiastic voice. Layering voice analysis on top of text analysis helps to humanize customers’ voices and to understand their own perspectives of impact.
Impact-Market Fit Requires Active and Regular Listening
A target market is not static, especially for growing companies. The impact on earlier customers who have been with the company for a longer time can be different than those who have been there for a shorter time. Doing a cohort analysis comparing new customers to mature ones can be seen as testing a “dosage effect”—do customers using the service longer experience increasing impact?
We observed this with Wiom, a broadband company with around 100,000 low- to middle-income customers in New Delhi, India. Decodis did a deep listening impact measurement with their customers first in 2022 and then again in 2023. By the second round of interviews, the results from our AI text and voice analysis picked up that new customers broadly appreciated the continuous nature of broadband but were unspecific about how it made a difference to their lives. With mature users, they spoke more about a select few use cases more often and more emphatically than others. Mature customers felt that the stable broadband access that Wiom provides made a difference in how their children could use the internet for education and how it helped the adults run a business from home. But we were surprised to hear something else—customers raving about how much more comfortable and confident they are about doing online payments and banking with broadband than they had been with mobile data.
This all supported the notion that for some company offerings, customer impact is felt only over time rather than right away. By doing deep listening over time, a company creates a more dynamic thesis of impact, tweaking their vision of impact to fit the reality of customers. In Wiom’s case, being able to generate deeper qualitative insights was critical for understanding customer responses in the temporal context of their user journey, as well as in relation to Wiom’s dynamic impact thesis. Additionally, the strength of this unexpected benefit would not have been captured without the nuanced text and voice analytics that deep listening can bring.
Large Sample Sizes Enabled by AI-Based Listening Tool
Impact investors have both early- and growth-stage companies in their portfolios. As a business develops, understanding new customer segments can help in each go-to-market strategy. As a company continues to evolve, the product, marketing, and service is often tweaked to increase uptake and satisfaction. Deep listening is important as companies go through these stages to hone those changes toward greater impact.
Having a large enough sample to conduct segmentation analysis to develop archetypes was critical to improve impact market fit for Entri, an ONI portfolio company operating in the education space. Entri offers an online platform that provides test preparation and skill development resources in Malayalam, Tamil, Hindi, Telugu, and Kannada, catering to jobseekers across various regions. This deeper understanding of customer segments proved pivotal, particularly in grasping Entri's largest archetype and most impacted demographic: busy women balancing household responsibilities while endeavoring to enter or advance within the job market. Moreover, Entri's method of gauging impact primarily focused on tracking job placements, leaving other reasons for customer satisfaction less evident. Our deep listening analysis revealed that Entri's true value lies in improving job readiness and marketability. These results challenge conventional metrics by spotlighting its broader influence on skills enhancement, improved confidence, and expanded professional opportunities, transcending the impact of one job placement with a lifelong ability to win, hold and succeed current and future jobs. In other words, through deep listening we were able to not only identify how the impact narrative needed to shift, but also put those who described being most impacted front and center. Ultimately, this provided Entri with insights into what these customers valued and how Entri could tweak its offering and track its impact through the scaling phase.
What Is the Benefit to Investors and Investees?
Impact investors like us are attracted to investments with intentionality of positive impact on people and the planet. Over investment cycles, we all look for signals of validation through impact assessments. Analysis at investee level helps us validate our hypotheses for each investment and to reflect on our learnings. Moreover, aggregating these insights at the portfolio level helps in assessing our progress on impact goals.
Impact assessments and deep listening are learning opportunities for fine-tuning an impact investor's theses for investments. Understanding of the “Impact-Market Fit” can help craft investment themes that are meaningful for specific customer segments, with local nuances built in. For example, for the past five years, ONI’s investment approach has been to focus on improving the lives of India’s Next Half Billion (NHB)—the 500 million first-time internet users that have come online via their mobile phones. Listening to the many NHB customers reached through our portfolio companies has helped us craft locally nuanced, rigorous, and agile impact theses for sustainable businesses in India.
An Industry Shift in How We Use Impact Measurement
As an industry, we have embraced the need to measure impact but now we must move to the next frontier. For investors, this means augmenting existing impact evaluation practices with deep listening tools. Moreover, investees will need to engage deeply with the impact measurement process and demonstrate greater agility to customer feedback. Doing impact measurement in a deeper and more insightful way can help investors and investees to align on course corrections rather than solely focus on long-term metrics. The result will be more adaptable, agile investees who will respond, literally, to their customers’ voices. Investors will value and support investees’ ability to appropriately course correct, a necessary feature in early-stage companies. Investees can be more focused on making their businesses sustainably meet the needs of their customers rather than trying to chase a static set of impact measures.
This requires openness to new ideas and an adaptation of processes. But a cohesive shift towards more nuanced, human-centered impact measurement can not only standardize, yet individualize impact measurement but also unlock value in a socially responsible manner.
Read more stories by Sushant Kumar, Sarvesh Kanodia, Daryl Collins & Rohini Roy.