(Illustration by iStock/Nuthawut Somsuk)
Generative AI is transforming entry-level jobs faster than most workforce organizations can adapt their trainings. Indeed, rushing to adopt AI tools risks deepening the inequities these organizations exist to address. How can workforce providers and higher education adapt their offerings quickly enough to help prepare learners better for the changed world of work, and can Gen AI help them do it?
Our organizations—Per Scholas and the American Institutes for Research (AIR)—have been early adopters of AI in the workforce field, and we see its potential to increase efficiency and deepen learning supports for those who need them most. But our optimism comes with a healthy dose of skepticism. We do not want to adopt AI for its own sake; we must show that it advances our mission to expand economic opportunity. Which AI solutions are a good fit and actually add long-term value, instead of just creating noise? Which ones amplify progress towards organizational mission versus draining resources?
To avoid the hamster wheel of endless pilots, we are pairing AI adoption with intentional design and rapid-cycle learning, to build a flywheel of continuous growth and adaptation, grounded in data and evidence. Five years into our partnership across multiple AI pilots, we’ve reached some important insights into what it takes to reach this north star: an intentional and iterative approach to AI adoption, a culture that sustains responsible and effective integration of AI, and how to fund it.
Going Slow to Go Fast
Being an early adopter doesn’t mean we need to go at lightning speed. In four key areas, we’ve found that slowing down implementation allows teams to learn, iterate, and achieve stronger long-term results.
1. Being explicit (and getting perspective) about expectations. Passionate, experienced staff are essential, but passion—as well as pressure to stay current with the latest AI tools and use cases—can create blind spots.
Getting help from an external rapid-cycle learning partner can help address this risk. A big value-add of the Per Scholas–AIR learning partnership has been rich discussions to co-develop a detailed theory of change and logic model up front. These discussions not only clarify key outputs and why they matter, help to surface the pathways through which change is expected to occur, but also make implicit assumptions explicit. Treating the theory of change as a living document to revisit has also helped us document our real-time learning, as well as encouraging honest conversations on when and why we pivot.
2. Using data to change assumptions to facts. Meaningful metrics around expected outputs and outcomes need to be designed early. We have found it useful to start with ambition, defining the indicators we want for decision-making in the form of both “leading indicators” and lagging (final) indicators. Both are important. True mission amplifier outcomes—like job attainment—take too long for the pace of change we need. We therefore need to identify meaningful leading indicators that can help us measure, test, improve, and pivot in new directions.
3. Finding low-hanging fruit. After identifying the ideal, we get really practical. We review existing data practices to identify what we have and where there are data gaps, mapping closely to the theory of change. Perfect cannot be the enemy of good, so we begin first with what we can collect with minimal system changes and then integrate that data into operations. It’s useful to log the ones we can’t address as items to prioritize for future data enhancements and/or episodic studies. Rapid-cycle learning partnerships help teams use existing data and complement these with additional data collection (interviews, observations, pulse surveys) to generate richer insight, as well as creating space for understanding results and implications.
4. Focusing on high leverage points. “Build it and they will come” happens more frequently than it should in social impact programs, not only at the program level but also as it relates to AI tool adoption. We have often observed uneven AI adoption during early discovery, where the main barrier was not resistance, per se, but simply a desire to continue using AI tools they were already familiar with. That finding is feeding into concerted design and testing of different implementation and technical assistance strategies. Getting the small stuff right increases the odds of down-the-line impacts.
Creating and Sustaining a Culture of Learning
Agile, data-driven adaptation of work requires a culture change. It also requires rethinking how we do our own work, whether that means creating new opportunities directly or understanding how to do it well through research.
Three key ingredients seem necessary to create the kind of culture of learning that can accomplish this:
1. A shared north star for focus. Per Scholas’s north star has been “job, jobs, jobs,” and focusing on impacts and job placement provides clarity and helps align different initiatives. This accountability—not just around training but also around jobs—has been a critical impetus for understanding how AI is changing the labor market and how to leverage AI to find solutions. It has motivated Per Scholas to be an early adopter of initiatives that embed AI literacy and applied AI tools directly into technical training to help learners graduate as AI natives. It has also sharpened the focus on staff AI literacy and the use of AI to accelerate learning through career coaching, employer partnerships, and wraparound support.
At AIR, a commitment to actionable research delivered through integrated learning partnerships has led to a similar focus. In research, rigor has sometimes been used as an excuse not to tackle hard questions, but focusing on the most urgent questions that decision makers currently face, and then finding the best possible way to answer those questions while being transparent about the limits of the strengths and weaknesses of our findings This approach has allowed us to be strong partners for real-time learning.
2. A culture of pivoting in response to data. Sometimes innovations and rapid-cycle learning are useful in and of themselves but fail to add up to bigger changes. It is therefore crucial to zoom out and ask, “What happens if we are right/wrong? What will that change? What would we do differently if we were doing it again? What do we carry over to other initiatives?”
Learning from “null results” can only happen when the organizational leadership and culture reward honest conversations around “failure.” Cross-initiative learning offers a powerful example. In our first experimental study collaboration on leveraging an intelligent tutoring system, we saw strong gains in learner certification. However, nearly half of learners never used the tool. That insight shaped our current pilot on using AI for job placement, and broader AI strategies at Per Scholas.
3. Don’t silo the innovation and learning function; build core skills that enable agility. Per Scholas has been fertile ground for intentional AI adoption and rapid-cycle learning because of how the organization thinks about innovation and data-driven decision-making. Instead of being nested in an innovation team, expectations around innovation, piloting, AI adoption, and data-driven decision-making are scattered across the organization. This approach requires organizational investment in building staff capacity as well as good design to spur participation and engagement.
On the research side, this approach requires building staff capacity for understanding the nitty-gritty of program partners’ implementation, systems for tracking evolution in programming, and the capacity for frequent adaptation of research designs to operational realities.
How can funders accelerate AI adoption to drive opportunity for all?
Not all organizations are fortunate enough to have the capacity to invest early in AI adoption and learn intentionally about what works and doesn’t. Here are a few areas of investment for philanthropy to consider so that AI helps lift all boats equally:
1. Bolster data infrastructure: The field is awash with AI tools and calls for proposals to either build more AI tools and agents or reduce the costs associated with existing ones. What is also needed is investment in the data infrastructure: data lakes, enterprise-level data systems, and integrated data from across systems, that allow AI tools to draw on the right data at the right time. As an AI leader at Per Scholas recently shared with us, “Data is in too many places. The promise of technology for so long has been the elimination of data silos. What we’ve concocted throughout every workplace is the hardening and scaling of data siloes through technology. We create so many different places where information lives.”
Now is the time to understand why such data siloes take root. It is also the time to experiment with how to support organizations in building integrated, usable data infrastructure.
2. Create and scale meaningful indicators that drive action: If we want organizations to pivot in real time based on data, we need more investment in defining meaningful indicators. This is especially important not just for achieving long-term outcomes but also for identifying “leading indicators” that might serve as early warnings systems to predict outcomes.
Organizations like Per Scholas are doing these one-off, in-house and would value shared learning on which indicators work for what. Shared investment in developing and aligning data indicators within and across programs would allow more opportunities to leverage existing data for core component analyses.
3. Support and create space for rapid-cycle learning: Practitioners and funders alike often equate research with impact evaluations and implementation studies that measure long-term results. These are important but not sufficient for the real-time transformative adaptation organizations need today. Bringing similar rigor and deep thinking to the program design and initial implementation stage is key. Early discovery activities—such as data analysis, user research, and landscape—can substantially strengthen program design from the start. This early investment is far more rewarding than trying to understand after the fact why a program didn’t work.
4. Create communities of practice to scale learning efficiently: Both Per Scholas and AIR receive requests from other nonprofits regarding how to leverage AI, suggesting that there is a space for philanthropy to help accelerate efficient AI adaptation and adoption. Peer learning is especially valuable because not all organizations can afford to learn by doing. Shared learning helps conserve scarce resources and remove friction as organizations modernize. For organizations that are more advanced, more tailored communities of practice that allow cross-learning across more mature organizations are needed so that proven programs can learn just as much as they share and teach.
We know that maintaining and growing access to economic opportunity will need all our ingenuity. AI can be a powerful accelerant but only if we marshal it well, truly creating a flywheel of collective impact. We hope a proactive approach to shared learning can help us and our peers meet the moment.
Read more stories by Samia Amin & Tamara Johnson.