(Illustration by Vreni Stollberger)
Across industries, a turn to artificial intelligence (AI) has become ubiquitous, if not completely hegemonic. It’s not just Meta, Google, Apple, and Amazon—nearly every large company has pivoted to AI. Logistics companies like DHL have rolled out AI-powered logistics management; Walmart has turned one location into an Intelligent Retail Lab; Citibank has begun supplementing person-to-person customer service with an Intelligent Virtual Agent. Meanwhile, startups proliferate, promising AI-powered everything: improving one’s writing, detecting contraband in baggage, inferring emotions from one’s face, monitoring remote employees’ actions, predicting fraudulent payments, or even generating art.
Unfortunately, alongside its increasing omnipresence, much of the AI developed by large corporations has exhibited a myriad of issues. There are countless public instances where AI was deployed with overtly racist, sexist, and discriminatory outcomes: Facial analysis algorithms used by law enforcement have misrecognized at least three Black men; an internal hiring tool at Amazon categorically excluded women based on facially neutral terms on their resumes; an automated tool allocating health care resources for 70 million Americans discriminated substantially against Black patients.
Even before deployment, these models require huge amounts of data to be developed in the first place. And, in an environment with insufficient data regulation and transparency, data is often obtained via questionable means. For instance, facial recognition AI, used for things like crowd surveillance, autonomous driving, or identity verification, was found to be built using individuals’ faces without their explicit consent. This is hardly an outlier; much of foundational image recognition AI was built using photographs posted on the photo-sharing website Flickr, where users were unlikely to have anticipated that researchers would pick apart their content and use it to build pervasive computer vision technology. And while many unsavory data collection practices remain evasive to regulation in the United States, some companies have reached the limits of this gray area. The Federal Trade Commision, for example, recently required that one firm delete all of its AI models and their associated data after the company illegally collected data on children without parental consent.
These problems emerge due to a variety of factors—including the racial and gender homogeneity of technical teams, the lack of representative data, the lack of transparent data auditing processes, and research agendas formulated without community input.
A better future for AI is possible—an AI future that isn’t motivated by vast accumulations of data gathered through the poorly regulated extraction of personal information or profit motives that are centered around maximizing and appealing to shareholders quarter after quarter. AI—and more broadly, technical systems that have disproportionate impacts on social and political life — should not be developed by entities that utilize data and model-building practices to the detriment of marginalized communities and the environment.
Our intent, as the first full-time employees of the newly formed Distributed AI Research Institute (DAIR), is to find a new path for technological research, which centers communities impacted most by emerging technologies. In this article, we examine some of the problems with AI and highlight ways in which existing research structures are unable to address these harms. We conclude by outlining a path forward for social change leaders to center the public interest when they fund and use AI research.
Case Study: Large Language Models
In recent years, a class of “large” machine learning models have emerged to address problems around language and visual media. But rather than mitigate problems associated with AI, these models exacerbate and scale them to a much larger set of problems. Moreover, by virtue of this scale and the amount of data needed to train the models, they create perverse incentives, such as collecting as much data as possible, centralizing computing resources, and luring scientists who could otherwise be doing basic research out of academia. These incentives have given rise to a rash of AI-centric startups, which have acquired funding from venture capitalists and other large-money players in Silicon Valley.
Large models take enormous amounts of data—such as billions of Flickr images, YouTube videos, Wikipedia articles, tweets, newspaper articles—and process that data using massive networks of computers. This process results in large mathematical models that are then used for a wide variety of tasks, such as automated image classification, natural language generation, and speech recognition. In contrast, smaller machine learning models might, for example, have a targeted goal of identifying a particular kind of bone abnormality in an X-ray image, and this task may be successfully accomplished with the limited computational resources of a university lab. These much larger models, however, have goals as broad and complex as “meaningfully answering any possible text question” and require resources far out of reach of all but the most well-funded corporate institutions and academic labs.
One such model is the large language model, or LLM. LLMs are models that can respond to prompts with human-like text in a variety of forms, such as essays, answers to questions, or chat responses. Boosters of LLMs believe they are a stepping stone to “Artificial General Intelligence”, or AGI, a “consciousness” that could be installed in robots and other artificial agents. Think WALL-E, or Lt. Commander Data from Star Trek: The Next Generation.
In a recent piece in The New York Times Magazine entitled “A.I. Is Mastering Language. Should We Trust What It Says?” Times contributing writer Steven Johnson writes approvingly about the development of one of these models, GPT-3, speculating that with its introduction, “We could be on the cusp of a genuine technological revolution.” At DAIR, we reject the framing of these models as “mastering language” on the merits, and agree with University of Washington linguist Emily M. Bender, who has written a more full-throated critique from the perspective of an experienced linguist: These models are not mastering language, but are oriented toward replicating patterns of existing text. Bender, along with PhD student and natural language processing researcher Angelina McMillian-Major, and former Google Ethical AI Team co-leads Timnit Gebru and Meg Mitchell, offered an extended critique of these models. In addition to the “parroting” criticism made above, these models have devastating environmental costs, are trained on data rife with religious, gender, and racial biases, and can expose private information from individuals. Critically, the data on which these large commercial models are trained on are often not public, which prevents third-party auditors from assessing their suitability for downstream tasks. While these models have been framed as “foundation models,” implying they can undergird all sorts of AI-related tasks, we ask: What are these models a foundation for? More importantly, how can these ever be stable foundations, when they are rife with the existing issues presented above?
We extend Bender et al.’s critique to move the frame of abstraction to a broader view, and note how we also need to critique the organizations creating these models, with an understanding of the political economy and flows of capital that they are garnering and generating. For example, OpenAI, an AI-centric organization funded by Peter Thiel and Elon Musk, was initially founded as a nonprofit with the stated goal of “ensur[ing] that artificial general intelligence benefits all of humanity.” However, in 2019, they changed the basis of their organization, setting up a peculiar institutional arrangement of having an exclusive license with Microsoft. But this is not even the most pernicious institutional setup around LLMs. There’s been a gold rush to get into the LLM game, with intense concentrations of venture capital funding these technologies and a mass exodus of Google AI researchers heading to join startups in the same nontransparent and unregulated space. It’s not clear how these institutional setups can be well-suited towards the fulfillment of technology in the public interest, especially as massive flows of capital accrue to a narrow set of institutional actors concentrated in the Global North, and more specifically, in Northern California.
Limitations of the Tech Industry
With alarm bells finally being rung around bias in AI tools, most prominent tech companies have codified their principles for the ethical creation of AI. Many of them have established teams around responsible AI and tech innovation. These principles and teams promise that their AI will “be socially beneficial,” “perform safely,” “be accountable” to relevant stakeholders. IBM puts it poetically, “The benefits of the AI era should touch the many, not just the elite few.”
However, these commitments to ethics are hollowed out by vagueness and legal hand-wringing—in practice, they’re often merely commitments to maintaining public image and mitigating future public relations disasters. In the absence of regulation for algorithmic transparency, impact assessments, and protection for data subjects and consumers, enforcement mechanisms are left internally to these companies; all while market fundamentalism, techno-solutionism, and hype around AI push Big Tech companies and AI-enabled startups to overpromise and undercut their ethical commitments. Even their meager promises are abandoned when it’s profitable.
These companies emphasize “responsible” AI development, echoing the language of the prior era of corporate social responsibility (CSR) efforts. Undergirding the effort to develop “responsible” or “safe” AI implies that AI is inevitable; that to create new technology, we need AI. These companies imply that AI must be a critical part of working out the inefficiencies of a global economy—it is the only vector of progress. Moreover, using the language of responsibility allows companies to acknowledge that potential harms exist, but without real accountability to those who are harmed. Stakeholders are defined narrowly, or in ways that acknowledge only shareholders, possible litigants in jurisdictions with more aggressive privacy and data protection law, and policy makers who may draft more punitive legislation that affects their bottom line. Metrics that consider “how much” harm has been done are narrowly scoped, allowing firms to claim that the benefits of innovation are responsible, compared to harms that go unmeasured and under-defined.
Smaller, AI-first companies replicate this problem by making appeals to “AI safety,” which is the belief that the development of AI is inevitable, but that it must be shepherded to avoid its devolution into a doomsday scenario. But the question remains: safety for whom?—especially when these models cost millions of dollars to train, emitting tons of greenhouse gasses into a world already in the throes of climate catastrophe. These models are already not safe for many living in the most rapidly impacted regions of the world, predominantly in the Global South, hundreds of thousands of whom have been made into climate refugees.
CSR efforts are far from sufficient. For Big Tech companies, their sheer size and market control can insulate them from public outcry when AI systems fail. Moreover, these responsibility efforts are often abandoned when they conflict with lucrative contracts and attractive fields of research. Google’s own principles—developed in response to massive employee resistance to a Pentagon contract known as Project Maven—are currently being ignored by their desire to bid on a new multi-million dollar cloud contract with the US Department of Defense. For their criticism of large language models, Gebru and Mitchell were fired from their positions as ethical AI leaders at Google, even though this is what they were hired to do.
A Future for AI in the Public Interest
We need a different path for AI. DAIR, as an AI institute, starts from the peculiar position of acknowledging that AI is not inevitable. AI is not even truly a coherent body of methodologies—as Meredith Whittaker and Emily Tucker have highlighted—let alone a singular path along which we must progress. We need different models of technological work that center marginalized communities if we are to have public interest AI. This means taking time to do slow, thoughtful work, which is directed and executed by people typically not prioritized in the tech industry: Black people, Indigenous people, people of color, low-income people, those in the Global South, disabled people, and gender and sexual minorities.
For instance, in recent work published at the machine learning conference NeurIPS by DAIR fellow Raesetje Sefala, she and her collaborators, including DAIR founder and executive director Timnit Gebru, spent years developing a dataset and a method for detecting and understanding neighborhood desegregation in South Africa in the years following Apartheid. Sefala, who herself grew up in a segregated South African township, led the effort to annotate satellite photos of different neighborhood types to understand how much and to what extent these neighborhoods have desegregated. Urban planners and policy makers can use these data to understand what works, and hasn't worked, to reverse the lasting effects of Apartheid.
In another example, DAIR fellow Milagros Miceli and her collaborators have worked with data laborers in Argentina and Bulgaria at data annotation and Business Process Outsourcing (BPO) firms. These data laborers undergird cutting-edge advances in AI by providing the raw materials with which these technologies are built. Technologies like object recognition or self-driving cars would not be possible if not for their labor. Workers at these firms are paid incredibly low wages as they spend their days attaching textual descriptions to images or selecting individual objects from larger images. Miceli’s team has found that power relationships are largely asymmetric, that instructions are very difficult to follow, and workers need channels to report their grievances to managers. Recent reporting by Karen Hao (in which Miceli was consulted) has discussed how data annotation companies drive down wages for annotators, forcing individuals in inflation-racked Venezuela into more and more precarious wage positions. Miceli’s collaborator Julian Posada has highlighted how entire families in Venezuela and Colombia have been drafted into doing work on these platforms.
Both of these projects highlight what it looks like when researchers design their projects to start from the existing social inequalities—in the former case, the violence from racial apartheid and colonialization, and in the latter case, economic crises exacerbated by the new market for data that fuels AI—and develop actionable research that can be used to improve life chances for people living under those regimes.
What Now?
For tech to work in the public interest, we need social change leaders to be attentive to what kind of dynamics they are enabling when partnering and funding technological research projects. To that end, it behooves us to ask several questions of ourselves and potential partners prior to acting:
1. Does this project put more resources into data collection and reinforce existing centers of technological power? Similar to the question posed by Stanford AI researcher Pratyusha Kalluri, this question asks which institutions are receiving investment and whether this is to the detriment of other organizations and collectives. Are resources going to the dot-orgs of large tech companies? Or are they going to groups who are underfunded and undervalued?
2. What is the composition of this research team? Too often, researchers adopt a parachuting model in which they drop into communities with a ready-made solution, which is ignored or, even worse, does more harm. While funding and flows of capital are relatively easy to allocate, trust is difficult to earn and easily lost. Do project teams contain individuals who are trusted by (and even better, are members of) affected communities?
3. How are resources being distributed among people affected by these technologies, and what kinds of knowledge does this privilege? Too often, even in situations where researchers are attentive to the concerns of affected communities, this can result in an asymmetric relationship in which researchers extract knowledge and skills from communities while not offering them anything in return, what New York University researcher Mona Sloane has called extractive participation. Are communities being compensated? Are they retaining ownership to the outcomes of that research? Do they have any means of actionable recourse if (and when) things don’t go as planned?
4. Does AI need to be part of the solution here? Counter to the dominant narratives of Big Tech and AI-first firms, we think that AI is the solution for a somewhat narrow set of social problems. Do we need to spend time developing AI tools for this problem? What are more effective, lower-tech solutions that could be considered first?
DAIR and our partners are spending time being deliberate about how we conduct research. In the next year, we aim to deepen the above questions and establish a research philosophy for doing work in a way that includes people affected by the harms of these technologies, compensates people who contribute knowledge, allows shared ownership and governance in resulting tools and data, and builds power in communities. Put another way, we will focus on technology and research that could work in the public interest. We broaden our definition of what skills and ways of knowing are deemed valuable in order to center those who are caught fighting and resisting the network of emerging technologies that constitute AI; we know that people who are harmed by technology can be intimately aware of its nuances in a way the designers of that technology are not.
In a word, to truly have public interest AI, we need technological development that is free from corporate influence and, instead, centers people who are typically at its margins. We hope you’ll join us.
Support SSIR’s coverage of cross-sector solutions to global challenges.
Help us further the reach of innovative ideas. Donate today.
Read more stories by Dylan Baker & Alex Hanna.