How Social Science Got Better: Overcoming Bias with More Evidence, Diversity, and Self-Reflection
Matt Grossmann
352 pages, Oxford University Press, 2021
Applications of social science theory and concepts are central to today’s public life, from social networks to employee trainings to voting reforms. Social science methods using big data and experiments are also of wide and increasing interest among practitioners. But do social science practices need to be reinvented for use in the real world?
In this excerpt from How Social Science Got Better: Overcoming Bias with More Evidence, Diversity, and Self-Reflection (2021, Oxford University Press), I argue that the social sciences were born to guide our practical ambitions and are improving in their ability to guide decision-makers. But their primary area of application is in multidimensional public policy choices, which inevitably combine the uncoverable patterns of social life with our value choices. Because our theories and pursuits stem from our collective goals, we need to reflect on the history of applied social science, which has often had too much hubris and too little diversity. There is no replacing the complexity of social science or the difficulty of building codified knowledge for use by society. But the social sciences are becoming more relevant, diverse, and reflective, learning from their history and public use.
The following excerpt comes from the beginning of Chapter 9, after a review of current practices in economics, psychology, sociology, political science, and anthropology and just before a history of the sordid role of social science in the eugenics movement and a mixed assessment of its role in the technology sector. Across disciplines, the social sciences are improving their methods, taking advantage of new data sources and more diverse scholars. But that does not imply any quick fixes for tough social problems. Instead, learning about social life with the tools of science is a long slog with slower accumulation and more controversial application than the natural sciences. The way forward is recognizing the progress in addressing our inherent challenges.—Matt Grossmann
+ + +
WIRED magazine editor Chris Anderson declared “The End of Theory” in 2008, arguing that “big data” had rendered traditional scientific theory and testing irrelevant. Even for the often-triumphalist magazine, Anderson’s claims were quite over the top. Examples like Google’s advertising and search algorithms, he claimed, showed that the scientific method was obsolete. With enough data, correlation was enough to assess causation. And near infinite experiments could be run without explaining or predicting their results. The explosion of biological and genetic data was showing that the tools of the technology industry would soon reign in science as well.
The thesis was “deeply wrong,” political scientist Gary King told me. There is “no way to eliminate theory because you need a lens” to look at easily downloaded data. We have made progress in one part of the empirical end of science: we have a lot of new measurement tools that allow us to “estimate new things, to gain in one specific area.” Indeed, social science still needs models to interpret and generalize data. Anderson’s claims were odd even for Google itself, given that its search engine was created by graduate students applying sociological theories of citation networks and its advertising sales market was the product of economists advancing auction theory. Google itself shows that the words we use and the connections we make between them are growing every year; the search engine is forced to evolve with us.
From racial bias in photo recognition software to gender biases in automated résumé reviews, we are learning that biases in initial training data can be codified by algorithms that supposedly let data speak for themselves. If you rely on pictures that are not representative or try to get software to emulate your prior decisions, computers will do what you ask them to do but will do so based on the letter of the code rather than the spirit of your intentions.
Many errors come from researchers’ qualitative wrangling with their initial data, rather than the computer’s search for solutions. Since these algorithms drift forward (incorporating their initial steps) and eventually put many of their choices in black boxes, they can hide the biases that we might uncover in traditional data analysis. As data scientist Andrea Joes-Rooy points out, data science projects perpetuate (if they do not uncover) “systematic errors, errors of choosing what to measure, and errors of exclusion.” Advances require close looks at the data-generating and data-analyzing processes. The availability of big data makes it more likely that scholars make false discoveries while making basic data literacy more important.
Despite the hype and the dangers, there are real gains. And some stem from productive exchange between scholars and real-world appliers of social science knowledge. Many of the industries of today (and tomorrow) are built on social science insights and scholarly tools. Social scientists want our studies to prove useful, both to help institutions carry out their current responsibilities and to transform or replace them to better achieve their aims. Political scientist Adam Berinsky, who has worked directly with technology companies on research, says they are now very interested: “I like talking directly to the platforms for real impact. The 2016 moment, it made them real interested [in issues like misinformation].” And reporters followed: “Now they want to know the research. Reporters are more interested in learning what do we actually know.”
And it turns out research is almost always conducted in the context of practical concerns. Social science was divorced from history not for the sake of general knowledge curiosity but to intervene in social life. In considering our goals, scholars need to account for how our desire to apply knowledge affects how and what we study.
The social sciences did not develop as pure basic sciences, only to later be transformed for applied pursuits. Instead, practical motivations were there from the beginning—and those aims were not always defensible. Most glaringly, American social science played a central role in promoting eugenics, the racism-inspired view that humankind should direct its reproduction to increase desirable genetic characteristics. That history is important not just for historical accounting, but also for how we should interpret our motivations today. If we can see that the racist impulses of prior generations led to poor research design and misinterpretations of evidence, we can also see how our desires for social change and usefulness influence our conclusions today. Thankfully, scholars are becoming more cognizant of the trade-offs among their applied and scientific goals.
Social science was put to practical use before it became a scholarly pursuit. John Graunt, the seventeenth-century pioneer of demography, produced public health statistics in demographic categories with the aim of comparing social systems. His friend William Petty, one of the first economists, was concerned with critiquing groups that were public charges, influencing fiscal and trade policy, and justifying English action in Ireland. In doing so, they made lots of estimations from limited data and assumptions about cultural group differences and values. They often estimated wildly from London statistics alone, for example, and then assumed its stereotypical differences from elsewhere.
Systematic social statistics were first collected to influence policy and reflected the assumptions and aspirations of their political goals. As historian William Deringer put it, “Britain’s new quantitative age was not fashioned by dispassionate scientific practitioners seeking ‘objective’ knowledge about society or the economy, nor by diligent bureaucrats trying to advance the interests of the state. Rather, political actors of various stripes, from eminent ministers and members of Parliament to hack writers and out-of-work accountants, found that numerical calculation offered an especially useful tool for carrying out political arguments.” They were especially important in partisan political debates, with each side investing in expertise.
Government statistics came from core debates over economic policy concerning redistribution, war, and depression. State (referring to government) is the root of the word “statistics,” as it initially referred to aggregate information about nations, especially the character of their aggregate social life. But individuals and private organizations preceded government in data collection. Data collection reflected prejudices about social groups as well as attempts to deflect responsibility and compare one’s government and society favorably to those in other places. Even “pure” mathematics advanced primarily through attempts to solve particular human problems.
As historian Theodore Porter finds, the broad role of social science in practice continued into the nineteenth and twentieth centuries: “statistics was itself a social science for most of the nineteenth century, and in many ways it was the prototype of empirical, problem-oriented social science. Law, administration, poverty relief, public works, crime, even revolution were all topics of social science, as practiced not just by academics but by officials in treasury ministries or bureaus of labor and trade, by prison superintendents, poor law commissioners, public health officers, and other state bureaucrats, as well as reformers of all kinds. These men and, in growing numbers, women were seen not merely as applying or dabbling in social science, but as practicing it.”
Social science helped to institutionalize the idea of public social problems in need of solution through the invention of aggregate social data, such as crime and employment statistics. Statistics were developed and advanced as much through applied social science as through natural science or mathematical questions. Early social science helped to advance war planning, census and bureaucratic administration, and city life, in the process moving quantitative comparisons of value and social categorization to the forefront of public life. That, in turn, made it important to intervene on what was measured and how, which both advanced the science of sampling and made it more likely that measurements would be challenged on criteria of what worked in political disputes.
The institutional development of economics was driven by railroad expansion and repeated economic crises. From the beginning, academics and regulators were learning the lessons of the prior crisis, which created problems if the circumstances surrounding them (such as a bank run or a commodity price drop) changed each time. Early economics also showed that failures of prediction did not necessarily deter additional input: if a new crisis developed, economic thought and models were still needed to address it. Economists shifted with the times and retained their influence.
Social science in Western Europe remained more politically diverse and bound by traditional practices of historians, but American social science came into its own role attached to new social professions and policy reformers. The first generation of discipline-based social scientists, according to historian Dorothy Ross, were motivated to “reconfirm the traditional principles of American governance and economy” on the basis of scientific knowledge. They reacted to the problems of the Gilded Age, including the socialist threat, but merged apparently timeless principles with American aspirations, hoping to secure and justify American progress.
Economists have taken the lead in driving policymaking, first convincing policymakers to focus on economic growth and development and only later growing concerned with addressing inequality. Economists changed common policy ideas and arguments on both the ideological left and the right—in both public debates in the media and behind-the-scenes analyses just for policymakers. Even economic ideas that sounded morally questionable to the public, such as valuing human lives, become standards as all sides of debates saw incentives to quantify their preferred costs or benefits and add new considerations to decision-making models.
Psychology gained public status based on its assumed usefulness in personal growth and solutions to public problems. But it has always borrowed from popular concerns: words like “learning,” “perception,” “depression,” and “stress” all entered disciplinary research early with their ordinary meanings and continue to have the highest usage in psychology journals. Psychology history shows that practical applications often came before theoretical and methodological innovation, with basic research later adopting tools from applied work.
The popular vision of the scientific approach became more circumscribed. Widespread ideas about the contents of the “scientific method,” including the use of that terminology, were developed in popular science writing and via commercial applications, rather than drawing the boundaries for scientific research within universities. The more that science could be broken down into a clear set of followable steps (such as hypotheses, experiment, and confirmation), the more it gained public acceptance, even if real scientific disciplines were messier and contained more diverse sequences. By codifying naturalistic scientific steps, social science sought to lead social improvement.
Health eventually became the central application of science. As federal and private health spending expanded dramatically, it became the dominant source of scientific funding and concern. Health research funding is now ten times greater than general science funding; partially as a result, applications to health have come to dominate research and even basic science prizes. Until the 1980s, most Nobel Prize solicitations were for simplifying through broad theory, but they have increasingly moved toward constructing applications, even in physics and chemistry. But the medical profession still maintains sovereignty over the application of health ideas. Even with the growth of evidence-based medicine, it has been difficult to stop procedures that are known to be worthless.
Most ideas about science application still involve a model of stages where ideas move from basic to applied to development research. Social science helped build these models to justify investment in research, not only within economics but also in the anthropology of innovation and the sociology of modernization. But social science lacks a “development” field because its applied work is rarely associated with products for industry. Technology and product development constitute proofs of scientific success in the public mind; credit eludes social science for social technologies, even those in wide use like messaging, surveys, deliberations, trainings, and software. Much of the technological in- novation in energy, public health, and transportation, however, comes from gradual, recombinant, and cooperative efforts based on small improvements and work with practitioners, rather than predesigned development sequences. Practitioners can often see recent trends and shifts in acceleration, while social scientists are focused more on stable patterns.
Humans can also react to social science knowledge by changing behavior, especially in policy and widespread practice—and not always to the benefit of social science. “When a measure becomes a target, it ceases to become a good measure,” goes Goodhart’s Law (as written by Marilyn Strathern). That means once social science identifies a goal, whether it is low poverty or high subjective well-being, the measurement of that goal becomes gameable and politicized, undermining its value. The more we demand that social science provide actionable prescriptions and measurable progress, the less it can objectively describe the social world.
But social scientists still see value in their research having a role in informing public policy. Although less enthused than natural scientists about directly developing products and services, they observe and champion increasing policy-relevant knowledge. My survey of major research university professors in the social sciences asked 1,141 respondents to assess whether their disciplines were becoming more or less policy relevant. All disciplines perceive policy relevant research as increasing. This trend is perceived at similar levels to the rise of randomized controlled trials and network data, though is reportedly less pronounced than the rise of big data. Compared to other indicators, this measure has lower variation across disciplines.
The question asked respondents to consider the last decade, so it is possible that social scientists would still recognize that their disciplines have a long his- tory of practical research. But the examples scholars gave suggest that they also perceive their recent shifts to policy standing out. Rather than the lists of policy recommendations and arguments that used to form the basis of political science articles in the early twentieth century, for example, today’s policy-relevant research includes studies of how policy is implemented in different states, whether existing policies reach their professed goals and have unintended consequences, and whether they feed back positively by helping the politicians who pass them.
This policy-relevant research is important for basic social questions, rather than a sidelight, and draws more explicitly from social science expertise and data. The practical ambitions of social science do not necessarily set it apart from natural and biological science. There are many engineering disciplines that focus on translating science into applications, including some with clear social goals like civil engineering and conservation biology. Disciplines can clearly apply scientific ideas to social goals like bridge construction and maintaining biodiversity without sacrificing their claims to objectivity in conceiving or applying knowledge. But the policy focus of social science, especially without a guiding light beyond social improvement (infused with liberal ideology), deserves more introspection. There is certainly reason from social science history to be skeptical that it can pursue social application without prejudice.