Do All AI Systems Need to Be Explainable?

Illustration by iStock/carloscastilla

As the capabilities of artificial intelligence (AI) evolve, they push the boundaries of human understanding. Instead of transparent, explainable mechanisms, many AI applications are “black boxes,” decision-making processes that are opaque or at least not easily interpretable by humans. But as government, activists, and the public at large have come to better understand the risks associated with inscrutable algorithms—say, through works like Cathy O'Neil’s Weapons of Math Destruction—calls for "Explainable AI" have begun to echo across boardrooms, tech conferences, and research labs. If an AI system makes a decision, the demand goes, humans should be able to understand how and why that decision was made.

The issue can seem straightforward. Frameworks like the European Union's General Data Protection Regulation potentially offer individuals a "right to explanation" against automated decisions; advocacy entities like the Electronic Frontier Foundation, OpenAI, and the ACLU consistently stress the significance of AI transparency, especially when outcomes have direct ramifications on individual rights; and this sentiment is echoed by consortia like the Partnership on AI, a collaboration between influential tech entities and human rights groups, emphasizing the transparent ethos in their foundational principles.

But while certain systems must be designed with a certain standard of explainability, others might not necessitate such transparency. Consider everyday tools and applications like recommendation algorithms on streaming platforms or the autocorrect feature on our smartphones: While it might be interesting to understand why a particular song or movie is suggested, or why a certain word was chosen to replace a typo, the intricate details of these algorithms aren't crucial for the average user's experience or trust in the system. Instead, what matters most is the system's effectiveness and reliability. Indeed, the pursuit of unwarranted transparency in such cases might lead to overwhelming amounts of information that users find redundant or even confusing. It's akin to asking someone who drives a car to understand the detailed mechanics of its engine: some might have an interest, but most are content knowing it will get them from point A to B safely.

The 'Black Box' Problem and the Desire for Transparency

Explainability is undoubtedly crucial in certain cases. In health care, for example, deep learning models have been used in hospitals to predict sudden deteriorations in patient health, such as sepsis or heart failure. But while these models can analyze vast amounts of patient data—from vital signs to lab results—and alert doctors to potential problems, the interpretive leaps which they can uniquely provide are a function of complex computations. As a result, the exact pathways and combinations of data points they use to arrive at their conclusions may not be clear to clinicians. This "black box" nature can make it challenging for doctors to fully trust the model's predictions without understanding its reasoning, especially in life-or-death situations. For this reason, AI models used to detect skin cancers or lesions provide visual heatmaps alongside their diagnoses, which highlight the specific areas of the skin image that the model found indicative of malignancy, thereby allowing dermatologists to understand the AI's focus and reasoning. By providing a visual representation of areas of concern, the AI system allows health care professionals to "see" what the model is detecting, enabling a doctor to cross-reference the AI's findings with their own expertise.

Another example is homeland security: advanced AI systems deployed in surveillance cameras in airports, stadiums, and other large public venues have the potential to detect security threats based on facial recognition, behavioral patterns, and more, processing vast amounts of data at incredible speeds. But while they can flag an individual or situation as suspicious, the intricate web of reasoning behind such a decision—combining facial recognition, movement patterns, and possibly even biometric data—can be difficult to fully articulate or understand. Explainable systems in this domain would tag specific actions (like an unattended bag) or behaviors (a person frequently looking over their shoulder) as indicators, rather than just flagging an individual without context. By tagging and detailing specific actions or behaviors that are considered suspicious, the AI system offers insights into its decision-making process, not only aiding security personnel in quick decision-making—and helping to avoid mistakes—but also assisting in refining and training the AI system further.

In the legal domain, some jurisdictions, particularly in the US and China, have started using AI systems to aid in determining the risk associated with granting bail or parole to individuals, analyzing numerous factors, including past behavior, family history, and more, to generate a risk score. But while the goal of protecting populations can make such systems a real asset, the reasoning leading to the decision cannot be reconstructed by humans. While they provide a risk score, the exact weightage or significance attributed to each factor, or how they interplay, can be elusive. This lack of clarity can be problematic, especially when dealing with individuals' liberties and rights.

No Simple Answers

The question arises: why not simply make sure that all AI systems are explainable?

For a start, the necessity for trade-offs between model explainability and performance flows out of the intrinsic characteristics of different machine learning models and the contradictions inherent in data representation and decision-making. Models are idealized, simplified representations of reality, which means they may necessarily sacrifice precision for tractability and explanatory power. In this sense, a model might not be fully "true" or precise, but it can still be extremely effective in understanding and predicting phenomena. The very characteristics that make some models more interpretable—their simplicity, clear decision boundaries, and reliance on fewer features—can limit their capacity to capture and utilize all available information. By the same token, models that utilize more data, in more intricate ways, can produce powerful decision-making that is, as a result, harder for people to articulate and understand.

To manage their use of AI models, leaders need to analyze and prioritize. What is the best balance between a precise, accurate depiction of reality and having a practical, effective tool for understanding, prediction, and intervention?

Here are five issues for leaders to consider as they consider what trade-offs are worth making:

1. Balancing complexity with simplicity: More complex models, like deep neural networks, can capture intricate relationships and nuances in data that simpler models might miss, which allows these more complex models to achieve greater accuracy. But the intricacy of the model can be precisely what makes them harder to interpret. Simpler models—like linear regression or decision trees—are easier to understand for the same reasons that they might not capture all the subtleties in the data.

To manage the tradeoffs between complexity and simplicity, leaders should balance risk and reward.

Analyze the potential risk associated with AI decision-making: if an incorrect decision could lead to significant harm or costs—as in health care or legal decisions—prioritize explainability, even if it sacrifices some performance. Leaders should seek to embrace appropriate complexity, making deliberate choices about complexity based on goals: to capture intricate data patterns, a more complex model might be suitable. But decision makers who use the AI outputs should always understand the model's inherent limitations in terms of interpretability.

2. Balancing generalization with particularization: Models that are highly interpretable, such as linear regression or shallow decision trees, make decisions based on clear and general rules. But these general rules might not always capture specific or intricate patterns in data, leading to potentially lower performance. Because more complex models can identify and use these more particular and intricate patterns, their outputs can be more difficult for people to interpret.

Leaders should acknowledge the importance of context. Not all AI applications require the same degree of explainability; AI used in health care diagnoses may demand higher transparency than AI used for movie recommendations. They should also educate stakeholders on model nuances. Regularly training stakeholders who will interact with or rely on the AI system’s general rules and specific intricacies will ensure that they're well-versed in its capabilities, limitations, and potential biases.

3. Emphasizing adaptability over overfit: Highly complex models can sometimes "memorize" the training data—which is called “overfit”—and in doing so, capture noise rather than the underlying data distribution. This can lead to high accuracy on training data, but it often results in poor generalization to new, unseen data. As a result, simpler, more interpretable models might not achieve as much accuracy on the training set, even though they can be more robust and generalizable.

Because adaptability to fresh data is paramount, leaders should ensure robustness over training accuracy, always assessing and monitoring the AI model's performance on new data. Moreover, leaders should seek to prioritize feedback loops, especially in critical domains: if an AI system makes a recommendation or prediction, human experts should have the final say, and their decisions should be looped back to refine the AI model.

4. Engineering can impede a model’s legibility: For simpler models to achieve high performance, substantial feature engineering might be necessary, which may involve manually creating new features from the data based on domain knowledge. But while engineered features can make the model perform better, they can also make the model's decisions harder to interpret if the transformations are not intuitive.

Leaders should choose engineered features with interpretability in mind. If your AI application requires feature engineering, ensure those features don't add unnecessary opacity, addressing the balance between enhancing performance and maintaining clarity. Enhanced performance shouldn’t compromise understandability. One solution can be to embrace a hybrid approach, merging machine with human decision making. While AI can offer rapid data processing and nuanced insights due to feature engineering, human expertise can provide the necessary context and interpretability ensuring clarity.

5. The trade-off between computational efficiency and effectiveness: While simpler, interpretable models often require less computational power and memory—which makes them more efficient for deployment—highly complex models might perform better but be computationally expensive to train and deploy. When deploying AI models, especially in real-time scenarios, weigh the benefits of model simplicity and computational ease against the potential performance gains of a more complex, computationally intensive model. Often, a simpler model might suffice, especially if computational resources are a constraint. Here, it is particularly important to stay updated and iterative: the field is rapidly evolving, and leaders should regularly revisit and refine AI deployments to ensure they meet the evolving standards and needs while ensuring models remain computationally efficient.

A Balanced Approach

The goal is not to swing entirely towards complete explainability at the expense of performance, or complete performance at the expense of explainability. It is about finding a balanced approach tailored to each AI application's unique risks and rewards, taking into account the human and environmental implications that are inextricably intertwined with the purpose of building a future we can all trust.

Read more stories by Hamilton Mann.