In the past couple of years, there has been increasing debate about “black box” models—very complex algorithms whose inner processes are obscure even for their developers—in machine learning. Many such models tend to be very accurate but at a considerable cost to transparency. This raises serious concerns when they are used to make high-stakes decisions that affect people’s lives. An interesting paper by Cynthia Rudin of Duke University makes an extreme case for prioritizing interpretable models in machine learning over ad hoc efforts to explain black boxes.
Rudin insists that an interpretable model, designed top-down to be understood by humans, fundamentally differs from the post-hoc explanations of black box models. Explainable AI techniques try to provide insight into how a black box arrives at its predictions, but such explanations are most often unreliable, inconsistent, or not fine-grained enough. Much more worryingly, some of those explanation methods are unfaithful to how the original model functions; this might mislead users about its actual behavior.
The paper challenges one of the common assumptions in machine learning: that there is some kind of inborn trade-off between accuracy and interpretability. Rudin argues that for most real-world problems, especially those touching structured data, it should be within reach to develop interpretable models performing as well as black boxes. She then gives examples of criminal justice and healthcare where interpretable approaches have matched or improved performance compared with opaque alternatives and computer vision.
But why, then, are black box models everywhere—and certainly very prevalent—in the most high-stakes applications? Rudin puts her finger on several: the sales motives of firms purveying proprietary algorithms, ease of applying off-the-shelf black box techniques, and lack of training in interpretable machine learning methods among too many practitioners. Still, she insists that the societal cost of running opaque models for making significant decisions outweighs these considerations.
The dangers posed by black boxes lie in the fact that these are complex systems, so the chance of accidental errors or hidden biases is high; they are opaque in that the meaning is barred from meaningful scrutiny and accountability; and they are inscrutable so that one can hardly combine their outputs with human judgment and domain knowledge, which is a critical requirement in many high-stakes contexts.
She also points out several promising directions in which more powerful, interpretable models could be developed, for example: optimized rule lists, scoring systems, or prototype-based neural networks for computer vision. She says these approaches often involve solving quite complex computational problems; however, the benefits of transparency to machine-assisted decision-making make the effort well worth it.
It concludes by calling on policymakers and organizations to demand interpretable models, where it is insufficient to settle for a black box with doubtful explanations. This might involve laws requiring interpretable approaches wherever feasible or, at a minimum, ensuring that organizations make some sort of good-faith attempt to develop transparent alternatives before resorting to opaque techniques.
