Too often, trust in an AI system is equated to explainability. What does the algorithm do? How does it work? Can I explain its outputs reasonably simply? If so, we tend to consider an AI system trustworthy. If one is building a recommendation engine that suggests to a salesperson a particular action as next best, one must understand how the algorithm is reaching its conclusion. What data and information is it learning from? And how can we effectively communicate this to the salesperson so that they trust the answer and act on it? Similarly, if one is predicting which customers are likely to churn (or demonstrate increased propensity to buy), then one must consider why the model is suggesting this and how to prevent churn.
Explainability is a discipline unto itself, with leading academics and practitioners working to decode the inner workings of ever more complex algorithms. Along with explainability, another rising field is the study of AI bias. Bias usually stems from the data sets used to train AI algorithms. Are these sets truly representative, or do they disadvantage a subpopulation via underrepresentation? Do they propagate poor decision-making by training algorithms on suboptimal behavior from the past? Bias can rightfully erode trust in AI systems. And it need not only arise in complex systems — even simple AI applications can demonstrate bias due to bad data or other flaws.
But sometimes trust has nothing to do with the algorithms or the data – it exists outside of the AI system. Let me illustrate.
I recently visited Haiyi Zhu, the Daniel P. Siewiorek Associate Professor of Human-Computer Interaction at Carnegie Mellon University. As part of my visit, I heard from a few of her doctoral students. One of the presentations was by Tzu-Sheng Kuo, who analyzed a housing assignment algorithm deployed in a mid-sized US city. As Tzu-Sheng told me, AI is increasingly playing a role in public services. For example, the allocation of housing services is driven by an algorithm. This algorithm uses personal information to predict the likelihood of an individual making emergency room visits, being a mental health inpatient, or finding themselves in jail. Based on the likelihood of one or more of these distinct events, the algorithm provides a score, which a case worker then uses to determine the individual’s relative rank on a housing waitlist. One could argue that the tool is objective and not biased towards a certain race. It is also explainable – we know what factors lead to a higher or lower ranking.
This mid-sized city is not the only one which uses a similar algorithm. Articles such as “Who’s homeless enough for housing? In San Francisco, an algorithm decides” or “Will algorithmic tools help or harm the homeless?” address the effectiveness of these algorithms in San Francisco and Los Angeles. The common subtext is that the algorithm itself poses risks. Among them is labeling: to identify someone as high risk may ensure that they remain high risk. If an individual is deemed to be high risk, then they may not receive certain housing services, which may reinforce behaviors that make them appear even more high risk in the future. Also, in some social workers’ view, the score now replaces what they would do – talk to unhoused individuals to determine their unique situation and needs. Another concern raised is that such algorithms shrink the problem to suit the scarcity of resources as opposed to working to solve the problem comprehensively. In response to these concerns, in the first article mentioned above, Joe Wilson says, “No computer is going to help me decide the worth of another human being, and who gets what, when and how much.” Joe is the executive director of Hospitality House, a community-based organization in the Tenderloin in San Francisco that provides services for people experiencing homelessness.
In this case, the housing algorithm is highly explainable, but many professionals would call it unhelpful or even counterproductive. Even if the algorithm was perfect and the data was current and representative, all its other issues would remain. Can it really be considered a trustworthy AI system? Ultimately, trustworthiness is complex and situation specific.