Algorithm Adoption and Explanations: An Experimental Study on Self and Other Perspectives (Job Market Paper) [Link]
with Fernanda Bravo (UCLA), Yaron Shaposhnik (University of Rochester), and Leon Valdes (University of Pittsburgh) [view abstract]
People are reluctant to follow machine-learning recommendation systems. To address this, research suggests providing explanations about the underlying algorithm to increase adoption. However, the degree to which adoption depends on the party impacted by a user’s decision (the user vs. a third party) and whether explanations boost adoption in both settings is not well understood. These questions are particularly relevant in contexts such as medical, judicial, and financial decisions, where a third party bears the main impact of a user’s decision. We examine these questions using controlled incentivized experiments. We design a prediction task where participants observe fictitious objects and must predict their color with the aid of algorithmic recommendations. We manipulate whether (i) a participant receives an explanation about the algorithm and (ii) the impacted party is the participant (Self treatment) or a matched individual (Other treatment). Our findings reveal that, in the absence of explanations, algorithmic adoption is similar regardless of the impacted party. We also find that explanations significantly increase adoption in Self, where they help attenuate negative responses to algorithm errors over time. However, this pattern is not observed in Other, where explanations have no discernible effect—leading to significantly lower adoption than in Self in the last rounds. These results suggest that further strategies—beyond explanations—need to be explored to boost adoption in settings where the impact is predominantly felt by a third party.
Explaining Model Behavior Across Space and Time: Differential and Intertemporal Explanations [Link]
with Yaron Shaposhnik (University of Rochester) [view abstract]
We introduce the notion of differential explanation as information provided to users of machine learning models that explains why a model assigns different predicted values to two observations. We propose a model-agnostic feature importance method based on SHapley Additive exPlanations (SHAP, Lundberg and Lee 2017) to generate such explanations, which can be viewed as a generalization of SHAP. We demonstrate the applicability of the method across a wide range of prediction problems in Operations Management, Information Systems, Finance, and Marketing. We prove that the method is coherent and that, in the case of linear models, it produces correct answers. Finally, we extend the method to explain changes in predictions over time in dynamic environments by introducing the concept of intertemporal explanations. We generate these explanations in the context of waiting time prediction to inform customers why their predicted waiting time changes in a way that may be perceived as abrupt.
The recent rise of machine learning (ML) has been leveraged by practitioners and researchers to provide new solutions to an ever-growing number of business problems. As with other ML applications, these solutions rely on model selection, which is typically achieved by evaluating certain metrics on models separately and selecting the model whose evaluations (i.e., accuracy-related loss and/or certain interpretability measures) are optimal. However, empirical evidence suggests that, in practice, multiple models often attain competitive results. While models’ overall performance could be similar, they could operate quite differently. This results in an implicit tradeoff in models’ performance throughout the feature space which resolving requires new model selection tools. This paper explores methods for comparing predictive models in an interpretable manner to uncover the tradeoff and help resolve it. To this end, we propose various methods that synthesize ideas from supervised learning, unsupervised learning, dimensionality reduction, and visualization to demonstrate how they can be used to inform model developers about the model selection process. Using various datasets and a simple Python interface, we demonstrate how practitioners and researchers could benefit from applying these approaches to better understand the broader impact of their model selection choices.
This monograph systematically reviews recent developments in causal inference methods and their applications in marketing. It covers both well-established techniques and emerging methodologies, and reviews how machine learning enhances causal inference. For each method, five recent academic papers in marketing are discussed. Additionally, this monograph provides simplified code for developing simulated data (using Python) and hypothetical examples of data analysis (using Stata). We expect this monograph to serve as a useful resource both to current and future researchers in marketing.