AI Lunch Series: João Caldeira

12:00 - 1:00 p.m., Wednesday, February 17, 2021

“Deeply Uncertain: Comparing Methods of Uncertainty Quantification in Deep Learning Algorithms”
Dr. João Caldeira (Google)

Abstract: “In recent years, many methods for quantifying uncertainty in the output of deep learning algorithms have been developed. In this talk we will introduce three of the most prominent methods - Bayesian Neural Networks (BNN), Concrete Dropout (CD), and Deep Ensembles (DE) - and compare them to the standard analytic error propagation in the context of a single pendulum. We will also relate the output of these methods to terms more familiar in the context of the physical sciences. Our results highlight some pitfalls that may occur when using these UQ methods. For example, when the variation of noise in the training set is small, all methods predicted the same relative uncertainty independently of the inputs. This issue is particularly hard to avoid in BNN. On the other hand, when the test set contains samples far from the training distribution, we found that no methods sufficiently increased the uncertainties associated to their predictions. This problem was particularly clear for CD. In light of these results, we make some recommendations for usage and interpretation of UQ methods.”

Bio: João Caldeira completed a PhD in string theory at the University of Chicago in 2018. He moved on to a postdoc at Fermilab centered on applying machine learning and quantum computing techniques to problems in astrophysics. Since June 2020, he has been a Site Reliability Engineer at Google.