Bayesian inference was once a gold standard for learning with neural networks, providingaccurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of theparameterspace. Inthispaper,weconstruct low-dimensional subspaces of parameter space, such as the first principal components of the stochastic gradient descent (SGD) trajectory, which contain diverse sets of high performing models. Inthesesubspaces,weareabletoapply elliptical slice sampling and variational inference,whichstruggleinthefullparameterspace. We show that Bayesian model averaging over the induced posterior in these subspaces produces accurate predictions and well-calibrated predictive uncertainty for both regression and image classification.

http://auai.org/uai2019/proceedings/papers/435.pdf