Self-Distribution Distillation: Efficient Uncertainty Estimation
Apollo - University of Cambridge Repository, 2022
Online
unknown
Zugriff:
Deep learning is increasingly being applied in safety-critical domains. For these scenarios it is important to know the level of uncertainty in a model’s prediction to ensure appropriate decisions are made by the system. Deep ensembles are the de-facto standard approach to obtaining various measures of uncertainty. However, ensembles often significantly increase the resources required in the training and/or deployment phases. Approaches have been developed that typically address the costs in one of these phases. In this work we propose a novel training approach, self-distribution distillation (S2D), which is able to efficiently train a single model that can estimate uncertainties. Furthermore it is possible to build ensembles of these models and apply hierarchical ensemble distillation approaches. Experiments on CIFAR-100 showed that S2D models outperformed standard models and Monte-Carlo dropout. Additional out-of-distribution detection experiments on LSUN, Tiny ImageNet, SVHN showed that even a stan- dard deep ensemble can be outperformed using S2D based ensembles and novel distilled models.
Titel: |
Self-Distribution Distillation: Efficient Uncertainty Estimation
|
---|---|
Autor/in / Beteiligte Person: | Fathullah, Y ; Gales, MJF |
Link: | |
Veröffentlichung: | Apollo - University of Cambridge Repository, 2022 |
Medientyp: | unknown |
DOI: | 10.17863/cam.85540 |
Sonstiges: |
|