A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Training
In: DATE 2024 - 27th IEEE/ACM Design, Automation and Test in Europe ; https://hal.science/hal-04380270 ; DATE 2024 - 27th IEEE/ACM Design, 2024
Online
Konferenz
Zugriff:
International audience ; Training Deep Neural Networks (DNNs) can be computationally demanding, particularly when dealing with large models. Recent work has aimed to mitigate this computational challenge by introducing 8-bit floating-point (FP8) formats for multiplication. However, accumulations are still done in either half (16-bit) or single (32-bit) precision arithmetic. In this paper, we investigate lowering accumulator word length while maintaining the same model accuracy. We present a multiply-accumulate (MAC) unit with FP8 multiplier inputs and FP12 accumulations, which leverages an optimized stochastic rounding (SR) implementation to mitigate swamping errors that commonly arise during low precision accumulations. We investigate the hardware implications and accuracy impact associated with varying the number of random bits used for rounding operations. We additionally attempt to reduce MAC area and power by proposing a new scheme to support SR in floating-point MAC and by removing support for subnormal values. Our optimized eager SR unit significantly reduces delay and area when compared to a classic lazy SR design. Moreover, when compared to MACs utilizing single-or half-precision adders, our design showcases notable savings in all metrics. Furthermore, our approach consistently maintains near baseline accuracy across a diverse range of computer vision tasks, making it a promising alternative for low-precision DNN training.
Titel: |
A Stochastic Rounding-Enabled Low-Precision Floating-Point MAC for DNN Training
|
---|---|
Autor/in / Beteiligte Person: | Ben Ali, Sami ; Filip, Silviu-Ioan ; Sentieys, Olivier ; Architectures matérielles spécialisées pour l’ère post loi-de-Moore (TARAN) ; Inria Rennes – Bretagne Atlantique ; Institut National de Recherche en Informatique et en Automatique (Inria)-Institut National de Recherche en Informatique et en Automatique (Inria)-ARCHITECTURE (IRISA-D3) ; Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) ; Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) ; Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-Institut National de Recherche en Informatique et en Automatique (Inria)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique) ; Institut Mines-Télécom Paris (IMT)-Institut Mines-Télécom Paris (IMT)-Université de Rennes (UR)-Institut National des Sciences Appliquées - Rennes (INSA Rennes) ; Institut Mines-Télécom Paris (IMT)-Institut Mines-Télécom Paris (IMT)-Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA) ; Institut National des Sciences Appliquées (INSA)-Institut National des Sciences Appliquées (INSA)-Université de Bretagne Sud (UBS)-École normale supérieure - Rennes (ENS Rennes)-CentraleSupélec-Centre National de la Recherche Scientifique (CNRS)-IMT Atlantique (IMT Atlantique) ; Institut Mines-Télécom Paris (IMT)-Institut Mines-Télécom Paris (IMT) ; ANR-18-CE23-0012,AdequateDL,Accélérateurs Approximatifs pour Apprentissage Profond(2018) ; ANR-21-CE24-0015,RE-TRUSTING,Architectures matérielles fiable pour l'Intelligence Artificielle de confiance(2021) ; ANR-10-LABX-0007,COMIN Labs,Digital Communication and Information Sciences for the Future Internet(2010) ; ANR-18-CE25-0017,RAKES,Accélérer le calcul parallèle à l'aide de méthodes de diffusion grâce aux réseaux sur puce hybride radio(2018) ; ANR-23-PEIA-0010,HOLIGRAIL,HOLIistic approaches to GReener model Architectures for Inference and Learning(2023) |
Link: | |
Zeitschrift: | DATE 2024 - 27th IEEE/ACM Design, Automation and Test in Europe ; https://hal.science/hal-04380270 ; DATE 2024 - 27th IEEE/ACM Design, 2024 |
Veröffentlichung: | HAL CCSD, 2024 |
Medientyp: | Konferenz |
Schlagwort: |
|
Sonstiges: |
|