A transistor operations model for deep learning energy consumption scaling law

Date published

2022-12-14

Free to read from

Supervisor/s

Journal Title

Journal ISSN

Volume Title

Publisher

IEEE

Department

Type

Article

ISSN

2691-4581

Format

Citation

Li C, Tsourdos A, Guo W. (2024) A transistor operations model for deep learning energy consumption scaling law. IEEE Transactions on Artificial Intelligence, Volume 5, Issue 1, January 2024, pp. 192-204

Abstract

Deep Neural Networks (DNN) has transformed the automation of a wide range of industries and finds increasing ubiquity in society. The high complexity of DNN models and its widespread adoption has led to global energy consumption doubling every 3-4 months. Current energy consumption measures largely monitor system wide consumption or make linear assumptions of DNN models. The former approach captures other unrelated energy consumption anomalies, whilst the latter does not accurately reflect nonlinear computations. In this paper, we are the first to develop a bottom-up Transistor Operations (TOs) approach to expose the role of non-linear activation functions and neural network structure. As there will be inevitable energy measurement errors at the core level, we statistically model the energy scaling laws as opposed to absolute consumption values. We offer models for both feedforward DNNs and convolution neural networks (CNNs) on a variety of data sets and hardware configurations - achieving a 93.6% - 99.5% precision. This outperforms existing FLOPs-based methods and our TOs method can be further extended to other DNN models.

Description

Software Description

Software Language

Github

Keywords

Energy Consumption, Deep Learning, Model Architecture, Transistor Operations

DOI

Rights

Attribution-NonCommercial 4.0 International

Relationships

Relationships

Resources

Funder/s

European Union funding: 778305