Analytical guarantees on numerical precision of deep neural networks

Abstract

The acclaimed successes of neural networks often overshadow their tremendous complexity. We focus on numerical precision - A key parameter defining the complexity of neural networks. First, we present theoretical bounds on the accuracy in presence of limited precision. Interestingly, these bounds can be computed via the back-propagation algorithm. Hence, by combining our theoretical analysis and the backpropagation algorithm, we are able to readily determine the minimum precision needed to preserve accuracy without having to resort to timeconsuming fixed-point simulations. Wc provide numerical evidcncc showing how our approach allows us to maintain high accuracy but with lower complexity than state-of-the-art binary networks.