Flex-sfu: Activation function acceleration with non-uniform piecewise approximation

Abstract

Modern deep neural networks (DNN) increasingly use activation functions with computationally complex operations. This creates a challenge for current hardware accelerators, which are primarily optimized for convolutions and matrix-matrix multiplications. This work introduces Flex-SFU, a lightweight hardware accelerator for activation functions that uses nonuniform piecewise interpolation and supports multiple data formats, including both linear and quadratic function segments. We optimize the parameters of these function approximations offline to provide drop-in replacements for existing activation functions. Flex-SFU incorporates an address decoding unit based on a hardware binary-tree search, enabling nonuniform interpolation and floating-point support. This approach achieves, on average, a improvement in mean squared error compared to previous piecewise linear interpolation methods. Our …