Winograd Convolution for Deep Neural Networks: Efficient Point Selection
A defining feature of each Winograd convolution algorithm is a set of real-value points where polynomials are sampled. The choice of points impacts the numeric accuracy of the algorithm, but the optimal set of points for small convolutions remains unknown. Existing work considers only small integers and simple fractions as candidate points. In this work, we propose a novel approach to point selection using points of the form
\(\lbrace -\frac{1}{c},-c,c,\frac{1}{c}\rbrace\)
using the full range of real-valued numbers for
c
. We show that groups of this form cause cancellations in the Winograd transform matrices that reduce numeric error. We find empirically that the error for different values of
c
forms a rough curve across the range of real-value numbers. It is therefore possible to localize the values of
c
that lead to lower error. We show that it is not necessary to choose integers or simple fractions as evaluation points, and that lower errors can be achieved with non-obvious real-valued points. We study a range of sizes for small convolutions and achieve reduction in error ranging from 2% to around 59% for both 1D and 2D convolution, when compared to state of the art. Furthermore, we identify patterns in cases when we select a subset of our proposed points that will always lead to a lower error. Finally, we implement a complete Winograd convolution layer and use it to run state-of-the-art deep convolution neural networks on real datasets and show that our proposed points achieve reduction in error, ranging from 22% to 63%, while also showing how an increased Winograd output size can result in execution speed-up for some cases.
No keywords indexed for this article. Browse by subject →
Kaiming He, Xiangyu Zhang, Shaoqing Ren et al.
Yangqing Jia, Evan Shelhamer, Jeff Donahue et al.
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton
Weisong Shi, Jie Cao, Quan Zhang et al.
Christian Szegedy, Wei Liu, Yangqing Jia et al.
Christian Szegedy, Wei Liu, Yangqing Jia et al.
- Published
- Nov 30, 2022
- Vol/Issue
- 21(6)
- Pages
- 1-28
- License
- View
You May Also Like
Eunjin Jeong, Jangryul Kim · 2022
91 citations