Stable Low-Rank Tensor Decomposition for Compression of Convolutional Neural Network

0534541 - ÚTIA 2021 RIV GB eng C - Conference Paper (international conference)
Phan, A. H. - Sobolev, K. - Sozykin, K. - Ermilov, D. - Gusak, J. - Tichavský, Petr - Glukhov, V. - Oseledets, I. - Cichocki, A.
Stable Low-Rank Tensor Decomposition for Compression of Convolutional Neural Network.
ECCV 2020. Cham: Springer Nature Switzerland AG 2020, 2020 - (Vedaldi, A.; Bischof, H.; Brox, T.; Frahm, J.), s. 522-539. Lecture Notes in Computer Science, LNCS, 12374. ISBN 978-3-030-58525-9. ISSN 0302-9743. E-ISSN 1611-3349.
[European Conference on Computer Vision 2020 /16./. Glasgow (GB), 23.08.2020-28.08.2020]
Institutional support: RVO:67985556
Keywords : Convolutional neural network acceleration * Low-rank tensor decomposition * Degeneracy correction
OECD category: Electrical and electronic engineering
http://library.utia.cas.cz/separaty/2020/SI/tichavsky-0534541.pdf

Most state-of-the-art deep neural networks are overparameterized and exhibit a high computational cost. A straightforward approach to this problem is to replace convolutional kernels with its low-rank tensor approximations, whereas the Canonical Polyadic tensor Decomposition is one of the most suited models. However, fitting the convolutional tensors by numerical optimization algorithms often encounters diverging components, i.e.,extremely large rank-one tensors but canceling each other. Such degeneracy often causes the non-interpretable result and numerical instability for the neural network ne-tuning. This paper is the first study on degeneracy in the tensor decomposition of convolutional kernels. We present a novel method, which can stabilize the low-rank approximation of convolutional kernels and ensure efficient compression while preserving the high quality performance of the neural networks. We evaluate our approach on popular CNN architectures for image classification and show that our method results in much lower accuracy degradation and provides consistent performance.

Permanent Link: http://hdl.handle.net/11104/0313191