Recent Trends in Learning from Data

0521198 - ÚI 2021 RIV CH eng M - Monography Chapter
Kůrková, Věra
Limitations of Shallow Networks.
Recent Trends in Learning from Data. Cham: Springer, 2020 - (Oneto, L.; Navarin, N.; Sperduti, A.; Anguita, D.), s. 129-154. Studies in Computational Intelligence, 896. ISBN 978-3-030-43882-1
R&D Projects: GA ČR(CZ) GA18-23827S
Institutional support: RVO:67985807
Keywords : shallow and deep networks * model complexity * probabilistic lower bounds
OECD category: Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)

Although originally biologically inspired neural networks were introduced as multilayer computational models, shallow networks have been dominant in applications till the recent renewal of interest in deep architectures. Experimental evidence and successfull application of deep networks pose theoretical questions asking: When and why are deep networks better than shallow ones? This chapter presents some probabilistic and constructive results on limitations of shallow networks. It shows implications of geometrical properties of high-dimensional spaces for probabilistic lower bounds on network complexity. The bounds depend on covering numbers of dictionaries of computational units and sizes of domains of functions to be computed. Probabilistic results are complemented by constructive ones built using Hadamard matrices and pseudo-noise sequences.
Permanent Link: http://hdl.handle.net/11104/0307155