Number of the records: 1  

A Comparison of Regularization Techniques for Shallow Neural Networks Trained on Small Datasets

  1. 1.
    SYSNO ASEP0546161
    Document TypeC - Proceedings Paper (int. conf.)
    R&D Document TypeConference Paper
    TitleA Comparison of Regularization Techniques for Shallow Neural Networks Trained on Small Datasets
    Author(s) Tumpach, Jiří (UIVT-O) ORCID, SAI
    Kalina, Jan (UIVT-O) RID, SAI, ORCID
    Holeňa, Martin (UIVT-O) SAI, RID
    Number of authors3
    Source TitleProceedings of the 21st Conference Information Technologies – Applications and Theory (ITAT 2021). - Aachen : Technical University & CreateSpace Independent Publishing, 2021 / Brejová B. ; Ciencialová L. ; Holeňa M. ; Mráz F. ; Pardubská D. ; Plátek M. ; Vinař T. - ISSN 1613-0073
    Pagess. 94-103
    Number of pages10 s.
    Publication formOnline - E
    ActionITAT 2021: Information Technologies - Applications and Theory /21./
    Event date24.09.2021 - 28.09.2021
    VEvent locationHeľpa
    CountrySK - Slovakia
    Event typeEUR
    Languageeng - English
    CountryDE - Germany
    Keywordsartificial neural networks ; regularization ; robustness ; optimization
    Subject RIVIN - Informatics, Computer Science
    OECD categoryComputer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
    R&D ProjectsGA18-18080S GA ČR - Czech Science Foundation (CSF)
    GA19-05704S GA ČR - Czech Science Foundation (CSF)
    Institutional supportUIVT-O - RVO:67985807
    EID SCOPUS85116716777
    AnnotationNeural networks are frequently used as regression models. Their training is usually difficult when the model is subject to a small training dataset with numerous outliers. This paper investigates the effects of various regularisation techniques that can help with this kind of problem. We analysed the effects of the model size, loss selection, L2 weight regularisation, L2 activity regularisation, Dropout, and Alpha Dropout. We collected 30 different datasets, each of which has been split by ten-fold cross-validation. As an evaluation metric, we used cumulative distribution functions (CDFs) of L1 and L2 losses to aggregate results from different datasets without a considerable amount of distortion. Distributions of the metrics are shown, and thorough statistical tests were conducted. Surprisingly, the results show that Dropout models are not suited for our objective. The most effective approach is the choice of model size and L2 types of regularisations.
    WorkplaceInstitute of Computer Science
    ContactTereza Šírová, sirova@cs.cas.cz, Tel.: 266 053 800
    Year of Publishing2022
    Electronic addresshttps://ics.upjs.sk/~antoni/ceur-ws.org/Vol-0000/paper38.pdf
Number of the records: 1  

  This site uses cookies to make them easier to browse. Learn more about how we use cookies.