Skip to main content
SpringerLink
Log in

Multi-objective Bayesian Optimization for Neural Architecture Search

  • Conference paper
  • First Online:
Artificial Intelligence and Soft Computing (ICAISC 2022)

Part of the book series: Lecture Notes in Computer Science ((LNAI,volume 13588))

Included in the following conference series:

  • 526 Accesses

Abstract

A novel multi-objective algorithm denoted as MO-BayONet is proposed for the Neural Architecture Search (NAS) in this paper. The method based on Bayesian optimization encodes the candidate architectures directly as lists of layers and constructs an extra feature vector for the corresponding surrogate model. The general method allows to accompany the search for the optimal network by additional criteria besides the network performance. The NAS method is applied to combine classification accuracy with network size on two benchmark datasets here. The results indicate that MO-BayONet is able to outperform an available genetic algorithm based approach.

The work is supported by the project GA 22-02067S (“AppNeCo: Approximate Neurocomputing”) of the Czech Science Foundation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 69.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 89.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Archetti, F., Candelieri, A.: Bayesian Optimization and Data Science. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-24494-1

    Book  MATH  Google Scholar 

  2. Brochu, E., Cora, V.M., de Freitas, N.: A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning (2010)

    Google Scholar 

  3. Deb, K., Pratap, A., Agarwal, S., Meyarivan, T.: A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Trans. Evol. Comput. 6(2), 182–197 (2002). https://doi.org/10.1109/4235.996017

    Article  Google Scholar 

  4. Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: a survey. J. Mach. Learn. Res. 20(1), 1997–2017 (2019)

    MathSciNet  MATH  Google Scholar 

  5. Eriksson, D., et al.: Latency-aware neural architecture search with multi-objective Bayesian optimization. CoRR abs/2106.11890 (2021). arxiv.org/abs/2106.11890

  6. Fortin, F.A., De Rainville, F.M., Gardner, M.A., Parizeau, M., Gagné, C.: DEAP: evolutionary algorithms made easy. J. Mach. Learn. Res. 13, 2171–2175 (2012)

    MathSciNet  Google Scholar 

  7. Galuzio, P.P., de Vasconcelos Segundo, E.H., dos Santos Coelho, L., Mariani, V.C.: MOBOpt - multi-objective Bayesian optimization. SoftwareX 12, 100520 (2020). https://doi.org/10.1016/j.softx.2020.100520. http://www.sciencedirect.com/science/article/pii/S2352711020300911

  8. Goodfellow, I., et al.: TensorFlow: large-scale machine learning on heterogeneous systems (2015). Software available from tensorflow.org. https://www.tensorflow.org/

  9. Goodfellow, I., Bengio, Y., Courville, A.: Deep Learning. MIT Press (2016). http://www.deeplearningbook.org

  10. Kandasamy, K., Krishnamurthy, A., Schneider, J., Póczos, B.: Parallelised Bayesian optimisation via Thompson sampling. In: AISTATS. Proceedings of Machine Learning Research, vol. 84, pp. 133–142. PMLR (2018)

    Google Scholar 

  11. Kandasamy, K., Neiswanger, W., Schneider, J., Póczos, B., Xing, E.P.: Neural architecture search with Bayesian optimisation and optimal transport. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS 2018, Red Hook, NY, USA, pp. 2020–2029. Curran Associates Inc. (2018)

    Google Scholar 

  12. Kitano, H.: Designing neural networks using genetic algorithms with graph generation system. Complex Syst. 4, 461–476 (1990)

    Google Scholar 

  13. Krizhevsky, A., Nair, V., Hinton, G.: The CIFAR-10 dataset. http://www.cs.toronto.edu/kriz/cifar.html

  14. Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539

    Article  Google Scholar 

  15. LeCun, Y., Cortes, C.: The MNIST database of handwritten digits (2012). http://research.microsoft.com/apps/pubs/default.aspx?id=204699

  16. Miikkulainen, R., et al.: Evolving deep neural networks. CoRR abs/1703.00548 (2017). http://arxiv.org/abs/1703.00548

  17. Mrazek, V., Sarwar, S.S., Sekanina, L., Vasicek, Z., Roy, K.: Design of power-efficient approximate multipliers for approximate artificial neural networks. In: 2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD), pp. 1–7 (2016). https://doi.org/10.1145/2966986.2967021

  18. Rasmussen, C.E., Nickisch, H.: Gaussian processes for machine learning (GPML) toolbox. J. Mach. Learn. Res. 11, 3011–3015 (2010)

    MathSciNet  MATH  Google Scholar 

  19. Real, E., Aggarwal, A., Huang, Y., Le, Q.: Regularized evolution for image classifier architecture search. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, February 2018. https://doi.org/10.1609/aaai.v33i01.33014780

  20. Snoek, J., Larochelle, H., Adams, R.P.: Practical Bayesian optimization of machine learning algorithms. In: Proceedings of the 25th International Conference on Neural Information Processing Systems, NIPS 2012, Red Hook, NY, USA, vol. 2, pp. 2951–2959. Curran Associates Inc. (2012)

    Google Scholar 

  21. Vidnerová, P., Kalina, J.: Bayonet (2022). https://github.com/PetraVidnerova/BayONet

  22. Vidnerova, P., Neruda, R.: Evolving keras architectures for sensor data analysis. In: 2017 Federated Conference on Computer Science and Information Systems (FedCSIS), pp. 109–112, September 2017. https://doi.org/10.15439/2017F241

  23. White, C., Neiswanger, W., Nolen, S., Savani, Y.: A study on encodings for neural architecture search. In: Advances in Neural Information Processing Systems (2020)

    Google Scholar 

  24. White, C., Neiswanger, W., Savani, Y.: BANANAS: Bayesian optimization with neural architectures for neural architecture search. In: AAAI Conference on Artificial Intelligence (AAAI-2021) (2021)

    Google Scholar 

  25. Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for benchmarking machine learning algorithms (2017)

    Google Scholar 

  26. Xu, J., Zhou, W., Fu, Z., Zhou, H., Li, L.: A survey on green deep learning (2021)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Institute of Computer Science, The Czech Academy of Sciences, Prague, Czech Republic

    Petra Vidnerová & Jan Kalina

Authors
  1. Petra Vidnerová
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jan Kalina
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Petra Vidnerová .

Editor information

Editors and Affiliations

  1. Systems Research Institute of the Polish Academy of Sciences, Warsaw, Poland

    Leszek Rutkowski

  2. Częstochowa University of Technology, Częstochowa, Poland

    Rafał Scherer

  3. Częstochowa University of Technology, Częstochowa, Poland

    Marcin Korytkowski

  4. University of Alberta, Edmonton, AB, Canada

    Witold Pedrycz

  5. AGH University of Science and Technology, Kraków, Poland

    Ryszard Tadeusiewicz

  6. University of Louisville, Louisville, KY, USA

    Jacek M. Zurada

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Vidnerová, P., Kalina, J. (2023). Multi-objective Bayesian Optimization for Neural Architecture Search. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2022. Lecture Notes in Computer Science(), vol 13588. Springer, Cham. https://doi.org/10.1007/978-3-031-23492-7_13

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-23492-7_13

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-23491-0

  • Online ISBN: 978-3-031-23492-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation