Abstract
The aim of this work is to search for intuitive interpretations of penalized statistical estimators. Penalized estimates of the parameters of three models obtained by Bayesian reasoning are explained here to correspond to the intuition. First, the paper considers Bayesian estimates of the mean and covariance matrix for the multivariate normal distribution. Second, a connection of a robust regularized version of Mahalanobis distance with Bayesian estimation is discussed. Third, regularization networks, which represent a common nonparametric tool for regression modeling, are presented as Bayesian methods as well. On the whole, selected important multivariate and/or regression models are considered and novel interpretations are formulated.
The research was supported by the project 21-05325S (“Modern nonparametric methods in econometrics”) of the Czech Science Foundation.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Beaumont, M.A.: Approximate Bayesian computation. Annu. Rev. Stat. Appl. 6, 379–403 (2019)
Bryant, J., Zhang, J.L.: Bayesian Demographic Estimation and Forecasting. CRC Press, Boca Raton (2019)
Cerioli, A., Riani, M., Atkinson, A.C., Corbellini, A.: The power of monitoring: how to make the most of a contaminated multivariate sample. Stat. Methods Appl. 27, 559–587 (2018)
Chan, J., Koop, G., Poirier, D.J., Tobias, J.L.: Bayesian Econometric Methods, 2nd edn. Cambridge University Press, Cambridge (2020)
Cohen, G.: Algorithmic strategies for precious metals price forecasting. Mathematics 2022, 1134 (2022)
Dashdondov, K., Kim, M.H.: Mahalanobis distance based multivariate outlier detection to improve performance of hypertension prediction. Neural Process. Lett. 55, 265–277 (2023)
Efron, B., Morris, C.: Stein’s estimation rule and its competitors–an empirical Bayes approach. J. Am. Stat. Assoc. 68, 117–130 (1973)
Evans, I.G.: Bayesian estimation of parameters of a multivariate normal distribution. J. Roy. Stat. Soc. B 27, 279–283 (1965)
Evgeniou, T., Pontil, M., Poggio, T.: Regularization networks and support vector machines. Adv. Comput. Math. 13, 1–50 (2000)
Guo, Y., Hastie, T., Tibshirani, R.: Regularized discriminant analysis and its application in microarrays. Biostatistics 8, 86–100 (2007)
Haff, L.R.: Empirical Bayes estimation of the multivariate normal covariance matrix. Ann. Stat. 8, 586–597 (1980)
Hastie, T., Tibshirani, R., Wainwright, M.: Statistical Learning with Sparsity. The Lasso and Generalizations. CRC Press, Boca Raton (2015)
Haykin, S.O.: Neural Networks and Learning Machines: A Comprehensive Foundation, 2nd edn. Prentice Hall, Upper Saddle River (2009)
Hinne, M., Gronau, Q.F., van den Bergh, D., Wagenmakers, E.J.: A conceptual introduction to Bayesian model averaging. Adv. Methods Pract. Psychol. Sci. 3, 200–215 (2020)
Hlubinka, D., Šiman, M.: On generalized elliptical quantiles in the nonlinear quantile regression setup. TEST 24, 249–264 (2015)
Hlubinka, D., Šiman, M.: On parametric elliptical regression quantiles. REVSTAT Stat. J. 18, 257–280 (2020)
Johnson, A.A., Ott, M.Q., Dogucu, M.: Bayes Rules! An Introduction to Applied Bayesian Modeling. Chapman & Hall/CRC, Boca Raton (2022)
Jurečková, J., Picek, J., Schindler, M.: Robust Statistical Methods with R, 2nd edn. CRC Press, Boca Raton (2019)
Kalina, J.: On robust information extraction from high-dimensional data. Serb. J. Manag. 9, 131–144 (2014)
Kalina, J., Tichavský, J.: On robust estimation of error variance in (highly) robust regression. Meas. Sci. Rev. 20, 6–14 (2020)
Ledoit, O., Wolf, M.: A well-conditioned estimator for large-dimensional covariance matrices. J. Multivar. Anal. 88, 365–411 (2004)
Pacifico, A.: Robust open Bayesian analysis: overfitting, model uncertainty, and endogeneity issues in multiple regression models. Economet. Rev. 40, 148–176 (2021)
Pfarrhofer, M., Piribauer, P.: Flexible shrinkage in high-dimensional Bayesian spatial autoregressive models. Spat. Stat. 29, 109–128 (2019)
Pillonetto, G.: System identification using kernel-based regularization: new insights on stability and consistency issues. Automatica 93, 321–332 (2018)
Qiu, J., Jammalamadaka, S.R., Ning, N.: Multivariate time series analysis from a Bayesian machine learning perspective. Ann. Math. Artif. Intell. 88, 1061–1082 (2020)
Roelant, E., Van Aelst, S., Willems, G.: The minimum weighted covariance determinant estimator. Metrika 70, 177–204 (2009)
Saleh, A.K.M.E., Picek, J., Kalina, J.: R-estimation of the parameters of a multiple regression model with measurement errors. Metrika 75, 311–328 (2012)
Stein, C.: Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In: Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability, University of California Press, Berkeley, vol. 1, pp. 197–206 (1956)
Acknowledgements
The authors would like to thank Jiří Grim and Lubomír Soukup (both ÚTIA AV ČR) for discussion about Bayesian estimation.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kalina, J., Peštová, B. (2023). On the Bayesian Interpretation of Penalized Statistical Estimators. In: Rutkowski, L., Scherer, R., Korytkowski, M., Pedrycz, W., Tadeusiewicz, R., Zurada, J.M. (eds) Artificial Intelligence and Soft Computing. ICAISC 2023. Lecture Notes in Computer Science(), vol 14126. Springer, Cham. https://doi.org/10.1007/978-3-031-42508-0_31
Download citation
DOI: https://doi.org/10.1007/978-3-031-42508-0_31
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-42507-3
Online ISBN: 978-3-031-42508-0
eBook Packages: Computer ScienceComputer Science (R0)