Balancing Exploitation and Exploration via Fully Probabilistic Design of Decision Policies

0503817 - ÚTIA 2020 RIV PT eng C - Conference Paper (international conference)
Kárný, Miroslav - Hůla, František
Balancing Exploitation and Exploration via Fully Probabilistic Design of Decision Policies.
Proceedings of the 11th International Conference on Agents and Artificial Intelligence. Vol. 2. Setúbal: SciTePress, 2019 - (Rocha, A.; Steels, L.; van den Herik, J.), s. 857-864. ISBN 978-989-758-350-6.
[International Conference on Agents and Artificial Intelligence. Praha (CZ), 19.02.2019-21.02.2019]
R&D Projects: GA ČR(CZ) GA16-09848S; GA ČR(CZ) GA18-15970S
Institutional support: RVO:67985556
Keywords : exploitation * exploration * adaptive systems * Bayesian estimation * fully probabilistic design * Markov decision process
OECD category: Computer sciences, information science, bioinformathics (hardware development to be 2.2, social aspect to be 5.8)
http://library.utia.cas.cz/separaty/2019/AS/hula-0503817.pdf

Adaptive decision making learns an environment model serving a design of a decision policy. The policy-generated actions influence both the acquired reward and the future knowledge. The optimal policy properly balances exploitation with exploration. The inherent dimensionality curse of decision making under incomplete knowledge prevents the realisation of the optimal design. This has stimulated repetitive attempts to reach this balance at least approximately. Usually, either: (a) the exploitative reward is enriched by a part reflecting the exploration quality and a feasible approximate certainty-equivalent design is made, or (b) an explorative random noise is added to the purely exploitative actions. This paper avoids the inauspicious (a) and improves (b) by employing the non-standard fully probabilistic design (FPD) of decision policies, which naturally generates random actions. Monte-Carlo experiments confirm its achieved quality. The quality stems from methodological contributions, which include: (i) an improvement of the relation between FPD and standard Markov decision processes, (ii) a design of an adaptive tuning of an FPD-parameter. The latter also suits for the tuning of the temperature in both simulated annealing and Boltzmann’s machine.
Permanent Link: http://hdl.handle.net/11104/0295689