Exploring multi-agent deep reinforcement learning in IEEE very small size soccer

Abstract

Robot soccer is regarded as a prime example of a dynamic and cooperative multi-agent environment, as it can demonstrate a variety of complexities. Reinforcement learning is a promising technique for optimizing decision-making in these complex systems, as it has recently achieved great success due to advances in deep neural networks, as shown in problems such as autonomous driving, games, and robotics. In multi-agent systems reinforcement learning re- search is tackling challenges such as cooperation, partial observability, decentralized execution, communication, and complex dynamics. On difficult tasks, modeling the complete problem in the learning environment can be too difficult for the algorithms to solve. We can simplify the environment to enable learning, however, policies learned in simplified environments are usually not optimal in the full environment. This study explores whether deep multi-agent re- inforcement learning outperforms single-agent counterparts in an IEEE Very Small Size Soccer setting, a task that presents a challenging problem of cooperation and competition with two teams facing each other, each having three robots. We investigate diverse learning paradigms efficacies in achieving the core objective of goal scoring, assessing cooperation by compar- ing the results of multi-agent and single-agent paradigms. Results indicate that simplifications made to the learning environment to facilitate learning may diminish cooperation’s importance and also introduce biases, driving the learning process towards conflicting policies misaligned with the original challenge.