Abstract: In this paper, the role diversity based on reinforcement learning in robot soccer is studied. Through simulation and analysis, it is shown that the Q algorithm infinite-horizon discounted model in is not suitable to this task. Instead of that, average-reward model is used for improving the algorithm. Simulation experiments show that the convergence rate in learning and the system performance are twice increased after improvement.