Abstract: In this paper, a new reinforcement learning(CVRL) algorithm with continuous vector output is proposed as to the navigation problem of mobile robot. CVRL is hierarchically structured. The lower layer is composed with several groups of unit actions and real-valued vector output can be produced based on action combination. The higher layer is a Q-Learning unit defined on the space of combined action, its responsibility is the selection of a proper combined actions. The detailed implementation of the CVRL navigation controller is given, and the simulation results demonstrate its effectiveness.