Abstract: In view of the problems such as repeated exploration and incomplete environmental exploration existing in the autonomous exploration method of mobile robots, a hierarchical and ordered environmental exploration method for mobile robots based on reinforcement learning is proposed. This method constructs a hierarchical framework of local forward exploration and global reverse backtracking, and decouples the exploration process into two stages. For the local exploration stage, a soft actor-critic algorithm is proposed. Taking the local environmental exploration map as input, it extracts environmental features through neural networks, and combines exploration rewards and safety rewards to achieve efficient local depth-first exploration. In the global backtracking stage, the backtracking points are determined by the potential field method and the path is optimized by combining the image mask, based on the multi-tree trajectory recording structure. And thus, the ordered backtracking and path optimization in global unexplored areas are realized. Experimental results show that compared with the front-point method and the next-best-view method, the proposed method has higher environmental exploration coverage rate and exploration efficiency. Meanwhile, the ablation experiment also verifies the necessities of the local exploration module for improving the exploration efficiency and the global backtracking mechanism for improving the exploration coverage rate. Finally, the real-machine experiment verifies that the proposed method has a faster exploration speed and a higher exploration efficiency compared with existing environmental exploration methods.