In order to improve the flexibility and wearability of exoskeleton, a wearable lower limb exoskeleton based on the flexible drive joint is designed. In view of the different focus points at different phases in the control of lower limb exoskeleton, a hybrid control strategy based on dual-mode switching is proposed. Firstly, a series elastomer based on double parallel springs is designed to solve the flexibility problem of the lower limb exoskeleton joint, and it is installed in the drive module of exoskeleton joint. The feedback of joint torque and position information is achieved through two encoders. Then, a dual-mode switching control strategy is proposed by analyzing the movement characteristics of the exoskeleton at different gait phases. The adaptive impedance control algorithm is adopted at the stance phase to improve stability and impact resistance, and the active disturbance rejection and fast terminal sliding mode control algorithm is adopted at the swing phase to improve response speed and tracking accuracy. Finally, control simulation and active-passive tracking experiments are carried out to verify the superiority of the proposed algorithm to traditional PID (proportional-integral-derivative) and active disturbance rejection control algorithms. The results of passive tracking experiment show that the convergence time can reach 0.28 s when the convergence range of joint error is ±5%. In the active tracking experiment, the maximum RMSEs (root mean squared errors) of the hip and knee joints are 0.47° and 1.28° respectively while the experimenters wear the exoskeleton. These experimental results show that the human movement intention can be tracked in real time by the proposed control algorithm, and the requirements for human-machine interaction flexibility are satisfied.