Shared Control of Multi-Robot Formations Based on the Eye-Hand Dual-modal Human-Robot Interface
QIN Liujie1,2, SONG Guangming1,2, MAO Juzheng1,2, LIU Shengsong3, ZENG Hong1,2, SONG Aiguo1,2
1. Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China; 2. State Key Laboratory of Bioelectronics, Southeast University, Nanjing 210096, China; 3. State Grid Jiangsu Electric Power Co., Ltd., Nanjing 210024, China
秦留界, 宋光明, 毛巨正, 刘盛松, 曾洪, 宋爱国. 基于手眼双模态人机接口的移动机器人编队共享控制[J]. 机器人, 2022, 44(3): 343-351,360.DOI: 10.13973/j.cnki.robot.210274.
QIN Liujie, SONG Guangming, MAO Juzheng, LIU Shengsong, ZENG Hong, SONG Aiguo. Shared Control of Multi-Robot Formations Based on the Eye-Hand Dual-modal Human-Robot Interface. ROBOT, 2022, 44(3): 343-351,360. DOI: 10.13973/j.cnki.robot.210274.
Abstract：The traditional multi-robot formation control system based on the single-modal human-robot interface with the haptic device has poor performance in complex motion control such as formation switching and partial obstacle avoidance. So a shared control method based on the dual-modal human-robot interface with the force feedback haptic device and the eye tracker is proposed. Firstly, the operator's hand input and the eye tracking signals are mapped to formation movement and formation switching commands respectively. Secondly, a shared control framework composed of the master teleoperation controller and the slave autonomous controller is designed. The master teleoperation controller is used to receive and issue operator's control commands and controls the formation movement and the formation switching, while the slave autonomous controller autonomously complete formation keeping, external obstacle avoidance and internal collision avoidance tasks according to the system status. Finally, the obstacle avoidance experiments, the formation switching experiments and the comparison experiments are designed to prove the feasibility of the control method. The experimental results show that compared with the traditional single-modal bilateral teleoperation control, the proposed method reduces the operating load, and increases the control efficiency by 17.42%.
 王伟嘉，郑雅婷，林国政，等．集群机器人研究综述[J]．机器人，2020，42(2)：232-256.Wang W J, Zheng Y T, Lin G Z, et al. Swarm robotics: A review [J]. Robot, 2020, 42(2): 232-256.  Elkilany B G, Abouelsoud A A, Fathelbab A M R. Adaptive formation control of robot swarms using optimized potential field method[C]//IEEE International Conference on Industrial Technology. Piscataway, USA: IEEE, 2017: 721-725.  伍锡如，邢梦媛．分数阶多机器人的领航－跟随型环形编队控制[J]．控制理论与应用，2021，38(1)：103-109.Wu X R, Xing M Y. Annular formation control of the leader-follower multi-robot based on fractional order[J]. Control Theory & Applications, 2021, 38(1): 103-109.  Lee G, Chwa D. Decentralized behavior-based formation control of multiple robots considering obstacle avoidance[J]. Intelligent Service Robotics, 2018, 11: 127-138.  潘无为，姜大鹏，庞永杰，等．人工势场和虚拟结构相结合的多水下机器人编队控制[J]．兵工学报，2017，38(2)：326-334.Pan W W, Jiang D P, Pang Y J, et al. A multi-AUV formation algorithm combining artificial potential field and virtual structure [J]. Acta Armamentarii, 2017, 38(2): 326-334.  Roy D, Chowdhury A, Maitra M, et al. Multi-robot virtual structure switching and formation changing strategy in an unknown occluded environment[C]//IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA: IEEE, 2018: 4854-4861.  Luo J, Lin Z D, Li Y N, et al. A teleoperation framework for mobile robots based on shared control[J]. IEEE Robotics and Automation Letters, 2020, 5(2): 377-384.  Music S, Salvietti G, Dohmann P B g, et al. Human-robot team interaction through wearable haptics for cooperative manipulation[J]. IEEE Transactions on Haptics, 2019, 12(3): 350-362.  Zhang Y, Song G M, Wei Z, et al. Bilateral teleoperation of a group of mobile robots for cooperative tasks[J]. Intelligent Service Robotics, 2016, 9: 311-321.  Sun H Y, Song G M, Wei Z, et al. P-like controllers with collision avoidance for passive bilateral teleoperation of a UAV[J]. Industrial Robot, 2018, 45(1): 152-166.  张颖，宋光明，孙慧玉，等．多移动机器人双边遥操作系统中反馈力信息设计与研究[J]．东南大学学报（自然科学版），2017，47(1)：50-55.Zhang Y, Song G M, Sun H Y, et al. Design and research on feedback force in bilateral teleoperation system for multiple mobile robots[J]. Journal of Southeast University (Natural Science Edition), 2017, 47(1): 50-55.  Sun H Y, Song G M, Wei Z, et al. Energy-optimized consensus formation control for the time-delayed bilateral teleoperation system of UAVs[J]. International Journal of Aerospace Engineering, 2018. DOI: 10.1155/2018/2857674.  Ju C Y, Son H I. A distributed swarm control for an agricultural multiple unmanned aerial vehicle system[J]. Proceedings of the Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering, 2019, 233(10): 1298-1308.  Rodriguez-Seda E J, Troy J J, Erignac C A, et al. Bilateral teleoperation of multiple mobile agents: Coordinated motion and collision avoidance[J]. IEEE Transactions on Control Systems Technology, 2010, 18(4): 984-992.  Lin C W, Khong M H, Liu Y C. Experiments on human-in-theloop coordination for multirobot system with task abstraction [J]. IEEE Transactions on Automation Science and Engineering, 2015, 12(3): 981-989.  Zhang Y, Qiao G F, Song G M, et al. Smooth formation switching of the multiple robots in bilateral teleoperation systems[C]// IEEE International Conference on Robotics and Biomimetics. Piscataway, USA: IEEE, 2018: 2428-2433.  Franchi A, Secchi C, Son H I, et al. Bilateral teleoperation of groups of mobile robots with time-varying topology[J]. IEEE Transactions on Robotics, 2012, 28(5): 1019-1033.  Wang M Y, Kogkas A A, Darzi A, et al. Free-view, 3D gazeguided, assistive robotic system for activities of daily living[C]// IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway, USA: IEEE, 2018: 2355-2361.  Aronson R M, Santini T, Kübler T C, et al. Eye-hand behavior in human-robot shared manipulation[C]//ACM/IEEE International Conference on Human-Robot Interaction. Piscataway, USA: IEEE, 2018: 4-13.  Gomes J, Marques F, Lourenço A, et al. Gaze-directed telemetry in high latency wireless communications: The case of robot teleoperation[C]//42nd Annual Conference of the IEEE Industrial Electronics Society. Piscataway, USA: IEEE, 2016: 704- 709.  Latif H O, Sherkat N, Lotfi A. Teleoperation through eye gaze (TeleGaze): A multimodal approach[C]//IEEE International Conference on Robotics and Biomimetics. Piscataway, USA: IEEE, 2009: 711-716.  Li Z X, Akkil D, Raisamo R. Gaze augmented hand-based kinesthetic interaction: What you see is what you feel[J]. IEEE Transactions on Haptics, 2019, 12(2): 114-127.  Lin C W, Liu Y C. Decentralized estimation and control for bilateral teleoperation of mobile robot network with task abstraction[C]//IEEE International Conference on Robotics and Automation. Piscataway, USA: IEEE, 2017: 5384-5391.  Zhou D J, Schwager M. Assistive collision avoidance for quadrotor swarm teleoperation[C]//IEEE International Conference on Robotics and Automation. Piscataway, USA: IEEE, 2016: 1249-1254.  李正平，鲜斌． 基于虚拟结构法的分布式多无人机鲁棒编队控制[J]．控制理论与应用， 2020， 37(11)： 2423-2431. Li Z P, Xian B. Robust distributed formation control of multiple unmanned aerial vehicles based on virtual structure[J]. Control Theory & Applications, 2020, 37(11): 2423-2431.  Li I H, Chien Y H, Wang W Y, et al. Hybrid intelligent algorithm for indoor path planning and trajectory-tracking control of wheeled mobile robot[J]. International Journal of Fuzzy Systems, 2016, 18: 595-608.