A Human-Robot Interaction System Based on Hybrid GazeBrain-Machine Interface and Shared Control
WANG Yanxin1, JI Peng2, ZENG Hong1, SONG Aiguo1, WU Changcheng3, XU Baoguo1, LI Huijun1
1. Jiangsu Key Lab of Remote Measurement and Control, State Key Laboratory of Bioelectronics, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China;
2. School of Electrical Engineering and Automation, Qilu University of Technology(Shandong Academy of Sciences), Jinan 250353, China;
3. College of Automation Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing 211106, China
王言鑫, 纪鹏, 曾洪, 宋爱国, 吴常铖, 徐宝国, 李会军. 基于混合视线-脑机接口与共享控制的人-机器人交互系统[J]. 机器人, 2018, 40(4): 431-439.DOI: 10.13973/j.cnki.robot.180134.
WANG Yanxin, JI Peng, ZENG Hong, SONG Aiguo, WU Changcheng, XU Baoguo, LI Huijun. A Human-Robot Interaction System Based on Hybrid GazeBrain-Machine Interface and Shared Control. ROBOT, 2018, 40(4): 431-439. DOI: 10.13973/j.cnki.robot.180134.
Abstract:A human-robot interaction system is designed with the hybrid gaze brain-machine interface and the shared control strategy, to make the user continuously control the robot end-effector in 2D space with his/her gaze and thought, meanwhile obtaining timely assistance from the machine intelligence in the task of avoiding obstacles and reaching target objects. Firstly, the movement speed of the robot end-effector is adjusted continuously and proportionally by the motor imagery strength of the user, in order to increase the user's sense of control and his/her engagement in the task. Next, a shared control strategy is proposed for controlling the movement direction of the robot end-effector, which dynamically fuses the direction commands from the user obtained by gaze tracking and from the robotic system for avoiding obstacles and reaching target objects. Such a shared control strategy adaptively adjusts the assist level for the user, so as to reduce the mental workload of the user and improve the success rate of completing the task. Finally, with respect to the constructed human-robot interaction system based on the hybrid gaze brain-machine interface and the shared control, experiments are conducted to verify its effectiveness.
[1] Millán J D R, Rupp R, Müller-Putz G R, et al. Combining brain-computer interfaces and assistive technologies:State-of-the-art and challenges[J]. Frontiers in Neuroscience, 2010, 4:No.161.
[2] Hochberg L R, Bacher D, Jarosiewicz B, et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm[J]. Nature, 2012, 485(7398):372.
[3] Morgante L, Morgante F, Moro E, et al. How many parkinsonian patients are suitable candidates for deep brain stimulation of subthalamic nucleus? Results of a questionnaire[J]. Parkinsonism and Related Disorders, 2007, 13(8):528-531.
[4] Gudayol-Ferré E, Peró-Cebollero M, González-Garrido A A, et al. Changes in brain connectivity related to the treatment of depression measured through fMRI:A systematic review[J]. Frontiers in Human Neuroscience, 2015, 9:No.582.
[5] Naseer N, Hong K S. Corrigendum:fNIRS-based brain-computer interfaces:A review[J]. Frontiers in Human Neuroscience, 2015, 9:No.3.
[6] Fukuma R, Yanagisawa T, Saitoh Y, et al. Corrigendum:Real-time control of a neuroprosthetic hand by magnetoencephalographic signals from paralysed patients[J]. Scientific Reports, 2016, 6:No.21781.
[7] Moghimi S, Kushki A, Guerguerian A M, et al. A review of EEG-based brain-computer interfaces as access pathways for individuals with severe disabilities[J]. Assistive Technology, 2013, 25(2):99-110.
[8] Carlson T, Millan J D R. Brain-controlled wheelchairs:A robotic architecture[J]. IEEE Robotics and Automation Magazine, 2013, 20(1):65-73.
[9] Li Z J, Zhao S N, Duan J D, et al. Human cooperative wheelchair with brain——machine interaction based on shared control strategy[J]. IEEE/ASME Transactions on Mechatronics, 2017, 22(1):185-195.
[10] He S H, Zhang R, Wang Q H, et al. A P300-based threshold-free brain switch and its application in wheelchair control[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2017, 25(6):715-725.
[11] Chen X G, Wang Y J, Nakanishi M, et al. High-speed spelling with a noninvasive brain-computer interface[J]. Proceedings of the National Academy of Sciences of the United States of America, 2015, 112(44):E6058-E6067.
[12] Chen Y Q, Ke Y F, Meng G F, et al. Enhancing performance of P300-Speller under mental workload by incorporating dual-task data during classifier training[J]. Computer Methods and Programs in Biomedicine, 2017, 152:35-43.
[13] Zhao X G, Chu Y Q, Han J D, et al. SSVEP-based brain-computer interface controlled functional electrical stimulation system for upper extremity rehabilitation[J]. IEEE Transactions on Systems, Man, and Cybernetics:Systems, 2016, 46(7):947-956.
[14] 徐宝国,彭思,宋爱国.基于运动想象脑电的上肢康复机器人[J].机器人,2011,33(3):307-313.Xu B G, Peng S, Song A G. Upper-limb rehabilitation robot based on motor imagery EEG[J]. Robot, 2011, 33(3):307-313.
[15] 俞建成,张进,李伟.基于事件相关电位的水下机械手脑电波控制[J].机器人,2017,39(4):395-404.Yu J C, Zhang J, Li W. Controlling an underwater manipulator via event-related potentials of brainwaves[J]. Robot, 2017, 39(4):395-404.
[16] 李敏,徐光华,谢俊,等.脑卒中意念控制的主被动运动康复技术[J].机器人,2017,39(5):759-768.Lin M, Xu G H, Xie J, et al. Motor rehabilitation with control based on human intent for stroke survivors[J]. Robot, 2017, 39(5):759-768.
[17] Wang Y X, Zeng H, and Liu J. Low-cost eye-tracking glasses with real-time head rotation compensation[C]//10th International Conference on Sensing Technology. Piscataway, USA:IEEE, 2016.
[18] Soekadar S R, Witkowski M, Gómez C, et al. Hybrid EEG/EOG-based brain/neural hand exoskeleton restores fully independent daily living activities after quadriplegia[J]. Science Robotics, 2016, 1(1):32-96.
[19] Yuan P, Chen X G, Wang Y J, et al. Enhancing performances of SSVEP-based brain-computer interfaces via exploiting inter-subject information[J]. Journal of Neural Engineering, 2015, 12(4):No.046006.
[20] Khan M J, Hong K S. Hybrid EEG-fNIRS-based eight-com-mand decoding for BCI:Application to quadcopter control[J]. Frontiers in Neurorobotics, 2017, 11:No.6.
[21] Hong K S, Khan M J. Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands:A review[J]. Frontiers in Neurorobotics, 2017, 11:No.35.
[22] Zeng H, Wang Y X, Wu C C, et al. Closed-loop hybrid gaze brain-machine interface based robotic arm control with augmented reality feedback[J]. Frontiers in Neurorobotics, 2017, 11:No.60.
[23] McMullen D P, Hotson G, Katyal K D, et al. Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial EEG, eye tracking, and computer vision to control a robotic upper limb prosthetic[J]. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 2014, 22(4):784-796.
[24] Frisoli A, Loconsole C, Leonardis D, et al. A new gaze-BCI-driven control of an upper limb exoskeleton for rehabilitation in real-world tasks[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part C:Applications and Reviews, 2012, 42(6):1169-1179.
[25] Kim Y J, Park S W, Yeom H G, et al. A study on a robot arm driven by three-dimensional trajectories predicted from non-invasive neural signals[J]. Biomedical Engineering Online, 2015, 14:No.81.
[26] 吴朝晖,俞一鹏,潘纲,等.脑机融合系统综述[J].生命科学,2014(6):645-649.Wu C H, Yu Y P, Pan G, et al. Brain-machine integrated systems[J]. Chinese Bulletin of Life Sciences, 2014(6):645-649.