A Real-time Expression Mimicking Method for Humanoid Robot Based on Dual LSTM Fusion
HUANG Zhong1,2, REN Fuji2,3, HU Min2, LIU Juan1
1. School of Physics and Electronic Engineering, Anqing Normal University, Anqing 246011, China;
2. Anhui Province Key Laboratory of Affective Computing and Advanced Intelligent Machine, Hefei 230009, China;
3. Faculty of Engineering, University of Tokushima, Tokushima 770800, Japan
Abstract：To improve space-time similarity and motion smoothness in robot expression imitation, a real-time expression mimicking method for humanoid robot based on dual LSTM (long short-term memory) fusion is proposed by combining with the sequence to sequence deep learning model. In offline mechanical modeling phase, an inverse mechanical model is constructed firstly to fulfill the inverse mapping from facial feature sequence to motor control sequence, and a motion tendency model is further presented to wrap the smoothness of continuous motor motion. Secondly, a weighted objective function is addressed to implement the fusion of the above two models as well as the parameter optimization. In online expression transfer phase, the facial feature sequence of performer is regarded as the input of the fusion model, and the frame-to-frame expression mimicking of robot is achieved by means of the end-to-end translation from the performer facial feature sequence to robot control sequence under the optimal parameters. The experimental results indicate that the control deviations of the fusion model is lower than 8%, meanwhile, the space-time similarity and motion smoothness in expression mimicking is greater than 85%. Compared with related methods, the proposed method has a significant improvement in the control deviation, space-time similarity and motion smoothness.
 Pour A G, Taheri A, Alemi M, et al. Human-robot facial expression reciprocal interaction platform:Case studies on children with autism[J]. International Journal of Social Robotics, 2018, 10(2):179-198.
 Ren F J, Huang Z. Automatic facial expression learning method based on humanoid robot XIN-REN[J]. IEEE Transactions on Human-Machine Systems, 2016, 46(6):810-821.
 Beer J M, Liles K R, Wu X, et al. Affective human-robot interaction[M]//Emotions and Affect in Human Factors and Human-Computer Interaction. Cambridge, USA:Academic Press, 2017:359-381.
 Mohammed O, Bailly G, Pellier D. Acquiring human-robot interaction skills with transfer learning techniques[C]//ACM/IEEE International Conference on Human-Robot Interaction. Piscataway, USA:IEEE, 2017:359-360.
 Alibeigi M, Ahmadabadi M N, Araabi B N. A fast, robust, and incremental model for learning high-level concepts from human motions by imitation[J]. IEEE Transactions on Robotics, 2017, 33(1):153-168.
 黄忠,任福继,胡敏.基于RBF神经网络的人形机器人在线面部表情模仿[J].机器人,2016,38(2):225-232.Huang Z, Ren F J, Hu M. Online facial expression imitation for humanoid robot based on RBF neural network[J]. Robot, 2016, 38(2):225-232.
 Shayganfar M, Rich C, Sidner C L. A design methodology for expressing emotion on robot faces[C]//IEEE/RSJ InternationalConference on Intelligent Robots and Systems. Piscataway, USA:IEEE, 2012:4577-4583.
 Trovato G, Zecca M, Kishi T, et al. Generation of humanoid robot's facial expressions for context-aware communication[J]. International Journal of Humanoid Robotics, 2013, 10(1):No.1350013.
 Tadesse Y, Priya S. Graphical facial expression analysis and design method:An approach to determine humanoid skin deformation[J]. Journal of Mechanisms and Robotics, 2012, 4(2):No.021010.
 马淦.仿人机器人表情与身体动作的人机友好交互研究[D].北京:北京理工大学,2015.Ma G. Human-robot friendly interaction using facial and body motion for a humanoid robot[D]. Beijing, Beijing Institute of Technology, 2015.
 信继忠,柯显信,杨阳,等.仿人机器人头颈协调系统的研制[J].机械科学与技术,2017,36(1):29-33.Xing J Z, Ke X X, Yang Y, et al. Development of head and neck coordination system of humanoid robot[J]. Mechanical Science and Technology for Aerospace Engineering, 2017, 36(1):29-33.
 Jaeckel P, Campbell N, Melhuish C. Facial behavior mapping-From video footage to a robot head[J]. Robotics and Autonomous Systems, 2008, 56(12):1042-1049.
 Magtanong E, Yamaguchi A, Takemura K, et al. Inverse kinematics solver for android faces with elastic skin[M]//Latest Advances in Robot Kinematics. Dordrecht, Netherlands:Springer, 2012:181-188.
 沈旭.基于序列深度学习的视频分析:建模表达与应用[D].合肥:中国科学技术大学,2017.Shen X. Video analysis based on sequence deep learning:Modelling, representation and application[D]. Hefei:University of Science and Technology of China, 2017.
 Ji Z, Xiong K L, Pang Y W, et al. Video summarization with attention-based encoder-decoder networks[EB/OL]. (2017-08-31)[2018-02-01]. https://arxiv.org/pdf/1708.09545.pdf.
 秦超龙,宋爱国,吴常铖,等.基于Unity3D与Kinect的康复训练机器人情景交互系统[J].仪器仪表学报,2017,38(3):530-536.Qin C L, Song A G, Wu C C, et al. Scenario interaction system of rehabilitation training robot based on Unity3D and Kinect[J]. Chinese Journal of Scientific Instrument, 2017, 38(3):530-536.
 Kompatsiari K, Tikhanoff V, Ciardo F, et al. The importance of mutual gaze in human-robot interaction[M]. Cham, Switzerlands:Springer, 2017:443-452.
 杨文璐,郑加悦,占婵,等.基于Kinect传感器的瞳孔定位算法[J].传感器与微系统,2015,34(2):125-128.Yang W L, Zheng J Y, Zhan C, et al. Pupil localization algorithmbased on Kinect sensor[J]. Transducer and Microsystem Technologies, 2015, 34(2):125-128.
 Hochreiter S, Schmidhuber J. Long short-term memory[J]. Neural Computation, 1997, 9(8):1735-1780.
 朱特浩,赵群飞,夏泽洋.利用Kinect的人体动作视觉感知算法[J].机器人,2014,36(6):647-653.Zhu T H, Zhao Q F, Xia Z Y. A visual perception algorithm for human motion by a Kinect[J]. Robot, 2014, 36(6):647-653.