基于脉冲神经网络的双目事件相机深度估计方法

Stereo Event Camera-based Depth Estimation with Spiking Neural Network

  • 摘要: 事件相机具有高时间分辨率、高动态范围和低功耗等优点,而受神经动力学启发的脉冲神经网络(SNN)具有事件驱动特性,可以模拟大脑的信息处理机制,二者的结合为模仿生物视觉提供了很好的基础。为此,提出了一种新颖的单步推理SNN,利用稀疏的双目事件流实现稠密深度估计,重点解决了如何利用事件数据的高时间分辨率进行高精度立体匹配的问题。具体地,设计了一种更适合SNN兴奋传递的事件数据表达方式,并引入特征关联模块以精准匹配双目事件流中的时空特征,使用U形编-解码器和多预测头架构提取并解码双目事件特征,通过单步推理直接完成深度预测任务。在开源数据集MVSEC的室内、室外序列上的实验结果表明,所提方法在精度与能耗上均优于当前先进的ANN(人工神经网络)、SNN和ANN-SNN混合基线方法,消融实验也进一步验证了各模块的有效性。

     

    Abstract: Event cameras have advantages such as high temporal resolution, high dynamic range, and low latency, and low power consumption, while pulse neural networks (SNNs) inspired by neurodynamics have event driven characteristics that can simulate the information processing mechanism of the brain. Their combination provides a good foundation for mimicking biological vision. Therefore, a novel single step inference SNN is proposed to achieve dense depth estimation from sparse binocular event streams, focusing on how to utilize the high temporal resolution of event data for high-precision stereo matching. Specifically, an event data expression method more suitable for SNN excitation transfer is designed, and a feature correlation module is introduced to accurately match the spatiotemporal features in the binocular event stream. A U-shaped encoder-decoder and a multi-prediction-head architecture are used to extract and decode binocular event features, and the deep prediction task is directly completed through single step inference. The experimental results on indoor and outdoor sequences of the open source dataset MVSEC show that the proposed method outperforms the current advanced ANN (artificial neural network), SNN, and ANN-SNN hybrid baseline methods in terms of accuracy and energy consumption. The ablation experiment further verifies the effectiveness of each module.

     

/

返回文章
返回