we introduce e-raft dense optical flow from event cameras. Predicting dense optical flow from events is challenging due to the sparse and asynchronous nature of event data. Our method can accurately predict per-pixel optical flow even in very challenging scenarios where frame-based methods fail. The two key components that enable these results are our e-raft, a novel optical flow method for event-based vision, and d-flow, a dataset with high-quality optical flow ground tools, HDR scenes, as well as day and night sequences. We incorporate cost volumes and iterative refinement into our event-based approach to increase accuracy for large displacements. Therefore, our approach can explicitly use contrast information present in the event stream to boost performance. We start with splitting events into two successive sequences to produce two event representations, respectively. The encoded features are then used to compute the cost volume or to provide additional context to the network. Finally, we propose a differentiable forward working scheme to exploit the sequential nature of event data. We incorporate it to fine-tune the network to further increase the performance without any additional

parameters eraft achieves 0.47 end point error on misac, which is 23 percent lower than ev flownet under the same training setting. In this clip, we follow the visualization strategy of prior work that masks predictions where grantors is available. Indeed, prior work often only visualizes predicted flow at pixels where events have been triggered. This obscures a major limitation of any sec that becomes apparent when dense predictions of the networks are shown. Visualizing the dense predictions reveals that the methods only predict blurry optical flow. [Music] This is the case because the calibration between camera and lighter is inaccurate and that there is a temporal offset due to the lack of hardware synchronization of the sensors. This results in poor alignment between optical flow ground tools and event data, as well as severe limitations for the training of neural networks. To overcome these issues, we propose a new data set with high quality accurately aligned data, each a resolution for the event cameras, 10,000 optical flow maps for dense optical flow estimation with event cameras.

training ear raft on dsec flow leads to sharp optical flow predictions finally raft drastically outperforms eeveeflownet with an endpoint error that is almost three times lower in comparison on dsec flow. Our method also performs well at night, where noisy event data and flashing street lights are especially challenging. Our code and dataset are publicly available.

E-RAFT: Dense Optical Flow from Event Cameras (3DV 2021)

Welcome to the YouTube Channel of the Robotics and Perception Group led by Prof. Davide Scaramuzza. Our lab was founded in February 2012 and is part of the Department of Informatics at the University of Zurich, and the Department of Neuroinformatics, of the University of Zurich and ETH Zurich.

Our mission is to develop autonomous machines that can navigate all by themselves using only onboard cameras, without relying on external infrastructure, such as GPS or motion capture systems. Our interests encompass both ground and micro flying robots, as well as multi-robot heterogeneous systems consisting of the combination of these two. We do not want our machines to be passive, but active, in that they should react to and navigate within their environment so as to gain the best knowledge from it.