rapid perception and localization

Sprinting, jumping, parkour, backflips — the motions we want legged robots to perform are exactly the ones that break conventional cameras. Rapid motion causes severe blur and compromises pose estimation right when it matters most. We use event cameras, which offer exceptional temporal resolution and dynamic range and largely avoid this blur, but their sparse, asynchronous data stream requires new methods for feature extraction, association, and tracking. This project develops those methods and the datasets needed to validate them.

Terrain scanning with an RGB-D camera and event-camera-based pose estimation identify steppable regions for foothold planning in cluttered, rubble-like terrain.

Research and Publications

Event Camera-Based Visual Odometry for Dynamic Motion Tracking of a Legged Robot Using Adaptive Time Surface

An adaptive time-surface representation for event-camera data that supports visual odometry through highly dynamic legged motion, keeping track of pose and surroundings during moves that would blur a conventional camera. Presented at IROS 2023.

Adaptive time-surface visual odometry running through dynamic quadruped motion.

CEAR: Comprehensive Event Camera Dataset for Rapid Perception of Agile Quadruped Robots

CEAR is a multimodal dataset combining event-camera data with RGB-D, IMU, LiDAR, and joint encoders, all mounted on a Mini Cheetah. It includes more than 100 real-world sequences across a range of gaits and acrobatic movements under diverse lighting conditions — the first event-camera dataset capturing this range of dynamic quadruped motion. Published in IEEE Robotics and Automation Letters, 2024.

An overview of the CEAR dataset.

See the UMass Robotics rapid perception and localization project for the broader research overview.