Student Projects

Develop a vision-based aerial transportation system with reinforcement / imitation learning.

Explore online fine-tuning in the real world of sub-optimal policies.

This project investigates how neuromorphic vision can enhance motion understanding for mobile systems. By leveraging high-temporal-resolution sensing from event cameras, it aims to reliably separate ego-motion from independently moving objects for improved perception and navigation.

This project explores lightweight 3D scene representations for embedded vision systems such as drones. By exploiting the sparsity and high temporal resolution of neuromorphic sensors, it aims to achieve compact yet visually faithful 3D reconstructions suitable for resource-limited devices.

This project explores how event-based sensing can enhance perception for autonomous driving systems. It investigates the integration of asynchronous, high-temporal-resolution visual signals with conventional sensing modalities to improve robustness, latency, and reliability under challenging real-world conditions.

Automating mitral valve segmentation for AI-assisted preoperative planning.

Recent research has demonstrated significant success in integrating foundational models with robotic systems. In this project, we aim to investigate how these foundational models can enhance the vision-based navigation of UAVs. The drone will utilize learned semantic relationships from extensive world-scale data to actively explore and navigate through unfamiliar environments. While previous research primarily focused on ground-based robots, our project seeks to explore the potential of integrating foundational models with aerial robots to enhance agility and flexibility.

Use Inverse Reinforcement Learning (IRL) to learn reward functions from previous expert drone demonstrations.

We aim to learn vision-based policies in the real world using embedded optimization layers within reinforcement learning.

This research project aims to develop and evaluate a meta model-based reinforcement learning (RL) framework for addressing variable dynamics in flight control.

Design a model-based / learning-based controller that is aware of state observability and sensor perception objectives for object reconstruction using a quadrotor with an egocentric camera.

Explore the possibility of high-speed drone racing using raw RGB camera images only.

Explore the possibility of catching/picking up an object during high-speed agile flight, with potential application in fast turnaround delivery.

Develop a learning-based decoder for event cameras to facilitate efficient real-world tasks.

This project develops event-based representation learning methods for control tasks in environments with visual distractors, leveraging sparse, high-temporal-resolution event data to improve robustness and efficiency over traditional frame-based approaches.

This thesis develops hardware-optimized recurrent neural network architectures with novel parallelization and kernel-level strategies to efficiently process event-based vision data for real-time neuromorphic and GPU-based applications.

This thesis explores novel spiking neural network architectures that leverage event-based vision data and emergent neural synchronization to enable efficient and robust temporal reasoning for dynamic scene understanding and sequential tasks.

Single-Photon Avalanche Diodes (SPADs) represent a transformative technology for robotic perception, overcoming the limitations of conventional cameras in high-speed, low-light, and high-dynamic-range environments. While Event Cameras have advanced agile robotics, they remain limited by contrast sensitivity; conversely, SPADs offer superior microsecond temporal resolution and single-photon sensitivity. However, the unique nature of SPAD data presents a significant challenge, as current Simultaneous Localization and Mapping (SLAM) pipelines are not optimized for this distinct modality, necessitating the development of specialized processing frameworks.

In this project, you will investigate the use of event-based cameras for vision-based landing on celestial bodies such as Mars or the Moon.

Humanoid robots require tactile sensing to achieve robust dexterous manipulation beyond the limits of vision-based perception. This project develops an event-based tactile sensor to provide low-power, high-bandwidth force estimation from material deformation, with the goal of integrating it into a human-scale robotic hand.

This project explores generative modeling of drone flight paths conditioned on natural language commands and spatial constraints, aiming to produce plausible 3D trajectories for training reinforcement learning policies. We investigate model architectures, data sources, and trajectory extraction methods to ensure generated paths are both physically feasible and stylistically aligned with textual descriptions.

This project develops an end-to-end pipeline for object segmentation using event camera data, leveraging their microsecond latency, high dynamic range, and sparse asynchronous output to overcome the limitations of frame-based vision. By combining advanced deep learning architectures with rigorous benchmarking, the approach aims to deliver accurate, temporally consistent, and robust segmentation for real-time applications in navigation and control.

Surgical environments present an extreme HDR challenge, causing standard cameras to lose critical detail due to overexposure. This project introduces a neuromorphic sensor fusion framework integrating RGB sensors with asynchronous event cameras to overcome these limitations. By exploiting the event camera's superior dynamic range ($>120$ dB) and temporal resolution, our method recovers texture and motion in saturated areas. The result is robust HDR reconstruction and low-latency tracking that outperforms traditional optical image enhancement techniques, ensuring reliable computer vision performance in harsh intraoperative lighting. This project is conducted at RPG and Balgrist under the supervision of Prof. Dr. Fürnstahl.

This project addresses autonomous landing of a high-speed quadrotor after aggressive racing, using only onboard camera data without reliable state estimation. The drone must land safely within track boundaries while avoiding obstacles, despite track symmetries and limited information.

This project explores using reinforcement learning to reduce the computational load of visual–inertial odometry by selecting only the most informative camera frames for processing. An RL policy decides when a new frame is needed versus when inertial propagation is sufficient, integrating IMU cues, feature statistics, and uncertainty estimates into a hybrid VIO pipeline.

This project targets robust landing in GPS-denied, highly dynamic scenarios—such as moving or vibrating platforms—where conventional RGB-camera SLAM fails due to blur and latency. By leveraging event cameras’ high temporal resolution and robustness to rapid motion, drones can maintain accurate localization and stability during challenging, fast landings.

In this project, we are going to develop a vision-based reinforcement learning policy for drone navigation in dynamic environments. The policy should adapt to two potentially conflicting navigation objectives: maximizing the visibility of a visual object as a perceptual constraint and obstacle avoidance to ensure safe flight.

This project focuses on developing robust reinforcement learning controllers for agile drone navigation using adaptive curricula. Commonly, these controllers are trained with a static, pre-defined curriculum. The goal is to develop a dynamic, adaptive curriculum that evolves online based on the agents' performance to increase the robustness of the controllers.

Vision-based reinforcement learning controllers can achieve impressive drone flight performance, but training them is often slow and data-hungry because standard networks must re-learn the same behaviors across equivalent viewpoints and orientations. In this project, we will speed up vision-based drone control policy learning by using structured (symmetry-aware / equivariant) neural networks that encode physical and geometric symmetries directly into the policy. By enforcing these structure constraints, the controller can generalize better across rotations and scene variations, improving sample efficiency and sim-to-real transfer. Applicants should have a solid understanding of reinforcement learning, machine learning experience (Jax, PyTorch), and programming experience in Python and C++.

In this project, we investigate how DiffSim (differentiable simulation) can accelerate learning for drone racing by enabling end-to-end gradient-based training with Backpropagation Through Time (BPTT). Instead of relying solely on sample-inefficient trial-and-error, we use a differentiable simulator to propagate learning signals through the drone dynamics and control pipeline over time, allowing the controller to improve from trajectories much more efficiently. Our objective is to develop a training framework that learns high-speed, gate-to-gate racing behaviors in significantly less wall-clock time, while maintaining stable and agile flight. Applicants should be comfortable with control and learning for dynamical systems, have machine learning experience (e.g., Jax, PyTorch), and be proficient in C++ and Python.

This project leverages the Neural Robot Dynamics (NeRD) framework to build a high-fidelity, high-speed neural simulator for a quadrotor UAV. Rather than relying solely on classical rigid-body dynamics and hand-crafted aero/actuation models, we will train a neural network to replace key low-level simulation components (e.g., unmodeled dynamics, actuator response, residual forces), enabling faster rollouts without sacrificing accuracy. The simulator will be fine-tuned on real flight logs to learn a vehicle-specific model and reduce the sim-to-real gap. Thanks to the model’s differentiability, the resulting engine also supports differentiable simulation (DiffSim) for gradient-based system identification, trajectory optimization, and policy learning. Ultimately, we aim to accelerate training of advanced flight control policies and improve zero-shot transfer by matching simulation to the target platform’s true dynamics. Applicants should be comfortable with control and learning for dynamical systems, have ML experience (e.g., JAX/PyTorch), and be proficient in C++ and Python.

The goal is to investigate how the latest differentiable simulation strategies push the limits of learning real-world tasks such as agile flight.

We aim to co-design a controller and the shape of a glider for a perching maneuver, involving deployment on the real system.

In this project, we will use uncertainty-aware localization and an MPC layer within an RL policy to navigate challenging environments.

This project investigates perception-aware path planning for autonomous drones in sparse and challenging environments. By combining vision-based localization with combinatorial and mixed-integer optimization, we aim to enable reliable real-world flight where sensing and planning are tightly coupled.