Real-Time, Physics-Based Co-Simulator for Off-Road Autonomy
At WPI’s Autonomous Vehicle Mobility Institute (AVMI), I am leading the research and development of a real-time co-simulator that integrates physics-based modeling and simulation, autonomy, and mixed-reality testing to support the development and evaluation of the next generation of autonomous off-road vehicles.
Key features
- Real-time simulation
- Physics-based vehicle dynamics and terramechanics
- Novel virtual driveline concept for modeling independent-wheel-drive vehicles
- High-fidelity off-road environment creation
- Physics-based sensor simulation
- Human-in/on/off-the-loop simulation
- Full autonomy stack including intelligent perception, mobility-based planning, advanced control
- Mixed-reality co-simulation
- Real-synthetic data fusion
- Real-world, synthetic, and real-synthetic fused data generation
- Intelligent cyber-physical off-road platform
Complete Co-Sim Architecture
Integrates four elements into one immersive mixed-reality co-simulation environment: a physics-based virtual world with vehicle dynamics and sensor models, an intelligent autonomy stack, human-in/on/off-the-loop simulation through a physical cockpit and immersive display, and a cyber-physical off-road vehicle platform. A 5G data stream links them in real time, so virtual and real components operate as a single system.
Capabilities of the simulator
The platform is designed to provide a modular, ready-to-use research environment in which users can focus on their specific area of interest without having to develop or configure the entire autonomy pipeline. For example, a control researcher can develop and evaluate new control algorithms while leveraging existing vehicle models, sensor models, perception, and planning modules provided by the platform. Similarly, researchers can independently investigate perception, planning, vehicle dynamics, or other components while interfacing with the rest of the integrated system. This enables efficient development, testing, and validation of autonomous systems in complex on- and off-road environments.
Modeling and Simulation of Vehicle System Dynamics
In this system, a standalone simulator was developed for real-time, physics-based simulation of vehicle dynamics at 1 kHz, including nonlinear tire–ground interactions, multibody steering and suspension models, and engine and driveline dynamics. It also incorporates a novel virtual driveline concept for modeling and simulating vehicles with independent wheel-drive motors.
High-Fidelity Terrain Generation
Uses a novel integration of a variational autoencoder (VAE) and W–M noise to generate 1 km × 1 km landscapes with detailed surface roughness at spatial resolutions as fine as 2 cm. A newly developed method for quantitatively assessing vehicle topographical mobility provides constraints for the generation process, enabling terrain to be tailored to mobility evaluations of specific vehicle configurations.
Off-Road Environments for Sensor Simulation and Vehicle–Environment Interaction
Creates off-road environments with both photorealism and structural realism in under five minutes. An automated pipeline adds surface textures to the terrain topography and places obstacles in designated regions. Currently, the system supports generation of forest, desert, and mountain environments.
Modeling and Simulation of Vehicle Exteroceptive Sensors
The system supports real-time, physics-based simulation of RGB cameras and 3D LiDAR, with segmentation outputs for both images and point clouds.
Intelligent Perception
As part of the autonomy stack, the system integrates camera imagery and 3D LiDAR data for semantic scene understanding and depth estimation. A SegFormer-B2 transformer performs image semantic segmentation, while the results are fused with LiDAR measurements to associate identified scene regions with depth information.
Real-Time Topographical Mobility Assessment
The autonomy stack can evaluate vehicle topographical mobility in real time using the point cloud from every single LiDAR sweep, identifying terrain regions the vehicle can traverse without chassis–terrain collision. The resulting topographical mobility assessment informs real-time motion planning.
Mobility-Aware Motion Planning and Advanced Control
The system supports global and local path planning using predefined go/no-go maps and real-time topographical mobility assessments. A baseline trajectory-following controller with adaptive speed control is currently implemented. A model predictive control (MPC) strategy based on the virtual driveline concept is being integrated into the autonomy stack to enhance the vehicle’s off-road maneuverability.
Full Autonomy Stack
The system integrates multimodal perception, real-time topographical mobility assessment, motion planning, and vehicle control into a unified autonomy framework. Together with the vehicle simulator, it forms a complete pipeline for simulating and evaluating autonomous off-road vehicle operation.
Intelligent Cyber-Physical System
A purpose-built electric off-road UGV with independent-wheel-drive motors serves as the physical counterpart of the simulator. It carries cutting-edge sensors and controllers, including a 3D LiDAR, a stereo camera, GNSS/INS, and wheel encoders, with dSPACE embedded computers running perception, planning, and control through a real-time RTMaps/ROS 2 sensor-fusion system. The platform validates simulation-developed autonomy on real terrain and collects real-world data for model training and validation.
Mixed-Reality Environment
Connects virtual simulation with the cyber-physical vehicle platform through 5G communication, enabling the exchange and fusion of real and simulated sensor data. This environment supports simulation validation and fidelity improvement while expanding the range of operational scenarios available for testing physical vehicles.
Human-in-the-Loop Simulation
A human-in-the-loop driving interface places the operator directly inside the simulation. A professional direct-drive cockpit with up to 25 N·m of force feedback lets the driver feel the simulated tire–terrain forces, while a curved immersive display cave with a 210° field of view or a VR headset presents the environment. The cockpit uses the same steering, throttle, and brake commands as the autonomy controller, so manual driving and autonomy supervision run in one loop.
Annotated Synthetic Sensor Data Generation
The simulator doubles as a data generation system, recording synchronized RGB images, pixel-level semantic masks, raw and semantically labeled LiDAR point clouds, and vehicle state, all saved in standard ROS 2 bags. All annotations are generated automatically and exactly by the renderer, with no manual labeling.
A dataset of 1 million synchronized, annotated LiDAR and camera samples with vehicle state information, covering forest, desert, and mountain environments, will be publicly released soon.
Every component is developed in-house, from the vehicle dynamics solver and the physics-based sensor models to the realistic environment generation and the autonomy stack itself. Nothing is adapted from an existing simulator.