Independent
closed-loop evaluation
of AD stacks
Self-driving software sees the road mostly through a laser scanner (LiDAR). This site measures how much of its normal driving the software keeps when that picture is deliberately made worse, step by step, in a driving simulator. Every entry drives the same seven traffic situations, with the normal picture and three levels of damage, five times each. We score what the car actually did, in three groups: Safety (did it get close to hitting something), Comfort & handling (did it brake, swerve or jolt more) and Operation (did it finish the trip, and how much longer did it take). A score of 100 means the software kept all of its normal driving when the sensor got worse; lower means it lost some. Each entry is compared only with its own normal driving, so the score measures robustness, not how good the driving is overall. Scores are provisional (see the methodology); since score version 1.2 a collision counts only if it happens after the degradation was switched on, and two entries whose uncertainty ranges overlap should not be called better or worse than each other. How the score works →
Safety
Collisions after onset, minimum TTC, TTC warnings and lane invasions — retained at S1–S3 relative to S0.
Comfort & handling
Longitudinal jerk, jerk magnitude, yaw acceleration, yaw rate, lateral acceleration, peak deceleration, MRM events and hard / severe braking — dynamic burden and emergency handling.
Operation
Completion time, no-progress duration, blocked events and route completion — the operational cost of finishing the task.
| Model | Robustness index | Absolute at S3 (mean over scenarios) | Provenance | |||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Model creator · detector · fusion · controller | CDB Index equal-weight mean | Safety 0–100 | Comfort & handling 0–100 | Operation 0–100 | Collisions after onset, per run | Min TTC s | Route completion fraction | Completion time s | Runs accepted | Verified |
Grey rows are configurations that exist in the reference stack (tum_models) but have not been run through the full 140-slot matrix; they carry no numbers until a validated package is submitted. Intervals are 95 % percentile bootstrap over accepted runs resampled within every scenario × severity cell (2 000 draws, seed 20260907). Absolute columns are shown for context and do not enter the index — a faster, more complete run can still collide.
What we do to the sensor
The car's LiDAR (a laser scanner that measures distances as a cloud of 3-D points) is made worse on purpose. S0 is the normal, undamaged picture; S1, S2 and S3 are three levels of damage. At each damage level every point farther than 50 m is removed, and of the remaining points only 75 % (S1), 50 % (S2) or 25 % (S3) are kept. Which points survive is random, but the random choices are fixed in advance so every rerun is identical. The software also uses the laser to work out where the car is on its map; that part keeps the normal picture, so only the spotting of other road users is tested. The damage is switched on 20 simulated seconds after the car starts driving, and every run counts the laser points before and after to prove the damage really took effect.
How many drives, and which count
7 traffic situations × 4 levels (normal + 3 damage levels) × 5 repeats = 140 drives per entry. Each combination is driven five times because tiny timing differences make no two drives exactly alike. A drive only counts if an automatic check confirms three things: the damage was really applied, the laser kept sending its usual 10 scans per second, and every record file is present and unedited. "Unedited" is checked with a digital fingerprint of each file (a SHA-256 hash), which changes completely if even one character is altered. Drives that fail the check are listed separately and redone; they never count.
Everything else stays fixed
Same driving simulator (CARLA 0.9.16) and city map (Town10HD), same simulated research car (vEDGAR 1.0.3, a digital copy of the test vehicle EDGAR of the Technical University of Munich), same start, same goal, same fixed random choices. The driving software is always Autoware, a widely used open-source self-driving program. Autoware can run different object detectors (the part that finds cars and people in the laser picture), with or without help from cameras. An entry states which detector and which of these options it uses; everything else, including the controller that turns the planned path into steering and pedal commands, stays the same. A different controller counts as a separate entry.
Seven traffic situations
Stopped lead: stop behind a standing car. Slow lead: follow a car doing 9 km/h. Urban curve: an empty curved road, the control case with nothing to detect. Occluded pedestrian: someone steps out from behind a parked car. Cross traffic: a car runs its red light into the junction while ours has green. Construction obstacle: the lane is blocked. Dynamic cut-in: a car moves into our lane just ahead. Each situation needs the sensor for different things, so scores are shown per situation before they are averaged. Full description →