Methodology

Score version . The numbers on this site are computed by a small published program; if this page and that program ever disagree, the program is what counts (for programmers: cdb_score/spec.py and cdb_score/score.py in the leaderboard repository).

The question

A self-driving car "sees" mostly through its sensors. One of the most important is LiDAR, a spinning laser scanner that measures the distance to everything around the car many times per second. In real life that picture gets worse: rain or fog swallows some of the laser returns, dirt covers part of the sensor, far-away objects return too few points.

This bench asks one question: when the LiDAR picture gets worse in a controlled way, how much of its normal driving does a given self-driving software keep? We do not grade the object detector on its own. We grade what the car actually did — did it get closer to hitting something, did it drive less smoothly, did it take longer to finish its trip?

Words used on this site

The bench

"The bench" is the fixed test course every entry goes through. Only the part being compared (for example the object detector) changes between entries; everything else stays identical.

What happens in one run

1. The simulated car starts at the same place on the same city map (CARLA Town10HD) and gets the same destination.
2. The traffic situation of the scenario is placed on the road (for example a car 30 m ahead that then drives slowly).
3. The driving software takes control. 20 simulated seconds later, the degradation for this run's level is switched on and stays on until the end.
4. The run ends when the car has stopped at its goal or can no longer make progress. Every recorded value is written to a per-run file, full_metrics.json.

The degradation that is applied

At every damage level, all LiDAR points farther than 50 m from the car are removed. Then each remaining point is kept only by chance (with a fixed random seed, so every run is reproducible): 75 % of points are kept at S1, 50 % at S2 and 25 % at S3. At S0 nothing is removed.

The software uses the laser for two jobs: spotting other road users, and working out where the car itself is by matching the scan against a stored map (localisation). Only the spotting part gets the damaged picture; localisation keeps the normal one. That way a change in driving can be traced to missed or late-detected road users, not to the car getting lost.

Each run writes down three moments on the simulator's clock: when the damage was asked for, when the damage program started changing the data, and when the first damaged scan reached the driving software. If the three are close together, the damage started when intended; a large gap would reveal a delay, and the run would be checked.

The seven scenarios

How many runs, and which ones count

7 scenarios × 4 levels × 5 repeats = 140 runs for a complete entry. A run only counts if an automatic check accepts it: the degradation was really applied (we measure the point count, we don't just trust the setting), the sensor kept sending data at the normal rate, and every required record is present and matches its checksum (a fingerprint that shows the file was not changed afterwards). A run that fails the check is kept on a separate list of rejected runs and is repeated; it never fills a slot.

The simulator settings are fixed for every entry: CARLA version 0.9.16; the simulated world moves forward in steps of 0.05 s (20 steps per simulated second); the same random seed (20260805) places the traffic; the same simulated car (vEDGAR 1.0.3); Autoware set up with the Technical University of Munich's standard configuration; and the same controller (the part that turns a planned path into steering and pedal commands).

How the score is computed

The idea in one sentence: for every measurement, compare the degraded runs with the same entry's own normal (S0) runs, count only the part that got worse, and express it as a share of an allowed amount.

Worked example

Reference entry, Slow lead scenario, measurement completion time (how long the trip took; shorter is better). Average over the five runs at each level:

1. Allowed amount (the "scale"). For completion time it is 25 % of the normal time: 0.25 × 127.3 s = 31.8 s. Getting 31.8 s slower or more would give retention 0.
2. Retention per level = 1 − (how much worse ÷ allowed amount), kept between 0 and 1.
3. Combine the levels, counting the strongest level S3 twice: (0.94 + 0.91 + 2 × 0.87) ÷ 4 = 0.90.
4. Do this for every measurement in every scenario. Each group score (Safety, Comfort & handling, Operation) is the plain average of all its scenario-and-measurement values, each counting equally, × 100. Skipped pairs (see below) are simply left out of that average. The CDB Index is the plain average of the three group scores.

Four rules that follow from this

• Getting better earns nothing. If a measurement improves under degradation (for example a faster trip), its retention is simply 1. The score measures loss, not gain.
• Every entry is compared with itself. A software that drives cautiously at S0 is not punished for that; only what it loses when the picture gets worse counts. Two entries with different normal behaviour can both score 100.
• Most allowed amounts equal the normal value itself. For most measurements, the allowed amount is the measurement's own S0 value: if a count doubles, that level's retention drops to 0. The table below lists each one.
• Allowed amounts have a minimum. If a normal value is close to zero (for example almost no jerky moments), a tiny change would otherwise look enormous, so each measurement has a floor (listed in the table below). For collisions and "blocked" events the allowed amount is exactly one: one new collision at a level sets that level's retention to 0.
• Missing is skipped, never guessed. Time-to-collision does not exist when there is no other road user (urban curve), and route completion does not exist for the stopped-lead task. These cells are left out, not filled with a made-up value.

For scenario q, measurement m (in score group a), level s ∈ {S1, S2, S3}:
b      = average of m over the S0 runs                    (normal value)
d_s    = average of m over the runs at level s
w_s    = max(0, sign · (d_s − b))                         (how much worse; sign = +1 if lower is better, −1 if higher is better)
r_qms  = 1 − min(1, w_s / scale_m(b))                     (retention at level s, between 0 and 1)
r_qm   = (r_S1 + r_S2 + 2 · r_S3) / 4                     (S3 counts twice)
Group_a = 100 · average of r_qm over all scenarios q and measurements m in group a
CDB     = (Safety + Comfort & handling + Operation) / 3

What is measured

Seventeen measurements, taken from the simulator's exact record of the car's motion (not from what the software believes). A few units, for scale: m/s² measures how quickly speed changes. Gravity is 9.8 m/s²; braking at 3 m/s² is a firm stop, like braking for an amber light, and above about 4 m/s² passengers are pushed noticeably into their seat belts. m/s³ (jerk) measures how suddenly that push starts or stops: the "jolt". Yaw is the car turning left or right, like a spinning top seen from above; rad/s is turning speed (1 radian ≈ 57°). Per km means per kilometre driven. They fall into three groups, the same three the thesis uses: Safety asks whether the car would hit something; Comfort & handling asks whether a passenger would feel uncomfortable or the software had to react abruptly; Operation asks whether the trip was completed and at what cost in time.

Hard braking, severe braking and emergency manoeuvres are counted under Comfort & handling, not Safety. This follows the thesis, which treats them as signs of how abruptly the car had to react, while Safety is kept for outcomes (contact, closeness, leaving the lane). Signals from inside the software, such as how well the car located itself on the map or how many objects it detected, are recorded for every run but are not part of the score, because they are not something the car did.

How sure we are

With five runs per cell, chance plays a real role. To show how much, the whole score is recomputed 2,000 times, each time drawing the five runs of every cell again at random from the ones we have (some twice, some not at all; this is called a bootstrap). We sort the 2,000 results, drop the lowest 2.5 % and the highest 2.5 %, and show the range that is left next to each score, for example "85–92". This 95 % interval is the range in which the score landed in 95 out of 100 recomputations.

If two entries' intervals overlap, the data cannot tell which one is really more robust, even if their numbers differ. The leaderboard still lists entries from highest to lowest score, but a higher place only means something when the intervals are clearly apart.

Score versions

The rules change only through a new score version; every version is recomputed from the same 140 runs of the reference entry, so the numbers below differ only because of the rule change.

What the score does not say

• It is not a real-world safety rating. It holds for this simulator, this map, this route, this vehicle model, this controller and this kind of degradation only. A score of 100 means "nothing measurable was lost here", not "safe".
• No collisions is not the same as safe. A car can avoid every collision and still brake later, drive more jerkily or take much longer. That is why the three groups are shown separately and the raw values stay visible in the table.
• Only collisions after the switch-on count. Before the degradation is switched on, an "S3" run and an "S0" run are identical, so a contact in that period cannot be blamed on the degradation. Since version 1.2 a contact counts only if it happens at or after the first sensor message that differs from normal (or, where no changed message is recorded, the first message after the switch-on; S0 runs get the same switch-on without any change). The times come from the simulator's own event log of each run. In the reference entry, one S2 and one S3 run of the dynamic cut-in scenario contain contacts between the two cars, 14.8 s and 14.6 s before the switch-on; they are kept in the raw record but no longer counted. Version 1.1 counted them, which pulled the cut-in Safety score down to 53.7 (now 72.5).
• The score shows where a change appeared, not which part caused it. Finding out whether the detector, the tracker or the planner is responsible would need extra experiments that swap one part at a time; those are not part of the bench yet.
• Levels are only comparable within one kind of degradation. The thesis behind this site also tested other kinds of sensor damage: a laser with fewer beams, fewer scans per second, scans that arrive late, and the detector "forgetting" objects it had actually seen. "S3" here (points beyond 50 m removed, 25 % of the rest kept) is not the same amount of damage as "S3" of any of those. They are shown separately on the Mechanisms page, never mixed into this index.
• This index is not the thesis's "robustness score". The thesis also reports its own robustness score, between 0 and 1, computed in a different way and over those other experiments. The two answer related but different questions, and their numbers must not be compared with each other.