Submit results

Anyone can drive their own configuration of the driving software through the bench and put it on the leaderboard. The same checks and the same scoring code that produced the reference entry are applied to every submission; nothing is scored by hand.

Open the submission form ↗

How it works

You drive the bench with your configuration and pack the per-run results into one file, package.json.
You upload that file anywhere reachable by an https link and fill in the submission form (a GitHub issue; you need a free GitHub account).
Within a few minutes a bot downloads the file, checks it, computes the scores and replies on your issue with a score table.
If the file is well formed, the bot opens a pull request that adds your entry. When a maintainer merges it, the leaderboard is rebuilt, your entry appears, and a "New submission" note with the date is shown on the home page. The date on this site only changes when a submission lands.

Complete or incomplete. A complete package covers all 28 cells (7 traffic situations × 4 levels × 5 repeats = 140 runs) and is ranked with a CDB Index and intervals. An incomplete package that is otherwise correct is accepted, but it is shown only per traffic situation, marked "incomplete", and never ranked, because a total over fewer situations would not be comparable. A package with a broken or missing measurement, an unknown scenario, or a degraded run without proof that the degradation was applied is rejected with a list of what to fix.

1 · Run the bench

The bench runner drives one row of a run list at a time on a machine with the reference simulator stack, retries a rejected run, and never lets a rejected run fill a slot. The list for a complete entry has 140 rows (scenario,severity,repeat).

python3 bench/cdb_bench_runner.py --matrix matrices/full_140.csv --out results/<entry-id> --execute

Each accepted run leaves a folder with manifest.yaml and metrics/full_metrics.json, computed by the same compute_run_metrics.py that produced the reference entry. What you can change between entries:

A detector that ships with the stack	Set `lidar_detection_model` (`centerpoint`, `centerpoint_tiny`, `transfusion`, `pointpainting`, `bevfusion_lidar`, `bevfusion_camera_lidar`) and `model_path` in the launch environment.
Your own detector	Read the degraded LiDAR data from `/sensing/lidar/degraded/pointcloud` and publish `autoware_perception_msgs/DetectedObjects` on `/perception/object_recognition/objects`. Everything after detection stays the reference build so the entry remains comparable.
Controller	Keep the baseline controller unless the entry is explicitly a controller comparison; a different controller is a different entry.

2 · Make the package

python -m cdb_score package --runs results/<entry-id>/runs --entry entry.json --out package.json

entry.json describes your configuration:

{"id": "myorg_centerpoint_baseline", "model": "Autoware 0.3.8 · CenterPoint", "creator": "My org",
 "detector": "centerpoint", "fusion_mode": "lidar", "controller_profile": "baseline",
 "stack_version": "autoware/microservice/mono:cuda-humble-x86_64-0.3.8", "commit": "…"}

The command copies the scored measurements of every accepted run, adds each run's SHA-256 fingerprint and leaves out rejected runs. The package format is cdb-submission/1.0; submissions/SCHEMA.json in the repository describes it field by field.

3 · Check it yourself

python -m cdb_score validate package.json   # PASS (complete), PARTIAL (incomplete) or FAIL, with reasons
python -m cdb_score score    package.json   # the numbers the bot will post

This is exactly the code the bot runs, so if it says PASS here it will say PASS on GitHub. The scoring code has no dependencies beyond Python 3.10+.

4 · Submit

Upload package.json so it has an https link: attach it to a GitHub release of your own repository, or paste it into a gist and use its "Raw" link.
Open the submission form, fill in the description of your configuration and paste the link.
Keep your raw run folders (recordings, simulator ground truth) for one year so that anyone can re-derive the measurements from source if a result is questioned.

What happens next

The bot replies with ✅ (complete, scored), 🟡 (incomplete, accepted, not ranked) or ❌ (list of problems). Edit the issue to try again; it re-checks automatically.
✅ and 🟡 open a pull request titled Submission: <entry id>. Merging it rebuilds leaderboard.json and the site within a few minutes.
Your entry is compared with its own S0 runs, so you do not need to beat anyone at normal driving to score well; see how the score is computed.

Access to the reference stack

The reference images (autoware/microservice/mono:cuda-humble-x86_64-0.3.8, CARLA tum_0.9.16, vEDGAR 1.0.3) and the tum_models detector weights are hosted on the TUM LRZ GitLab registry and currently need an account there. A publicly downloadable image set is the main open item before outside teams can run the bench without help; until then, contact the maintainer to arrange access or a run on the reference machines.