Applies to v0.1.0.

Segmentation dataset

Build a semantic-segmentation dataset (paired RGB images and ground-truth labels) straight from the simulator over the Python API. The built-in expert drives the track while the collector saves each frame.

What you get

dataset/
├── rgb/     frame_000001.jpg …   RGB image        (JPEG, light)
├── mask/    frame_000001.png …   segmentation GT  (PNG, LOSSLESS, colour-coded)
├── classes.json                  the class legend (name + RGB)
└── manifest.jsonl                one JSON line per frame (paths, shape, pose)

The label in mask/is the segmentation camera's flat, colour-coded render, saved losslessly. Every pixel is exactly one class colour, so it doubles as a viewable image and a precise label. You turn colour → integer class index at training time (see Labels & training).

How it works

Alongside the normal RGB camera, the sim renders a segmentation view where every pixel is painted the single solid colour of its class: road, lane, obstacle, and so on. That colour image is the label: no hand-annotation, and it lines up with the RGB frame pixel-for-pixel.

The collector connects over the Python API (SimServer, TCP 7720), hands the car to the built-in expert autopilot (route_follow), and for each frame pulls both images with get_frame_bin — saving the RGB as a JPEG and the segmentation render as a lossless PNG. Because the segmentation view renders flat, with no antialiasing (edges are never smoothed), a pixel is never a blend of two class colours, so converting colour back to a class index later is exact, with zero unmapped pixels.

Prerequisites

• Simulator running — launch Nano-sim (see Installation); its Python API is then reachable at 127.0.0.1:7720.
• The camera's Segmentation stream enabled in the Sensor Setup panel, and its name must contain your --device string (default veye).
• Python: run the collector inside a virtual environment with the dependencies installed (see Environment) for the one-time setup.

Run the collector

cd examples/collect_dataset

# basic: 3000 frames from the camera, one lap
python collect_seg_dataset.py --out dataset --frames 3000

# a good training set: several laps, temporal subsampling, expert steering noise
python collect_seg_dataset.py --out dataset --frames 6000 \
    --loops 6 --stride 3 --steer-noise 0.05

• --out (dataset) — output directory
• --device (veye) — substring of the camera name to capture
• --frames (2000) — how many frames to save
• --loops (1) — stop after this many completed laps
• --stride (1) — save every Nth frame; raise it to decorrelate consecutive frames
• --steer-noise / --throttle-noise (0.0) — expert noise for off-centre / recovery frames
• --warmup-ticks (5) — skip the first settling frames
• --jpeg-quality (95) — RGB JPEG quality (the mask is always lossless PNG)
• --host / --port (127.0.0.1 / 7720) — SimServer endpoint

It runs in sync mode: each tick advances physics deterministically and pairs exactly one RGB frame with one seg frame, so no pair is ever dropped or misaligned. It stops at --frames or after --loops laps, then halts the expert and disconnects cleanly.

How the collector works

The core of collect_seg_dataset.py is four pieces.

1 · Start the expert

route_follow takes action: "begin" to start the autopilot (optionally with steering / throttle noise for augmentation) and action: "halt" to stop it:

def begin_route(c, args):
    c.call("route_follow", {"action": "begin",
                            "steer_noise": args.steer_noise,
                            "throttle_noise": args.throttle_noise})

2 · Read the binary frame

get_frame_bin returns a length-prefixed binary blob: a 4-byte total, a 4-byte header length, a JSON header listing each sensor, then the concatenated pixel payloads. Images arrive bottom-up as raw RGB24, so the collector flips them top-down:

total  = struct.unpack("<I", recv(4))[0]
body   = recv(total)
hlen   = struct.unpack("<I", body[:4])[0]
header = json.loads(body[4:4 + hlen])          # {"sensors": [...]}
pixels = memoryview(body)[4 + hlen:]
# per image sensor: reshape to s["shape"], then arr[::-1] to flip bottom-up

3 · Pair RGB with segmentation

pick_image_keys finds the RGB/seg pair for the requested --device: the camera whose key contains seg becomes the label, and its sibling (same base key) is the RGB. If no seg image is present, the run aborts with a reminder to enable the Segmentation stream.

4 · Tick, subsample, save

The loop ticks in sync mode, skips --warmup-ticks settling frames, keeps every --stride-th frame, and writes rgb/frame_NNNNNN.jpg + mask/frame_NNNNNN.png plus a manifest.jsonl line (paths, shape, and the pose / route metadata). On a lap finish it counts the lap and resets; after --loops it breaks. A finally block always halts route_follow, leaves sync mode, and disconnects, so the sim is never left driving.

Masks stay lossless. RGB is saved as JPEG (q95) to save space, but the label is always a lossless PNG, because a label must never be resampled or lossily recompressed.