Cross-Sensor Validation Annotation: How to Build Ground Truth Across Camera, LiDAR, and Radar

Creating a reliable reference dataset (cross-sensor ground truth) is a challenge for autonomous vehicle (AV) developers. Each sensor sees the world in its own way: cameras provide semantics but are weak at determining distance; LiDAR provides 3D geometry but cannot see through heavy rain or fog; Radar measures speed but generates "noisy" sparse point clouds.

A systems approach to multimodal validation is needed. Let's look at how to build a data annotation process that transforms disparate sensor streams into a single, accurate "truth".

Quick Takes

• Without a clear connection among camera, LiDAR, and Radar data, the autopilot system can make errors due to phantom objects or incorrect depth estimates.
• Using the truth source hierarchy allows you to avoid inconsistencies between camera data.
• The practice of labeling sensor disagreements helps models understand the causes of "blindness" or glare in individual sensors.
• Consensus annotation combines auto-labeling and cross-validation by annotators to achieve high accuracy.

Truth Source Hierarchy

To avoid chaos during markup, annotators and automated algorithms must adhere to a clear hierarchy of truth sources. No sensor is perfect in all scenarios, so the priority varies depending on the characteristic being measured:

Markup rule: To define boundaries and position in 3D, the source is a LiDAR point cloud. To determine the type of object (e.g., truck or bus), a high-resolution camera image is used.

Sensor Disagreement Labeling

The critical step in labeling is when sensor data contradict each other. Instead of ignoring or smoothing these discrepancies, they need to be explicitly labeled (sensor-disagreement labeling).

Typical disagreement scenarios include:

1. Phantom objects from Radar. Radar shows a metal object beneath a bridge (such as a maintenance hole or road sign), but the camera and LiDAR detect open space.
2. Transparent and mirrored surfaces. LiDAR breaks through a storefront or bounces off a puddle, creating "artifacts" that the camera identifies as obstacles or road surfaces.
3. Occlusion. The camera sees a silhouette of a pedestrian behind a chain-link fence, but the LiDAR receives a reflection from the fence itself.

Solution: Add flags like is_radar_ghost, lidar_occluded, or camera_blinded to the Ground Truth metadata structure. This allows machine learning models to understand during training why a sensor is "misleading".

Why is this important for Automotive Perception Validation?

Without cross-sensor validation, an autonomous driving model is at risk of "phantom braking" or missed detection.

When you build cross-sensor ground truth, you create a holistic digital twin of the environment. This ensures that Sensor Fusion algorithms are trained on data in which each sensor complements the other's strengths, minimizing the perceptual system's "blind spots".

Keymakr Case Study

To understand how cross-sensor ground truth works in practice, let's look at the Keymakr team's case study for one of the world's largest technology companies.

Challenge

1. The customer was developing next-generation autopilot algorithms. The task was to build accurate 3D road markings and road edges from LiDAR data, to be combined with geospatial maps. The challenge was that the markings had to be marked with spatial polylines in 3D.
2. Occlusion problem. Cars and pedestrians were covering the road lines. In classic 2D, line breaks are the norm, but in 3D, the autopilot can perceive a broken line as the real end of the lane.
3. Aggregated 3D scenes. Instead of frame-by-frame processing, it was necessary to work with a continuous timeline while preserving semantic connections when lanes branched and merged.

Keymakr Engineering Solution

The Keymakr team turned the marking task into an R&D project by developing special tools.

1. Continuous polylines and auto-deletion of overlaps. Lines were drawn as a single object, and segments under obstacles were automatically deleted. This eliminated the risk of the autopilot misinterpreting gaps.
2. Aggregation in Segments.ai (Merge Mode). Instead of thousands of individual frames, the team combined them into a dense 3D point cloud of the entire route network. This allowed the creation of a holistic "skeleton" of the road within a 10-meter radius of the car.
3. "Confidence Level Points" system. Since 3D platforms supported attributes for the entire line, Keymakr created its own subsystem of local points. Each polyline vertex was assigned a visibility level (Normal, Low confidence, Interpolated).
4. Algorithm for measuring occlusions > 10m. The algorithm built into the converter automatically measured the overlap length and separated the lines, preserving the connections via Master ID.
5. Two-way converter (Reverse Converter). After export, the data was converted back for visual inspection on the 3D platform, ensuring 99% geometric and ID correspondence.

Results

1. +21% to the markup speed thanks to custom automations.
2. Absolute geometric and semantic integrity of the 3D dataset for autonomous driving algorithms.
3. Creation of a universal R&D toolkit for two-way conversion for complex 3D LiDAR projects.

FQA

What to do if LiDAR and the camera show a different location of the object?

This indicates problems with calibration (extrinsic calibration) or timestamp misalignment. If calibration is normal, priority in geometry is given to LiDAR, but the annotator should check the frames from the cameras before and after this moment to exclude motion blur.

Is it possible to build high-quality Ground Truth without Radar, using only LiDAR and cameras?

For 3D detection tasks, LiDAR + Camera is enough. To confirm the speed of objects in difficult weather conditions (rain, fog, snowfall), Radar is indispensable. Without it, automotive perception validation will be incomplete in difficult edge cases.

How to minimize manual labor when labeling millions of frames?

Auto-labeling pipelines based on multimodal models are used. They generate a pre-labeled Ground Truth, and human consensus (Human-in-the-loop) is used only for validation, fixing complex errors, and marking conflicting scenarios.

What is a sensor disagreement flag, and how does it help neural networks?

This is a special mark in the dataset that tells the model: "At this location, sensor A gives a false signal due to physical limitations (e.g., glare)." This prevents the neural network from being "penalized" during training, even as it learns to ignore artifacts from individual sensors correctly.