Sign in

V2X Data Annotation: Labeling Vehicle-to-Everything Communication for Cooperative Perception

V2X Data Annotation: Labeling Vehicle-to-Everything Communication for Cooperative Perception

Autonomous cars in traditional configurations face physical limitations: blind spots around corners, pedestrians blocked by oversized trucks, fog, or the limited range of their own LiDAR sensors. The concept of cooperative perception, enabled by V2X (Vehicle-to-Everything) technologies, helps the autopilot see the world through the eyes of other cars, road cameras, and sensors on infrastructure.

However, for fusion algorithms to combine data from cars and infrastructure in real time, an additional layer of data preparation is required. Let's analyze how V2X annotations are built, the challenges annotators face when creating a C-V2X dataset, and how infrastructure sensor ground truth is formed.

Quick Take

  • Cooperative perception data merges viewpoints into a single spatiotemporal continuum.
  • Roadside unit annotation provides a bird's-eye view, allowing you to see around the corners of buildings and detect obstacles hundreds of meters away.
  • When communicating via V2X, network delays create a "time shift." If the annotation does not account for millisecond-accurate timestamps, the cooperative system will generate a phantom object.
  • Creating a Cellular-V2X-compatible dataset requires double validation.

Cooperative Perception Space Architecture

In a V2X system, cooperative perception is built by combining two different viewpoints into a single 3D intersection scene:

Top level — Roadside unit (RSU):

  • Viewpoint: Fixed camera or LiDAR mounted on a pole or traffic light.
  • Perspective: Top-Down View.
  • Role: Provides a general panorama of the intersection, detects objects behind obstacles, and sees blind spots that are blocked by other vehicles.

Bottom level — Connected ego vehicle:

  • Viewpoint: Mobile onboard sensor system of the vehicle.
  • Perspective: Road level view.
  • Role: Provides details of objects in front of the car, recognizes the status of traffic lights at close range, and transmits its own kinematic data.

The result of the merger: Data from both levels enters a single Cooperative Space, where fusion algorithms synthesize them into a common 3D model of the environment.

Why does V2X Annotation require new approaches?

Traditional 3D annotation uses a connected coordinate system for a single vehicle. In V2X annotation, the coordinate system is extended to a global one (GPS/GNSS + HD Maps) or is tied to a specific intersection area.

The main V2X data channels are:

  1. V2V (Vehicle-to-Vehicle). Cars share onboard LiDAR point clouds and object detection rectangles.
  2. V2I (Vehicle-to-Infrastructure). The Roadside Unit (RSU) infrastructure module transmits data from high-altitude cameras and stationary LiDAR sensors.

Markers must create an end-to-end C-V2X dataset in which the same pedestrian crossing the road receives a unique ID, even if the car's onboard camera does not yet see him, but the RSU camera has already recorded him.

Anatomy of Vehicle-to-Infrastructure Labeling

The vehicle-to-infrastructure labeling process differs from classic dashcam video labeling. Infrastructure sensors are installed at high angles (3–10 meters above the ground) and have a different perspective.

Because the RSU is in a fixed position, the infrastructure marking acts as an infrastructure sensor ground truth – the "absolute truth" for calibrating the onboard systems of cars passing by.

Creating a C-V2X Dataset

Creating a dataset for cooperative perception data requires a clear sequence of steps:

1. Cross-sensor extrinsic calibration:

Step 1. Alignment of the coordinate systems of the vehicle's onboard sensors and the stationary RSU. All data is converted to a single metric coordinate system (for example, UTM or a locally accurate HD map).

2. Timestamp alignment:

Step 2. The delay in data transmission over the C-V2X network is from 10 to 100 ms. Annotators and automated scripts adjust object positions, taking into account the velocity vector and latency compensation.

3. Parallel marking and global ID tracking:

Step 3. Objects are placed simultaneously on the streams from the car and the RSU. A single Global Track ID is assigned. If the object disappears from the car's field of view, but remains in the RSU's field of view, its track is not interrupted.

4. Validation and disagreement labeling:

Step 4. Automatic verification: if the airborne LiDAR sees an object at point A and the infrastructure LiDAR sees an object at point B, a calibration or occlusion error flag is set for further manual verification.

Quality Control and QA in V2X Data Annotation

Safety at complex controlled intersections depends on the accuracy of cooperative perception data, so strict requirements are imposed on datasets:

  1. Synchronization accuracy. The deviation in the frame timing should not exceed 5 ms.
  2. Geometric accuracy of 3D-bounding boxes. The deviation of the object center between the car's vision and the RSU cannot exceed 10 cm.
  3. Edge cases processing. A special flag for situations when the C-V2X radio signal is interrupted (packet loss) or shaded by tall buildings.

Keymakr Case: Engineering Approach to Complex 3D LiDAR Labeling for Autopilots

Building sensor ground truth infrastructure and labeling complex V2X scenarios requires data developers to develop custom R&D tools. An example of such an approach is the Keymakr project.

Project challenge

The client was developing autonomous driving algorithms that required accurate 3D labeling of the road surface and its boundaries based on LiDAR point clouds for further synchronization with geospatial data.

Challenges the team faced

  • Complex occlusions. Moving and static objects constantly covered the labeling. In classic 2D, a line break is not critical, but in 3D, the autopilot can perceive a small "pause" in the data as a real lane end or hazard.
  • Aggregated 3D scenes. The work was carried out not with step-by-step individual frames, but with a continuous timeline, where it was necessary to preserve semantic connections when branching or merging lanes.

Engineering solution from Keymakr

To solve these problems, the Keymakr team turned the annotation process into a full-fledged R&D solution:

  • Aggregation of frames into 3D space. The team combined the stream of individual frames into a single three-dimensional point cloud of the entire route network. This allowed working with a holistic spatial model of the entire road and surrounding objects within a radius of 10 meters.
  • Continuous polylines and auto-deletion of overlaps. Lines were drawn as a single object, and segments that fell under obstacles were automatically deleted. This eliminated the risk that the model would perceive occlusion as the end of a continuous lane.
  • Local detailing "Confidence Level Points". Since the 3D platform supported attributes only for the object as a whole, Keymakr developed a custom system of local points near each polyline vertex. It fixed the visibility level of the area (Normal, Low confidence, Interpolated).
  • Automatic measurement of occlusions > 10 meters. The algorithm inside the converter automatically calculated the length of the shaded area and, if necessary, divided the line, preserving the connection through a single Master ID.
  • Two-way converter. To verify the data, a reverse import tool was developed into the 3D platform, which provided 100% transparency and compliance with the client's requirements.

Result

  • Markup speedup by 21% due to automation of routine measurements.
  • Geometric and semantic integrity of the 3D dataset for autonomous driving algorithms.
  • Creating an infrastructure and tools for two-way data conversion, which is now used for the most complex 3D LiDAR and V2X projects.

FAQ

Why can't infrastructure cameras (RSUs) completely replace the onboard sensors of an autopilot?

RSUs cover only a limited number of areas (complex intersections, interchanges, and bridges). On highways or country roads without V2X coverage, the car must rely on its own sensors.

What is C-V2X, and how is it different from regular V2X?

C-V2X (Cellular Vehicle-to-Everything) uses mobile networks (4G LTE/5G) to transmit data between devices (Direct Sidelink Communication) and through cell towers. This provides lower latency and longer range than the outdated Wi-Fi-based DSRC standard.

How is a V2X dataset marked if the signals from the car and the infrastructure are out of sync?

Interpolation and extrapolation algorithms (e.g., Kalman filters) are used. Based on the object's velocity vector, acceleration, and rotation angle, its position is calculated at the exact timestamp of the RSU frame, and the annotator then validates the resulting position.