Building Dataset: Calibration & Time Synchronization

Point projection visualization

To ensure the proper alignment between the used camera, LiDAR, radar, and DGPS, extrinsic calibration is essential. Even seemingly straightforward projections like the one above required some amount of effort to be implemented accurately.

Initially, extrinsic between LiDAR and DGPS is obtained through laser scanning during installation. Additionally, LiDAR-radar extrinsic can be relatively easily obtained. Therefore, the focus was on calibrating the camera and LiDAR.

Camera Intrinsic

For camera-LiDAR extrinsic calibration, I first needed to find the intrinsic parameters. Although intrinsic can be simply calculated using a few lines of code using OpenCV or MATLAB, I tried my best to carefully capture a chessboard to cover all image regions evenly.

After obtaining the intrinsic parameters (principal point, focal length, distortion coefficients), I was ready to calibrate the extrinsic.

Camera-LiDAR Extrinsic

The camera-LiDAR calibration involved the following steps:

1. Simultaneously capture the target board in both the camera image and LiDAR point cloud. It’s crucial that the camera image is undistorted and both sensors are time-synchronized.
2. Find the corners of the target board (correspondences) in both the image and LiDAR data. While finding corners in the image is relatively straightforward, doing the same in sparse LiDAR points can be challenging.
3. Calculate rotation and translation by minimizing errors between image corners and LiDAR corners projected into the image plane. In other words, given a set of correspondences and intrinsic K, we find R, t that have a minimum pixel error [u; v; 1] = K [R|t][x; y; z; 1].

Because of the difficulty in step 2, particularly in finding corners in the LiDAR points, led to a series of additional processes:

Plane fitting and projection → Find four edges with constraints (fixed edge length, perpendicularity) → Find corners and manually correct

Sunmyung helped me a lot in this process!

Time Synchronization

After successfully obtaining all extrinsics, I found that projections worked well in static scenarios, but they were less reliable during motion. The issue was traced back to time synchronization discrepancies.

It is natural to have timing errors since external triggers were not used. However, given that the camera operated at 30FPS and the LiDAR at 20FPS, theoretically, the closest available image should be within 33ms of the LiDAR even without precise synchronization. The problem was that I ignored the errors introduced during processing and saving LiDAR and image data.

To address this, I reduced the saving frequency of the image, turned off visualization during logging (although it was 1 Hz), and employed a real-time Ubuntu kernel with the help of Hyeonseo, resulting in improvements.

While this was not the best solution, I hoped that the network could still function reasonably well with some time sync error. In the future, using cameras that support triggers can be a more elegant solution.