Building Dataset: Sensor Connection

The next step is to decide how to connect sensors to receive and save data.

While I could have developed my own code to interface with LiDAR and DGPS systems given (enough) time, I opted for a more convenient approach by adopting ROS (Robot Operating System).

Driver

Fortunately, Camera, LiDAR, and DGPS sensors already offered official ROS drivers. However, the radar sensor did not, so I needed to create a custom driver to establish a connection.

Camera, LiDAR, and DGPS provide their official ROS driver but radar does not, so I had to make a driver to connect the radar sensor. Using code from WATonomous team as a baseline, Hot Truong helped me a lot to enhance the code efficiency. Specifically, we rewrite the code to process the sniffed radar packet into a point array while ensuring a complete radar cycle, a code to publish the point array to ROS PointCloud2 message.

Visualization

With all the sensors connected, I was able to receive data seamlessly. One of the advantages of using ROS is its ease of visualization and recording. With minimal effort, I could record, replay, and visualize camera images, LiDAR, and radar points.

Greenish dots are radar points, color denotes its RCS value

Saving & Parsing

When it came to data saving, I aimed to record sequences of data that contained particularly interesting scenes. For instance, scenes containing rare classes of objects (e.g., construction vehicles, ambulances) or scenes where vehicles changed lanes frequently. To achieve this, I wrote code that would save topics into rosbag when receiving keyboard input, with each rosbag containing data from a 22-second duration. Because I always disliked nuScenes not having the previous data of the first keyframe, I saved the ‘prev’ data of the first keyframe and the ‘next’ data of the last keyframe.

Finally, 20 seconds of data (from 1 to 21 seconds of recorded file) is ready to be parsed into 40 frames with 2 Hz keyframe frequency.

To parse and save data samples, I followed the data structure of nuScenes but managed data at a scene-level. Unlike the nuScenes has a single JSON info file and uses single huge folder for the whole dataset, I separately saved info files and folders for each scene because the dataset had to be changed often. I simply modified dataloader of the model (which uses mmdet3D codebase) to load a list of JSON files.

Below is the data (and folder) structure that I used:

2023-06-23                            # logging date
├── 2023-06-23-16-32-16               # logging time of scenes 
│   ├── sample
│   |   ├── CAM_FRONT
│   |   |   ├── 1684657510676759.jpg  # time of data acquisition
│   |   |   ├── ...
│   |   ├── LIDAR_TOP
│   |   |   ├── 1684657510676737.bin
│   |   └── RADAR_FRONT               # different from ns, radar sweeps are accumulated with its index 
│   ├── sweeps
│   |   └── LIDAR_TOP
│   └── info.pkl                      # Contains relationship between data, intrinsic, extrinsic, position ...
├── 2023-06-23-16-34-27
└── ...