Deploying: ROS Porting

I utilized ROS for sensor integration, simplifying the ROS topic subscribing process to my existing Python code using rospy. To enable real-time processing, I had to rewrite code for pre-processing the data received from sensors via ROS topics with the help of Wonjoon.

First, I accumulated multiple radar sweeps from past frames into a current frame by compensating the motion of the ego vehicle, thereby making sparse radar points denser. Now, when receiving radar data in real-time, I modified the code to accumulate multiple radar sweeps with the help of accurate ego position from DGPS. (It might be promising to use vehicle speed and steering information from CAN to calculate ego-motion to remove DGPS.)

Second, since the network has to operate with a cycle time of 0.1 seconds but a network requires a BEV feature map from 1 second ago, I needed to figure out how to deploy it. (When running CRN on an offline setting, I could achieve up to 20 FPS; however, I chose to operate it at 10 FPS considering potential I/O bottlenecks and other computations.)

The most ideal approach can be to generate a BEV feature map using a single frame and store it in a queue for later use. Since this BEV feature map is obtained at intervals of 0.1 seconds, data stored at the previous 10 indexes in the queue will allow us to use the BEV feature map from 1 second ago. However, we would have to correctly warp the BEV feature map into the current frame.

Instead, as a proof of concept, I saved images and radar points (instead of the BEV feature map) in queues, then processed two keyframes batch-wise. In this case, feature extractors have to process the same data twice, causing inefficiency. Nevertheless, in an ideal scenario, batch processing would maintain similar throughput, albeit with larger memory consumption. Given the sufficient memory resources of the industrial PC installed in our vehicle platform, I focused primarily on throughput and opted for this approach as a prototype, although it was somewhat cumbersome.

Real-time Implementation

The model utilizes a 384x1056 size image and radar points as input with a ResNet50 backbone, and detects objects within a 100m range, achieving a frame rate of 18 FPS using an i7 and RTX 3090. I expect that converting the model to TensorRT (referring to this repository) could further improve inference speed.

This video showcases a real-time demonstration of CRN, using camera and radar data replayed from a rosbag file. Although the performance may not be the best, it effectively demonstrates that the system is functional. Note that the data in this video is from entirely new data and was not included in the training dataset

Moreover, I also tested CRN in the rainy scene, which was never included in our dataset. The model had not seen rain before but succeeded in detecting objects!

This confirms that the trained network has generalized to previously unseen data, leaving only the task of making it operational in an actual vehicle.