What are the key challenges when implementing real-time perception in mobile robots?

14 Eki 11:33

When I was integrating a SLAM stack on a delivery robot for indoor corridors, the biggest headache turned out to be balancing latency against robustness. I started with a pure‑vision pipeline (stereo + optical flow) because the camera gave us high‑resolution data for narrow passages, but even with a lightweight CNN the frame‑to‑frame processing hit ~80 ms on our Jetson‑Xavier, which was too slow for reacting to fast‑moving people. Adding an IMU and wheel odometry in a tight EKF fusion loop shaved the latency down to ~30 ms because the filter could predict the pose while the vision thread was still crunching the image, and it also helped the robot stay stable when the lighting changed or a reflective floor threw off the feature matcher. The trade‑off was extra CPU cycles for the fusion code, but the overall load stayed manageable since the IMU updates are cheap. For dynamic obstacles we ended up using a hybrid approach: a lightweight CNN runs at 15 Hz to detect pedestrians and estimate bounding boxes, while the fused odometry provides a fast motion estimate that we feed into a simple velocity‑obstacle planner. The planner can re‑plan within 10 ms, so even if the perception module lags a bit, the robot still avoids collisions. In practice, I found that keeping a “fast path” (raw IMU + odometry) for short‑term obstacle avoidance and a “slow path” (full vision fusion) for map updates gave the best responsiveness without overloading the processor. If you’re looking for deeper dives, check out the recent papers on “tight-coupled visual‑inertial odometry” and the ROS2‑based “Dynamic Obstacle Layer” in the navigation stack – they both address the latency/robustness balance pretty nicely.