How does Tesla's Autopilot sensor suite work?

26 Tem 15:56

Tesla Autopilot still uses three groups of sensors: multiple wide- and narrow-field-of-view cameras, ultrasonic sensors, and, until recently, 77 GHz radars. In the early Fusion module stage, raw camera images are converted into vector features, while radar measurements (range and relative speed) and ultrasonic data (close obstacles) are added as additional channels. All these tensors are fed into a single shared neural block—typically a convolutional and transformer network—where the data is "coordinated," and then passed on to the trajectory planner. It's important to note that in current Tesla models, radars have been almost completely phased out (Full Self-Driving 2023 v9), relying solely on a vision-based approach. This simplifies the hardware stack but increases demands on the training dataset and resilience to poor weather conditions. Compared to traditional stacks where LiDAR provides precise 3D point clouds, Tesla has "blind spots"—for example, in dim nighttime conditions or heavy rain, cameras may fail to detect small objects that LiDAR would easily pick up. Given this, I believe a hybrid approach (cameras + LiDAR + radar) still has advantages in complex scenarios (urban density, sudden object appearances). Tesla, of course, saves on hardware and aims for a "pure" software solution, but the question remains: how quickly will their neural networks be able to cover the same "angles of view" that LiDAR systems currently handle in competing projects? Do you think it’s worth keeping a small set of LiDAR sensors in future versions to reduce risk in extreme conditions?

26 Tem 18:24

I'm particularly interested in how the sensor-fusion system resolves conflicts—when the camera and radar give differing readings, what criteria does the neural network use to decide which input to trust? Could you also explain how the ultrasonic sensors are weighted in low-speed scenarios?