Here is what Tesla AI can see (Video)

A rare look at what Tesla Autopilot can see and interpret

With the help of a few Tesla hackers, they were able to overlay the data with the snapshots of what the vehicle can see, which results in an interesting visualization of what Autopilot understands when looking through its sensors. It’s not a complete look at what Autopilot can see. For example, it doesn’t include the tracking of the lanes. But it does show how the system is tracking objects in its field of view and a way that we have never seen before. It looks like this:

Now a bit of an explanation about what the circles mean: circle color represents the type of the object:

  • green – moving
  • yellow – stopped
  • orange – stationary
  • red – unknown.

The circle size represents the distance to the object, the bigger the circle, the closer is the object (so we are not trying to encircle object, or approximate the size – the radar has no way of knowing this). A thick circle means this object has a label and is therefore likely being tracked by the autopilot. Fading in and out is due to the probability of existence changing.

Sometimes the circle is in some empty space – this is likely due to radar having some problems with determining object elevation, try to look higher or lower for a relevant object. Also at times, radar cannot determine elevation at all (value of -5 is used in that case). The circle is drawn at elevation 0 in this case.

Also keep in mind that the radar reports coordinates in 3D and we need to project them onto a 2D picture. Sometimes there are errors in such conversions as you might expect.