Object Tracking In 3D in AI & Artificial Intelligence

Tracking is essentially a State Estimation problem. We want to know the object's position and velocity at any given time. However, sensors are noisy. The Kalman Filter solves this by maintaining a 'Belief' about the object's state and updating it with every new measurement. It works in two steps: Predict (where should it be?) and Update (where did the sensor see it?). This recursive process allows for incredibly smooth and accurate tracking even when the sensor data is intermittent.

When tracking multiple objects, the hardest challenge is Data Association: which new detection belongs to which existing track? Modern systems like DeepSORT use both geometric cues (where is the box?) and appearance cues (what does the object look like?) to make this decision. By creating a 'Feature Embedding' of the object's appearance, the system can re-identify a person even after they have been completely occluded for several seconds, which is critical for robots operating in crowded public spaces.

Detecting an object is easy; following it in a dynamic 3D world is hard. In this lesson, we'll master Object Tracking—the technology that allows robots to interact with moving environments.

Object tracking involves identifying an object in one frame and maintaining its 'Identity' across subsequent frames, even if it moves, rotates, or is partially blocked.

In 3D, we don't just track pixels; we track 'Bounding Boxes' in 3D space using depth information from LiDAR or Stereo cameras. We often use 'SORT' or 'DeepSORT' algorithms.

Checkpoint: Why is simple pixel-matching insufficient for robust object tracking in robotics?

• →It's too colorful
• →Objects can change appearance due to lighting, rotation, or being partially blocked (occlusion), requiring more advanced state estimation

We use Kalman Filters to predict where an object *should* be in the next frame. This helps handle temporary occlusions (like a car passing behind a tree).

Data association is the hardest part: deciding which detection in the current frame belongs to which tracked object from the previous frame. We use the 'Intersection over Union' (IoU) metric.

Checkpoint: What does the 'Kalman Filter' contribute to the tracking process?

• →It detects the object
• →It predicts the object's next position based on its current velocity and direction, maintaining the track during noise or occlusion

By mastering 3D tracking, you enable robots to safely navigate around pedestrians, follow moving targets, and anticipate the actions of other agents.

Pro-tip: Use 'Multi-Object Tracking' (MOT) benchmarks to evaluate your system's performance on metrics like MOTA (Multi-Object Tracking Accuracy).

Checkpoint: True or False: 3D Tracking requires both a camera and a depth-sensing device (like LiDAR or a second camera) to function accurately in space.

• →True
• →False

Tracking engine calibrated! Your robot now has eyes on the prize.

Next, we'll dive into SLAM (Simultaneous Localization and Mapping), where the robot learns to build a map while figuring out where it is.

Compute Real 3D Tracking Distance. Finish computing the 3D Euclidean distance between two tracked object positions.