Why Physical Data is Key to Building Robust AI and Robotics

I’m convinced that physical data holds the key to making AI truly robust and precise. While there’s no formal proof in the shape of an accepted theorem, the empirical evidence is compelling. As the generative AI field explodes, we have solid reasons to maintain healthy skepticism about claims involving real sensor data and embodied agents.

Recognizing Synthetic Worlds and Their Limits

Many current generative image and video models produce results that look breathtakingly coherent and logical at first glance. Look closer, though, and a different picture emerges. The geometry of these “synthetic worlds” often bears little resemblance to actual physics. Perspective lines don’t converge cleanly, viewpoints jump around, proportions shift. It’s as if someone fresh to perspective drawing created them.

For entertainment purposes, this suffices. But the moment you place a robot into such a world—whether for navigation, manipulation, or interaction—the lack of geometric consistency becomes a real problem. For a robot, what matters isn’t whether an image looks beautiful. What matters is whether coordinates, depths, and angles are correct. Synthetic worlds can deceive. Physical sensor data cannot.

Why Real Sensor Data Remains Unbeaten

Rolling Shutter, Synchronization, and Geometric Distortion

Rolling-shutter effects during camera motion: Many affordable cameras rely on rolling shutter technology. Rather than capturing an entire image instantaneously, the sensor reads it line by line. This means different parts of a single frame come from slightly different moments in time—a fact that causes serious distortion during motion. ¹

Real-world consequences for robotics: In robotic applications involving movement—whether the camera itself moves or objects in the scene do—rolling shutter becomes problematic. Objects and structures warp, lines appear skewed, movements look distorted. ²

Algorithmic compensation: Even modern visual-inertial odometry (VIO) systems, which fuse camera images with IMU data (accelerometers and gyroscopes), must explicitly model rolling-shutter distortion to produce usable results. The Ctrl-VIO framework is one example of this necessity. ³

Sensor Fusion and Multi-Modal Data as a Path to Robust Perception

Combining camera with inertial measurement: Many current approaches merge camera data with IMU measurements—capturing acceleration and rotation—to compensate for motion distortion and obtain more accurate pose and depth estimates. ⁴

Event cameras as an emerging sensing modality: There’s also a growing wave of interest in event cameras, an experimental but increasingly practical technology. Rather than capturing regular frames, event cameras detect changes in light intensity with extremely high temporal resolution and remarkably low latency. In fast-moving scenarios or low-light conditions, they show clear advantages over conventional cameras. ⁵

Multimodal sensor data as foundation: Such multimodal, physically grounded sensor data isn’t a luxury. It’s likely the necessary foundation for embodied AI systems—systems that don’t merely perceive but act, navigate, and manipulate their environment.

Real-World Environments Instead of Idealized Datasets

The dataset problem: Many established training datasets for images, videos, and 3D simulations capture visually polished scenarios that are often idealized or overly clean. Real-world applications, by contrast, demand handling of poor lighting, sparse texture, motion, sensor blur, and complex multi-modal sensing. Current datasets rarely include these conditions.

Edge cases as a core challenge: Particularly problematic are edge cases: rapid motion, changing lighting, unstructured surfaces, sparse depth information. Many algorithms stumble here despite good training data. Training on realistic sensor data, by contrast, forces models to contend with uncertainty, noise, and actual physical distortions.

Why Physical Data Is Essential for ML Progress

The limitations of today’s AI models rarely stem from insufficient compute. Almost always, they come down to the quality and structure of training data. Models trained on synthetic or visually-centered worlds achieve impressive perceptual performance. Yet they fail to develop a stable understanding of physical consistency in the real world. Without precise geometry, correct synchronization, and reliable sensors, they lack the bedrock needed to act reliably in practice.

Physical data does something that purely visual or synthetically generated data cannot: it carries unavoidable signatures of reality. The interplay of light and material, motion and distortion, friction and timing, sensor noise and instrument drift. This very “messiness” is what teaches models to learn robust invariants. Systems trained this way develop resilience to the actual conditions under which robots and autonomous agents must operate.

For precise 3D perception, grasping, navigation, or collaborative work, you don’t close the gap between 85% and 99% accuracy with more compute. You close it with data that captures the world’s true complexity. That includes rolling-shutter characteristics, dynamics in poor light, multi-view synchronization, camera drift, and sensor-induced noise. Today’s web, video, and diffusion datasets barely touch these aspects, yet they’re essential for robotics and embodied AI.

In short: machine learning progress increasingly hinges on how well models grasp and predict physical signals. If you want to build the next generation of autonomous systems, you need to embrace reality more directly. Not just the aesthetics of generated imagery, but the raw, unpolished data of the actual world. That’s what creates the foundation for models that don’t merely see—they act.