Synthetic Data & Domain Adaptation¶

Why real data is not enough¶

Physical AI systems need training data that captures the diversity of the real world — different objects, lighting conditions, camera angles, failure modes, and edge cases. In traditional software AI, you collect data from production systems. In physical AI, data collection is fundamentally harder:

Rare events are the ones that matter most. A warehouse robot handles 10,000 packages per day. It encounters a crushed box once a week. It encounters two people in the same aisle once a month. Training a model to handle these rare scenarios requires either enormous amounts of real-world collection time or the ability to generate them on demand.

Labeling is prohibitively expensive. Pixel-level semantic segmentation of a single image takes 30–90 minutes of human annotation. 3D bounding boxes and 6-DOF pose annotations are even more expensive. In simulation, labels are free — the renderer knows exactly which pixel belongs to which object, the exact 3D pose of every entity, and the depth at every point.

Environment diversity requires physical effort. A real robot cell has one set of lighting, one background, one camera. To train a model that generalizes across warehouses, you would need to physically set up and collect data in hundreds of environments. In simulation, you generate unbounded environmental diversity procedurally.

Safety and ethics constrain collection. Generating training data for scenarios involving human-robot proximity, near-miss collisions, or failure modes is dangerous in the real world. Simulation allows unlimited generation of these scenarios safely.

Privacy is a concern. In warehouses, hospitals, or retail environments, camera data may contain personally identifiable information. Synthetic data sidesteps privacy constraints entirely.

Types of synthetic data¶

Rendered images and video¶

Photorealistic or stylized 2D frames generated by a rendering engine. These come with automatic ground-truth labels:

RGB images — the visual appearance, as a camera would see it
Depth maps — per-pixel distance from the camera
Semantic segmentation masks — per-pixel class labels (floor, wall, pallet, robot, person)
Instance segmentation — per-pixel object identity (pallet #1 vs. pallet #2)
Surface normals — per-pixel orientation of the surface
Optical flow — per-pixel motion between frames
2D and 3D bounding boxes — object detection targets
6-DOF poses — full position and orientation of every object

All of these are generated simultaneously, at zero marginal cost per label modality. This is the fundamental advantage of synthetic data — labeling is a byproduct of rendering, not a separate expensive process.

NVIDIA's Omniverse Replicator is the primary tool for generating this type of synthetic data at scale from Isaac Sim scenes.

Simulated sensor data¶

Beyond cameras, simulation can generate synthetic:

LiDAR point clouds — with accurate beam patterns, range noise, and multi-echo behavior
Radar returns — for automotive and industrial sensing
IMU readings — accelerometer and gyroscope data with realistic noise models
Force/torque sensor data — contact forces during manipulation
Tactile sensor data — pressure distributions during grasping

Each modality requires a sensor model within the simulator that approximates the noise characteristics and physics of the real sensor.

Procedurally generated scenarios¶

Rather than hand-designing each training scene, procedural generation uses rules and randomization to create scenes automatically. For example:

"Place 5–20 random objects from a library on a table surface, with random poses, under random lighting, with random background textures."

This is the approach used by Omniverse Replicator's Randomizer API and Isaac Sim's scene generation capabilities. Procedural generation scales the data pipeline from "an artist designs each scene" to "a script generates thousands of scene variants per hour."

Domain randomization¶

Domain randomization is both a sim-to-real transfer technique and a synthetic data generation strategy. The core idea: if you randomize enough parameters during training, the real world becomes "just another variation" that falls within the training distribution.

What gets randomized¶

Visual parameters:

Lighting: direction, intensity, color, number of light sources
Object textures: replaced with random patterns, colors, or images from a texture library
Background: random textures, distractors, environment maps
Camera: focal length, exposure, white balance, lens distortion, pose

Physical parameters:

Object masses and inertia
Friction coefficients (static, dynamic)
Damping values
Actuator gains and noise
Joint stiffness and play

Geometric parameters:

Object sizes (within realistic bounds)
Object placement positions and orientations
Obstacle positions
Table heights, shelf positions

Why it works¶

The intuition is similar to data augmentation in image classification (random crops, flips, color jitter) but applied at the environment level rather than the image level. By training on a wide distribution of visual and physical conditions, the model learns features that are invariant to the specific conditions — it learns the underlying task structure rather than memorizing artifacts of one particular setup.

The landmark demonstration was OpenAI's Dactyl project (2019), which trained a Shadow Hand robot to manipulate a Rubik's cube entirely in simulation with massive domain randomization, then transferred the policy directly to physical hardware with no fine-tuning.

Tobin, J., et al. (2017). "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World." arXiv:1703.06907
Prakash, A., et al. (2019). "Structured Domain Randomization: Bridging the Reality Gap by Context-Aware Synthetic Data." arXiv:1810.10093

Domain adaptation¶

Where domain randomization is a brute-force approach (make training data so diverse that reality is just one sample), domain adaptation is a learned approach: explicitly transform synthetic data to look realistic, or learn features that are invariant to the synthetic/real distinction.

Neural image-to-image translation¶

Use a generative model to translate rendered images into photorealistic images while preserving geometric structure and labels. The labels from the renderer carry over because the spatial content is preserved — only the visual appearance changes.

CycleGAN and similar unpaired image translation methods learn a mapping between the synthetic and real domains without requiring paired examples (you do not need the exact same scene rendered synthetically and photographed).

Zhu, J.-Y., et al. (2017). "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks." arXiv:1703.10593

Diffusion-based domain adaptation is the more recent approach. Models like Cosmos Transfer take a synthetic video plus control signals (depth maps, edge maps, segmentation masks) and generate a photorealistic version that preserves the spatial structure of the input. This is more powerful than GAN-based methods because diffusion models produce higher-quality, more diverse outputs.

Feature-level adaptation¶

Rather than adapting pixels, learn a feature extractor whose representations are indistinguishable between synthetic and real domains. The model trains on synthetic data with an adversarial loss that penalizes any feature difference between synthetic and real inputs. The resulting features are domain-invariant — the model cannot tell whether its input is synthetic or real.

Ganin, Y. & Lempitsky, V. (2015). "Unsupervised Domain Adaptation by Backpropagation." arXiv:1409.7495

The synthetic data pipeline¶

For a physical AI system, the full synthetic data pipeline looks like:

1. Scene construction
   └─ Build digital twin in Isaac Sim (USD scene, assets, physics)

2. Scenario generation
   └─ Procedurally randomize: objects, lighting, textures, physics

3. Simulation execution
   └─ Run the robot (scripted, teleoperation, or RL policy)
   └─ Record: camera frames, depth, segmentation, joint states, actions

4. Control signal extraction
   └─ Extract depth maps, edge maps, segmentation masks from recordings

5. Domain adaptation (optional)
   └─ Cosmos Transfer: synthetic video + control signals → photorealistic video

6. Dataset assembly
   └─ Package frames, labels, actions into training format (HDF5, WebDataset)

7. Model training
   └─ Train VLA, perception model, or policy on synthetic + real data

8. Evaluation
   └─ Test on held-out real-world data
   └─ Measure sim-to-real transfer performance
   └─ Iterate on simulation fidelity or randomization strategy

Steps 1–4 happen in simulation. Step 5 bridges the visual domain gap. Steps 6–8 are the standard ML workflow. The key insight is that the expensive, dangerous, slow part (data collection) is replaced by simulation, and the visual fidelity gap is closed by neural domain adaptation.

Quality metrics for synthetic data¶

Measuring synthetic data quality is an open research problem. The metrics that matter most:

Downstream task performance¶

The ultimate metric: does training on this synthetic data improve performance on the real task? Measured by task success rate, mean Average Precision (mAP) for detection, Intersection over Union (IoU) for segmentation, evaluated on real-world test data.

Frechet Inception Distance (FID)¶

Measures the statistical distance between distributions of synthetic and real images in a learned feature space. Lower FID means the synthetic images are more statistically similar to real images. Widely used for evaluating generative models.

Heusel, M., et al. (2017). "GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium." arXiv:1706.08500

Diversity coverage¶

Do the synthetic data span the range of conditions the model will encounter in production? Measured by clustering synthetic and real data in feature space and checking whether synthetic clusters cover real data clusters.

Label accuracy¶

Even though labels are generated automatically in simulation, they can be incorrect if the renderer has bugs, occlusion handling is wrong, or physics simulation produces unrealistic configurations. Manual spot-checking of generated labels remains important.

Key takeaways¶

Real-world data collection for physical AI is too slow, expensive, dangerous, and privacy-constrained to support the scale and diversity needed for robust models.
Synthetic data from simulation provides unlimited, perfectly labeled training data across arbitrary scene variations.
Domain randomization makes models robust by training on wide parameter distributions so that reality is just another sample.
Domain adaptation (neural image translation, diffusion-based transfer) bridges the visual gap between synthetic and real data.
The full pipeline flows from scene construction through simulation, domain adaptation, and dataset assembly to model training.