---
id: wiki-2026-0508-cv-synthesis
title: Computer Vision Synthesis (Synthetic Data)
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [synthetic data, sim2real, domain adaptation, NVIDIA Omniverse, Unity Perception, model collapse]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [computer-vision, synthetic-data, sim2real, domain-adaptation, omniverse, unity, autonomous-driving, robotics]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: NVIDIA Omniverse / Unity Perception / Blender / Diffusers
---

# CV Synthesis (Synthetic Data)

## 📌 한 줄 통찰
> **"매 가상 의 image 의 학습 data"**. 매 perfect ground truth + 매 rare event + 매 privacy. 매 sim-to-real domain gap 의 핵심 challenge. 매 modern: 매 generative AI (Stable Diffusion) + 매 photoreal sim (Omniverse, Gaussian Splatting). 매 model collapse 의 risk.

## 📖 핵심

### 매 motivation
1. **Perfect ground truth**: 매 pixel mask, 매 3D position, 매 depth, 매 segmentation 의 free.
2. **Rare event**: 매 accident, 매 edge case 의 endless.
3. **Privacy**: 매 face / plate 의 X.
4. **Cost**: 매 real annotation $$$ 의 X.
5. **Coverage**: 매 lighting / weather / pose 의 systematic.

### 매 source
- **Game engine** (Unity, Unreal): 매 photoreal.
- **Synthetic 3D pipeline** (Blender, Houdini).
- **NVIDIA Omniverse**: 매 industrial digital twin.
- **Diffusion model**: 매 prompt-based.
- **GAN**: 매 specific domain (face, etc).
- **Procedural generation**: 매 controllable.

### 매 응용
1. **Autonomous driving**: 매 CARLA, 매 Waymo Carcraft.
2. **Robotics**: 매 Isaac Sim, 매 sim2real.
3. **Surveillance**: 매 person re-id.
4. **Medical**: 매 augment rare condition.
5. **Aerial**: 매 drone training.
6. **Manufacturing**: 매 defect detection.
7. **Retail**: 매 product variation.

### Sim-to-Real domain gap
- **Visual gap**: 매 lighting / texture / shadow.
- **Distribution gap**: 매 prevalence.
- **Causal gap**: 매 dynamics / physics.

### 매 mitigation
- **Domain randomization**: 매 random texture / lighting.
- **Domain adaptation**: 매 GAN / CycleGAN.
- **Photorealism push**: 매 ray tracing, Gaussian splatting.
- **Hybrid training**: 매 real + synthetic.
- **Self-supervised**: 매 unlabeled real.

### 매 Model Collapse (modern concern)
- 매 synthetic data 만 의 train → 매 real distribution drift.
- 매 generation 의 amplify own bias.
- 매 mitigation: 매 fresh real 의 mix.
- → Shumailov et al. 2024.

### 매 platform
- **NVIDIA Omniverse / Replicator**: 매 enterprise.
- **Unity Perception SDK**: 매 game-engine.
- **Unreal MetaHuman**: 매 photoreal humans.
- **Mitsuba 3**: 매 differentiable rendering.
- **Kubric** (Google): 매 video synthesis.
- **Habitat / iGibson**: 매 robot indoor.
- **CARLA / AirSim**: 매 driving / drone.

### 매 generative augmentation
- **Stable Diffusion**: 매 prompt-driven.
- **ControlNet**: 매 layout-controlled.
- **DreamBooth + LoRA**: 매 specific class.
- **Inpainting**: 매 selective augment.

## 💻 패턴

### Unity Perception (label config)
```csharp
// 매 Unity Perception SDK
using UnityEngine.Perception.GroundTruth;

[CreateAssetMenu]
public class ProductLabelConfig : IdLabelConfig {}

// 매 Perception camera 의 attach
// 매 RGB + bounding box + segmentation + depth 의 auto.
```

### Domain randomization (Python)
```python
import random
import bpy  # 매 Blender API

def randomize_scene():
    # 매 light
    light = bpy.data.objects['Light']
    light.data.energy = random.uniform(500, 2000)
    light.location = (random.uniform(-5, 5), random.uniform(-5, 5), random.uniform(3, 10))
    
    # 매 camera angle
    cam = bpy.data.objects['Camera']
    cam.rotation_euler = (random.uniform(0, 0.5), random.uniform(0, 0.5), random.uniform(0, 6.28))
    
    # 매 background HDR
    world = bpy.data.worlds['World']
    hdri = random.choice(['hdri/sunset.exr', 'hdri/cloudy.exr', 'hdri/night.exr'])
    world.node_tree.nodes['Environment Texture'].image = bpy.data.images.load(hdri)
    
    # 매 material color
    for obj in bpy.context.scene.objects:
        if obj.data and obj.data.materials:
            obj.data.materials[0].diffuse_color = (
                random.random(), random.random(), random.random(), 1
            )

# 매 1000 frame 의 render
for i in range(1000):
    randomize_scene()
    bpy.context.scene.render.filepath = f'./synthetic/img_{i:04d}.png'
    bpy.ops.render.render(write_still=True)
```

### NVIDIA Omniverse Replicator (Python)
```python
import omni.replicator.core as rep

with rep.new_layer():
    camera = rep.create.camera(position=(0, 0, 1000))
    light = rep.create.light(rotation=(-90, 0, 0), light_type='distant')
    
    # 매 distractor objects
    distractors = rep.create.cube(count=20)
    
    # 매 target object
    target = rep.create.sphere()
    
    with rep.trigger.on_frame(num_frames=1000):
        with target:
            rep.modify.pose(
                position=rep.distribution.uniform((-200, -200, 0), (200, 200, 200)),
                rotation=rep.distribution.uniform((0, 0, 0), (360, 360, 360)),
            )
        with distractors:
            rep.randomizer.scatter_2d()
        with light:
            rep.modify.attribute('intensity', rep.distribution.uniform(500, 5000))
    
    # 매 writer (RGB + bbox + mask)
    writer = rep.WriterRegistry.get('BasicWriter')
    writer.initialize(output_dir='./synthetic/', rgb=True, bbox=True, mask=True)
    writer.attach([rep.create.render_product(camera, (1024, 1024))])
```

### CARLA driving sim
```python
import carla

client = carla.Client('localhost', 2000)
world = client.get_world()

# 매 spawn vehicle + camera
blueprint_library = world.get_blueprint_library()
vehicle_bp = blueprint_library.filter('vehicle.tesla.model3')[0]
spawn_point = world.get_map().get_spawn_points()[0]
vehicle = world.spawn_actor(vehicle_bp, spawn_point)

camera_bp = blueprint_library.find('sensor.camera.rgb')
camera_bp.set_attribute('image_size_x', '800')
camera = world.spawn_actor(camera_bp, carla.Transform(carla.Location(x=2.5, z=1.0)), attach_to=vehicle)

# 매 listen
camera.listen(lambda image: image.save_to_disk(f'./out/{image.frame:08d}.png'))

# 매 weather randomization
weather = carla.WeatherParameters(cloudiness=80, precipitation=30, sun_altitude_angle=45)
world.set_weather(weather)
```

### Diffusion-based augmentation
```python
from diffusers import StableDiffusionInpaintPipeline
import torch

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    'runwayml/stable-diffusion-inpainting', torch_dtype=torch.float16,
).to('cuda')

# 매 existing image + 매 mask → 매 inpaint with new variation
def augment_with_inpaint(image, mask, prompts):
    augmented = []
    for p in prompts:
        result = pipe(
            prompt=p,
            image=image,
            mask_image=mask,
            num_inference_steps=30,
            guidance_scale=7.5,
        ).images[0]
        augmented.append(result)
    return augmented

# 매 e.g., medical scan 의 condition variation
prompts = ['a benign tumor', 'a malignant tumor stage 1', '...']
```

### Sim2Real domain adaptation (CycleGAN)
```python
# 매 sim → real domain
from torch_cyclegan import CycleGAN

cyclegan = CycleGAN(
    generator_S2R=Generator(),
    generator_R2S=Generator(),
    discriminator_R=Discriminator(),
    discriminator_S=Discriminator(),
)

# 매 sim image 의 real-style transfer
real_styled = cyclegan.S2R(sim_image)
# 매 train downstream model on (sim_label, real_styled).
```

### Mix ratio (model collapse mitigation)
```python
def adaptive_mix(epoch, real_data, synthetic_data):
    """매 early: synthetic 의 lots, late: real 의 emphasize."""
    real_ratio = min(0.7, 0.2 + epoch * 0.05)
    n_real = int(BATCH_SIZE * real_ratio)
    n_synth = BATCH_SIZE - n_real
    
    return DataLoader(
        ConcatDataset([
            Subset(real_data, random.sample(range(len(real_data)), n_real)),
            Subset(synthetic_data, random.sample(range(len(synthetic_data)), n_synth)),
        ]),
        batch_size=BATCH_SIZE,
    )
```

## 🤔 결정 기준
| 상황 | Tool |
|---|---|
| Driving | CARLA / Waymo Carcraft |
| Robot indoor | Habitat / iGibson |
| Industrial | NVIDIA Omniverse |
| Bbox / segmentation | Unity Perception |
| Photoreal humans | MetaHuman + Unreal |
| Augmentation | Stable Diffusion + ControlNet |
| Domain gap | CycleGAN / domain randomization |
| Tabletop | Blender + scripted |

**기본값**: 매 photoreal + 매 domain randomization + 매 mix with real.

## 🔗 Graph
- 부모: [[Computer Vision|Computer-Vision]] · [[Synthetic-Data]] · [[Simulation]]
- 변형: [[Sim2Real]] · [[Domain-Adaptation]] · [[CycleGAN]]
- 응용: [[Autonomous Vehicles]] · [[Robotics]] · [[Unity-Perception]]
- Adjacent: [[Diffusion-Models]] · [[ControlNet]] · [[Model-Collapse]] · [[Algorithmic-Biology]]

## 🤖 LLM 활용
**언제**: 매 data scarcity. 매 rare event. 매 privacy-sensitive. 매 cost reduction. 매 systematic coverage.
**언제 X**: 매 real data abundant + cheap. 매 high domain-gap not addressed.

## ❌ 안티패턴
- **Synthetic 만 의 train**: 매 model collapse + sim2real gap.
- **Single environment**: 매 over-fit.
- **No domain randomization**: 매 unrealistic 학습.
- **Generative 의 cycle (synth → train → gen → train)**: 매 collapse.
- **No real validation**: 매 fake metric.

## 🧪 검증 / 중복
- Verified (Tobin domain randomization 2017, Tremblay 2018, NVIDIA Omniverse).
- 신뢰도 A.
- Related: [[Autonomous Vehicles]] · [[Diffusion-Models]] · [[Robotics]] · [[Algorithmic-Biology]] · [[Model-Collapse]].

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — sim2real + domain randomization + 매 Unity / Omniverse / CARLA / SD / CycleGAN code |