2nd/10_Wiki/Topics/AI_and_ML/CV_Synthesis.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-cv-synthesis	Computer Vision Synthesis (Synthetic Data)	10_Wiki/Topics	verified	self	synthetic data sim2real domain adaptation NVIDIA Omniverse Unity Perception model collapse	none	A	0.9	applied	computer-vision synthetic-data sim2real domain-adaptation omniverse unity autonomous-driving robotics		2026-05-10	pending	language framework Python NVIDIA Omniverse / Unity Perception / Blender / Diffusers

CV Synthesis (Synthetic Data)

📌 한 줄 통찰

"매 가상 의 image 의 학습 data". 매 perfect ground truth + 매 rare event + 매 privacy. 매 sim-to-real domain gap 의 핵심 challenge. 매 modern: 매 generative AI (Stable Diffusion) + 매 photoreal sim (Omniverse, Gaussian Splatting). 매 model collapse 의 risk.

📖 핵심

매 motivation

1. Perfect ground truth: 매 pixel mask, 매 3D position, 매 depth, 매 segmentation 의 free.
2. Rare event: 매 accident, 매 edge case 의 endless.
3. Privacy: 매 face / plate 의 X.
4. Cost: 매 real annotation $$$ 의 X.
5. Coverage: 매 lighting / weather / pose 의 systematic.

매 source

• Game engine (Unity, Unreal): 매 photoreal.
• Synthetic 3D pipeline (Blender, Houdini).
• NVIDIA Omniverse: 매 industrial digital twin.
• Diffusion model: 매 prompt-based.
• GAN: 매 specific domain (face, etc).
• Procedural generation: 매 controllable.

매 응용

1. Autonomous driving: 매 CARLA, 매 Waymo Carcraft.
2. Robotics: 매 Isaac Sim, 매 sim2real.
3. Surveillance: 매 person re-id.
4. Medical: 매 augment rare condition.
5. Aerial: 매 drone training.
6. Manufacturing: 매 defect detection.
7. Retail: 매 product variation.

Sim-to-Real domain gap

• Visual gap: 매 lighting / texture / shadow.
• Distribution gap: 매 prevalence.
• Causal gap: 매 dynamics / physics.

매 mitigation

• Domain randomization: 매 random texture / lighting.
• Domain adaptation: 매 GAN / CycleGAN.
• Photorealism push: 매 ray tracing, Gaussian splatting.
• Hybrid training: 매 real + synthetic.
• Self-supervised: 매 unlabeled real.

매 Model Collapse (modern concern)

• 매 synthetic data 만 의 train → 매 real distribution drift.
• 매 generation 의 amplify own bias.
• 매 mitigation: 매 fresh real 의 mix.
• → Shumailov et al. 2024.

매 platform

• NVIDIA Omniverse / Replicator: 매 enterprise.
• Unity Perception SDK: 매 game-engine.
• Unreal MetaHuman: 매 photoreal humans.
• Mitsuba 3: 매 differentiable rendering.
• Kubric (Google): 매 video synthesis.
• Habitat / iGibson: 매 robot indoor.
• CARLA / AirSim: 매 driving / drone.

매 generative augmentation

• Stable Diffusion: 매 prompt-driven.
• ControlNet: 매 layout-controlled.
• DreamBooth + LoRA: 매 specific class.
• Inpainting: 매 selective augment.

💻 패턴

Unity Perception (label config)

// 매 Unity Perception SDK
using UnityEngine.Perception.GroundTruth;

[CreateAssetMenu]
public class ProductLabelConfig : IdLabelConfig {}

// 매 Perception camera 의 attach
// 매 RGB + bounding box + segmentation + depth 의 auto.

Domain randomization (Python)

import random
import bpy  # 매 Blender API

def randomize_scene():
    # 매 light
    light = bpy.data.objects['Light']
    light.data.energy = random.uniform(500, 2000)
    light.location = (random.uniform(-5, 5), random.uniform(-5, 5), random.uniform(3, 10))
    
    # 매 camera angle
    cam = bpy.data.objects['Camera']
    cam.rotation_euler = (random.uniform(0, 0.5), random.uniform(0, 0.5), random.uniform(0, 6.28))
    
    # 매 background HDR
    world = bpy.data.worlds['World']
    hdri = random.choice(['hdri/sunset.exr', 'hdri/cloudy.exr', 'hdri/night.exr'])
    world.node_tree.nodes['Environment Texture'].image = bpy.data.images.load(hdri)
    
    # 매 material color
    for obj in bpy.context.scene.objects:
        if obj.data and obj.data.materials:
            obj.data.materials[0].diffuse_color = (
                random.random(), random.random(), random.random(), 1
            )

# 매 1000 frame 의 render
for i in range(1000):
    randomize_scene()
    bpy.context.scene.render.filepath = f'./synthetic/img_{i:04d}.png'
    bpy.ops.render.render(write_still=True)

NVIDIA Omniverse Replicator (Python)

import omni.replicator.core as rep

with rep.new_layer():
    camera = rep.create.camera(position=(0, 0, 1000))
    light = rep.create.light(rotation=(-90, 0, 0), light_type='distant')
    
    # 매 distractor objects
    distractors = rep.create.cube(count=20)
    
    # 매 target object
    target = rep.create.sphere()
    
    with rep.trigger.on_frame(num_frames=1000):
        with target:
            rep.modify.pose(
                position=rep.distribution.uniform((-200, -200, 0), (200, 200, 200)),
                rotation=rep.distribution.uniform((0, 0, 0), (360, 360, 360)),
            )
        with distractors:
            rep.randomizer.scatter_2d()
        with light:
            rep.modify.attribute('intensity', rep.distribution.uniform(500, 5000))
    
    # 매 writer (RGB + bbox + mask)
    writer = rep.WriterRegistry.get('BasicWriter')
    writer.initialize(output_dir='./synthetic/', rgb=True, bbox=True, mask=True)
    writer.attach([rep.create.render_product(camera, (1024, 1024))])

CARLA driving sim

import carla

client = carla.Client('localhost', 2000)
world = client.get_world()

# 매 spawn vehicle + camera
blueprint_library = world.get_blueprint_library()
vehicle_bp = blueprint_library.filter('vehicle.tesla.model3')[0]
spawn_point = world.get_map().get_spawn_points()[0]
vehicle = world.spawn_actor(vehicle_bp, spawn_point)

camera_bp = blueprint_library.find('sensor.camera.rgb')
camera_bp.set_attribute('image_size_x', '800')
camera = world.spawn_actor(camera_bp, carla.Transform(carla.Location(x=2.5, z=1.0)), attach_to=vehicle)

# 매 listen
camera.listen(lambda image: image.save_to_disk(f'./out/{image.frame:08d}.png'))

# 매 weather randomization
weather = carla.WeatherParameters(cloudiness=80, precipitation=30, sun_altitude_angle=45)
world.set_weather(weather)

Diffusion-based augmentation

from diffusers import StableDiffusionInpaintPipeline
import torch

pipe = StableDiffusionInpaintPipeline.from_pretrained(
    'runwayml/stable-diffusion-inpainting', torch_dtype=torch.float16,
).to('cuda')

# 매 existing image + 매 mask → 매 inpaint with new variation
def augment_with_inpaint(image, mask, prompts):
    augmented = []
    for p in prompts:
        result = pipe(
            prompt=p,
            image=image,
            mask_image=mask,
            num_inference_steps=30,
            guidance_scale=7.5,
        ).images[0]
        augmented.append(result)
    return augmented

# 매 e.g., medical scan 의 condition variation
prompts = ['a benign tumor', 'a malignant tumor stage 1', '...']

Sim2Real domain adaptation (CycleGAN)

# 매 sim → real domain
from torch_cyclegan import CycleGAN

cyclegan = CycleGAN(
    generator_S2R=Generator(),
    generator_R2S=Generator(),
    discriminator_R=Discriminator(),
    discriminator_S=Discriminator(),
)

# 매 sim image 의 real-style transfer
real_styled = cyclegan.S2R(sim_image)
# 매 train downstream model on (sim_label, real_styled).

Mix ratio (model collapse mitigation)

def adaptive_mix(epoch, real_data, synthetic_data):
    """매 early: synthetic 의 lots, late: real 의 emphasize."""
    real_ratio = min(0.7, 0.2 + epoch * 0.05)
    n_real = int(BATCH_SIZE * real_ratio)
    n_synth = BATCH_SIZE - n_real
    
    return DataLoader(
        ConcatDataset([
            Subset(real_data, random.sample(range(len(real_data)), n_real)),
            Subset(synthetic_data, random.sample(range(len(synthetic_data)), n_synth)),
        ]),
        batch_size=BATCH_SIZE,
    )

🤔 결정 기준

상황	Tool
Driving	CARLA / Waymo Carcraft
Robot indoor	Habitat / iGibson
Industrial	NVIDIA Omniverse
Bbox / segmentation	Unity Perception
Photoreal humans	MetaHuman + Unreal
Augmentation	Stable Diffusion + ControlNet
Domain gap	CycleGAN / domain randomization
Tabletop	Blender + scripted

기본값: 매 photoreal + 매 domain randomization + 매 mix with real.

🔗 Graph

• 부모: Computer Vision · Synthetic-Data · Simulation
• 변형: Sim2Real · Domain-Adaptation · CycleGAN
• 응용: Autonomous Vehicles · Robotics · Unity-Perception
• Adjacent: Diffusion-Models · ControlNet · Model-Collapse · Algorithmic-Biology

🤖 LLM 활용

언제: 매 data scarcity. 매 rare event. 매 privacy-sensitive. 매 cost reduction. 매 systematic coverage. 언제 X: 매 real data abundant + cheap. 매 high domain-gap not addressed.

❌ 안티패턴

• Synthetic 만 의 train: 매 model collapse + sim2real gap.
• Single environment: 매 over-fit.
• No domain randomization: 매 unrealistic 학습.
• Generative 의 cycle (synth → train → gen → train): 매 collapse.
• No real validation: 매 fake metric.

🧪 검증 / 중복

• Verified (Tobin domain randomization 2017, Tremblay 2018, NVIDIA Omniverse).
• 신뢰도 A.
• Related: Autonomous Vehicles · Diffusion-Models · Robotics · Algorithmic-Biology · Model-Collapse.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — sim2real + domain randomization + 매 Unity / Omniverse / CARLA / SD / CycleGAN code