---
id: ai-vision-multimodal-production
title: Vision / Multimodal — production patterns
category: Coding
status: draft
source_trust_level: B
verification_status: conceptual
created_at: 2026-05-09
updated_at: 2026-05-09
tags: [ai, vision, multimodal, vibe-coding]
tech_stack: { language: "TS / Python", applicable_to: ["AI"] }
applied_in: []
aliases: [GPT-4V, Claude vision, multimodal, image embedding, OCR, document parsing, visual QA]
---

# Vision / Multimodal Production

> Image / PDF / video 가 LLM input. **Claude / GPT-4V vision, OCR, table extraction, document parsing**.

## 📖 핵심 개념
- Image 가 base64 또는 URL.
- Token cost 가 image 별 (e.g. 1000+).
- Resolution / detail trade-off.
- Specialized model (OCR, table) 가 generic 보다 좋음.

## 💻 코드 패턴

### Anthropic Vision
```ts
const r = await client.messages.create({
  model: 'claude-opus-4-7',
  messages: [{
    role: 'user',
    content: [
      { type: 'image', source: { type: 'base64', media_type: 'image/png', data: base64 } },
      { type: 'text', text: 'What is in this image?' },
    ],
  }],
});
```

### OpenAI Vision
```ts
const r = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{
    role: 'user',
    content: [
      { type: 'image_url', image_url: { url: 'data:image/png;base64,...' } },
      { type: 'text', text: 'Describe' },
    ],
  }],
});
```

### URL vs base64
```ts
// URL: server 가 download.
{ type: 'image_url', image_url: { url: 'https://...' } }

// Base64: 매 token cost.
{ type: 'image_url', image_url: { url: 'data:image/png;base64,...' } }
```

→ URL = bandwidth efficient.

### Detail level (OpenAI)
```ts
{
  type: 'image_url',
  image_url: { url: '...', detail: 'low' | 'high' | 'auto' },
}
```

→ low = 85 token / image.
high = 765+ token (resolution 따라).

### PDF 처리
```python
# PDF → image 별
import fitz  # PyMuPDF
pdf = fitz.open('doc.pdf')
images = []
for page in pdf:
    pix = page.get_pixmap(dpi=150)
    images.append(pix.tobytes())

# 매 page → vision LLM
for img in images:
    r = llm.complete(image=img, prompt='Extract text')
```

→ 큰 PDF = 비싼.

### Anthropic PDF 직접
```ts
// Claude 가 PDF native.
content: [
  { type: 'document', source: { type: 'base64', media_type: 'application/pdf', data: pdfBase64 } },
  { type: 'text', text: 'Summarize' },
]
```

→ Claude 가 image conversion 안.

### OCR (정확)
```python
# Tesseract (free)
import pytesseract
text = pytesseract.image_to_string(image)

# AWS Textract (managed, $$)
# Google Document AI
# Azure Form Recognizer

# Specialized > generic LLM (정확 + cheap).
```

→ Pure text 만 = OCR.
Layout / structure = LLM.

### Table extraction
```python
# Camelot / Tabula (PDF)
import camelot
tables = camelot.read_pdf('doc.pdf')

# AWS Textract (best)
# Google Document AI
```

→ Specialized 가 generic LLM 보다 정확.

### Image embedding
```python
# CLIP
from transformers import CLIPModel, CLIPProcessor

model = CLIPModel.from_pretrained('openai/clip-vit-base-patch32')
processor = CLIPProcessor.from_pretrained('openai/clip-vit-base-patch32')

inputs = processor(images=image, return_tensors='pt')
emb = model.get_image_features(**inputs)
```

→ Image search (반대 image).

### Multimodal RAG
```
User: "이 chart 가 무엇?"
1. Image embed.
2. Vector search (image + text mix).
3. Top-K → vision LLM context.
```

### Cohere Embed Multimodal
```ts
const r = await cohere.embed({
  texts: [text],
  images: [imageUrl],
  model: 'embed-multilingual-v3.0',
});
```

### Use case
```
- Receipt / invoice 데이터 추출.
- ID document scan.
- Form fill (사진 → form).
- Chart / graph 분석.
- Code screenshot → text.
- UI screenshot → component.
- Medical (X-ray).
- Insurance (damage estimate).
```

### UI screenshot → code
```ts
const r = await client.messages.create({
  messages: [{
    role: 'user',
    content: [
      { type: 'image', source: { ... screenshot ... } },
      { type: 'text', text: 'Generate React + Tailwind component matching this UI.' },
    ],
  }],
});
```

→ v0 / Vercel / Builder.io.

### Receipt extraction
```python
prompt = '''
Extract from receipt:
{
  "merchant": "...",
  "date": "...",
  "items": [{ "name": "...", "price": ... }],
  "total": ...
}
'''
```

→ Structured output (JSON mode).

### Cost
```
Claude Opus + image: $0.10 / image (대략).
GPT-4o + image (high): $0.05 / image.
Gemini 1.5 Pro: $0.0035 / image.

→ 1000 image = $50-100.
Specialized (Textract) = $0.01-0.05 / page.
```

### Image preprocessing
```ts
// Resize (전 send)
import sharp from 'sharp';

const resized = await sharp(image)
  .resize(1024, 1024, { fit: 'inside' })
  .jpeg({ quality: 80 })
  .toBuffer();
```

→ 작은 = cheap. Quality 약간 ↓.

### Streaming
```ts
const stream = await client.messages.stream({...});
for await (const event of stream) {
  // Text response 가 stream.
}
```

→ Image 자체 stream X. Response stream OK.

### Video
```python
# Frame extraction
import cv2
cap = cv2.VideoCapture('video.mp4')
frames = []
while True:
    ret, frame = cap.read()
    if not ret: break
    if frame_idx % 30 == 0:  # Every 1 sec
        frames.append(frame)

# 매 frame → vision LLM
```

→ Long video = 매우 비싼. Sample frame.

### Gemini 의 video native
```python
# Gemini 1.5+
response = model.generate_content([
    Part.from_uri(file_uri='gs://bucket/video.mp4'),
    'Summarize'
])
```

→ Native video. 1 hour video OK.

### Browser agent
```
Vision = browser screenshot.
LLM = "click button at (x, y)".
```

→ [[AI_Browser_Agent_Patterns]].

### Custom vision model
```python
# Fine-tune (specific task)
- LayoutLM (document)
- Donut (OCR-free)
- Pix2Struct
- LLaVA / Llama-Vision

# vs API:
- Cost ↓ (큰 volume).
- 더 정확 (specific task).
- Self-host complexity.
```

### Bounding box
```
"Detect bounding box of all faces."

→ LLM 가 부정확. YOLO / Grounding DINO 가 specialized.
```

### Eval
```python
# Vision QA dataset (custom)
test_cases = [
    { 'image': '...', 'question': 'What color?', 'expected': 'red' },
]

for c in test_cases:
    answer = llm.vision(c['image'], c['question'])
    pass_ = match(answer, c['expected'])
```

### Privacy
```
- 매 image 가 server (OpenAI, Anthropic).
- Sensitive (medical, ID) = on-prem 또는 specialized.
- HIPAA / GDPR 가 일부 limit.
```

### Anti-pattern
```
- High-detail mode 모든 image: cost 폭발.
- Generic LLM 가 OCR: specialized 가 더 정확 + cheap.
- Image upload 매 query: cache.
- Privacy 무시.
```

## 🤔 의사결정 기준
| 작업 | 추천 |
|---|---|
| Generic image Q&A | Claude / GPT-4o / Gemini |
| OCR | Tesseract / Textract |
| Table | Textract / Document AI |
| UI → code | Vision LLM |
| Receipt | Vision LLM (structured output) |
| Video summary | Gemini 1.5+ |
| Image search | CLIP / Cohere multimodal |
| Bounding box | YOLO / DINO |

## ❌ 안티패턴
- **High detail 모든 image**: cost 폭발.
- **Generic LLM 가 OCR (큰 volume)**: 비싼.
- **Privacy 무시**: leak.
- **Image cache 안 함**: 중복.
- **Resize 안 함**: token 폭발.
- **Long video 가 매 frame**: $$$.

## 🤖 LLM 활용 힌트
- Generic LLM (Claude / GPT-4o / Gemini) 가 default.
- OCR / table = specialized.
- CLIP 가 image embed.
- Resize + URL 가 cost 절약.

## 🔗 관련 문서
- [[AI_Multimodal_Vision_Patterns]]
- [[AI_Vision_Agents]]
- [[AI_RAG_Production]]
