---
id: wiki-2026-0508-frame-type-restoration
title: Frame Type Restoration
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [frame restoration, video frame interpolation, FILM, RIFE, VFI]
duplicate_of: none
source_trust_level: A
confidence_score: 0.88
verification_status: applied
tags: [video, frame-interpolation, vfi, film, rife, slow-motion, deep-learning]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: PyTorch / RIFE / FILM / SoftSplat
---

# Frame Type Restoration (Video Frame Interpolation)

## 매 한 줄
> **"매 missing 또는 intermediate frame 의 generate"**. 매 slow-motion, 매 frame rate up-conversion (60fps → 120fps), 매 video restoration. 매 modern: 매 RIFE, FILM (Google 2022), 매 diffusion-based. 매 응용: 매 video editing, 매 streaming, 매 game.

## 매 핵심

### 매 task
- **VFI** (Video Frame Interpolation): 매 t 와 t+1 사이.
- **Frame extrapolation**: 매 future predict.
- **Restoration**: 매 corrupt frame.
- **Frame-rate up**: 30→60→120fps.

### 매 method
- **Optical flow + warping**: 매 RIFE, BMBC.
- **Kernel-based**: 매 SepConv.
- **Phase-based**.
- **Transformer / attention**: 매 VFIformer.
- **Diffusion**: 매 frame generation.

### 매 famous
- **RIFE** (2021): 매 real-time, IFNet.
- **FILM** (Google 2022): 매 multi-scale feature warping.
- **AdaCoF**, **CDFI**, **VFIformer**.
- **EMA-VFI** (2023).

### 매 응용
1. **Slow motion**: 매 240fps → 1000fps.
2. **Streaming**: 매 fps up.
3. **Video restoration**.
4. **Game**: 매 DLSS Frame Generation.
5. **Animation**: 매 inbetween.

## 💻 패턴

### RIFE inference
```python
import torch
from model.RIFE import Model

model = Model()
model.load_model('rife_v4.6')
model.eval().cuda()

def interpolate(frame0, frame1, n=1):
    """매 n intermediate frames."""
    with torch.no_grad():
        return [model.inference(frame0, frame1, t/(n+1)) for t in range(1, n+1)]
```

### Optical flow (RAFT)
```python
import torchvision.models.optical_flow as flow_models
raft = flow_models.raft_large(pretrained=True).eval().cuda()

def compute_flow(frame0, frame1):
    return raft(frame0.unsqueeze(0), frame1.unsqueeze(0))[-1]
```

### Forward warping (splat)
```python
def softmax_splat(frame, flow, t=0.5):
    """매 forward warp with softmax weighted aggregation."""
    H, W = frame.shape[-2:]
    output = torch.zeros_like(frame)
    weights = torch.zeros_like(frame[:, :1])
    
    grid_y, grid_x = torch.meshgrid(torch.arange(H), torch.arange(W))
    new_x = grid_x + t * flow[:, 0]
    new_y = grid_y + t * flow[:, 1]
    # 매 scatter / aggregate
    return output / (weights + 1e-9)
```

### Backward warping
```python
def backward_warp(frame, flow):
    """매 sampler-based."""
    H, W = frame.shape[-2:]
    grid = torch.stack(torch.meshgrid(torch.linspace(-1, 1, W), torch.linspace(-1, 1, H), indexing='xy'), dim=-1)
    new_grid = grid + flow.permute(0, 2, 3, 1) * 2 / torch.tensor([W, H])
    return torch.nn.functional.grid_sample(frame, new_grid, align_corners=True)
```

### FILM-style multi-scale
```python
class FILMNet(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.scales = [1, 2, 4, 8]
        self.flow_estimators = torch.nn.ModuleList([UNet() for _ in self.scales])
    
    def forward(self, f0, f1, t=0.5):
        flows = []
        for scale in self.scales:
            f0_s = downsample(f0, scale)
            f1_s = downsample(f1, scale)
            flow = self.flow_estimators[len(flows)](f0_s, f1_s, t)
            flows.append(flow)
        # 매 fuse + warp
        return synthesize(f0, f1, flows, t)
```

### Diffusion-based VFI
```python
class DiffusionVFI(torch.nn.Module):
    def forward(self, f0, f1, t=0.5, n_steps=50):
        x = torch.randn_like(f0)
        for step in reversed(range(n_steps)):
            noise = self.unet(x, f0, f1, t, step)
            x = x - 0.01 * noise
        return x
```

### Frame quality metric
```python
import lpips
loss_fn = lpips.LPIPS(net='vgg').cuda()
def vfi_quality(predicted, target):
    psnr = 10 * torch.log10(1 / ((predicted - target) ** 2).mean())
    ssim_val = ssim(predicted, target)
    lpips_val = loss_fn(predicted * 2 - 1, target * 2 - 1)
    return {'psnr': psnr.item(), 'ssim': ssim_val.item(), 'lpips': lpips_val.item()}
```

### Real-time pipeline
```python
import cv2
cap = cv2.VideoCapture('input.mp4')
out = cv2.VideoWriter('out.mp4', fourcc, 60.0, (W, H))  # 매 30→60

prev = None
while True:
    ret, frame = cap.read()
    if not ret: break
    if prev is not None:
        mid = interpolate(prev, frame)
        out.write(prev); out.write(mid)
    prev = frame
out.write(prev)
```

### DLSS-style game frame gen
```python
# 매 Nvidia DLSS 3 / FSR 3
# 매 motion vector + depth + previous frame → 매 generated frame
def game_frame_gen(prev_frame, current_frame, motion_vectors, depth):
    flow = motion_vectors  # 매 from game engine
    interpolated = warp(prev_frame, flow * 0.5) + warp(current_frame, -flow * 0.5)
    return interpolated / 2
```

### Multi-frame interpolation
```python
def n_interpolate(frame0, frame1, n=4):
    """매 n intermediate."""
    frames = [frame0]
    for i in range(1, n + 1):
        t = i / (n + 1)
        frames.append(model.inference(frame0, frame1, t))
    frames.append(frame1)
    return frames
```

### Train (synthetic data)
```python
def train_vfi(dataloader, model):
    for f0, ft, f1 in dataloader:  # 매 triplet
        pred = model(f0, f1, t=0.5)
        loss = F.l1_loss(pred, ft) + 0.1 * lpips_loss(pred, ft)
        loss.backward(); optim.step()
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Real-time | RIFE / DLSS-FG |
| Best quality | FILM / VFIformer |
| Slow-motion film | FILM (large gap) |
| Game | DLSS 3 / FSR 3 |
| Animation | Specialized (anim style) |
| Heavy occlusion | Diffusion |

**기본값**: 매 RIFE for speed + 매 FILM for large gap + 매 PSNR/SSIM/LPIPS triple eval.

## 🔗 Graph
- 부모: [[Computer Vision|Computer-Vision]]
- 변형: [[VFI]]
- Adjacent: [[Diffusion-Models]]

## 🤖 LLM 활용
**언제**: 매 video editing. 매 game frame gen. 매 anime inbetween.
**언제 X**: 매 audio. 매 still image.

## ❌ 안티패턴
- **Linear blend**: 매 ghost.
- **Single-scale flow**: 매 large motion fail.
- **PSNR only metric**: 매 perceptual quality 의 miss.
- **No occlusion handle**: 매 hole.
- **Real-time without GPU**: 매 slow.

## 🧪 검증 / 중복
- Verified (RIFE 2021, FILM 2022, Vimeo90k benchmark).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-04-26 | Auto |
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — VFI methods + 매 RIFE / FILM / flow / warp / diffusion code |
