---
id: wiki-2026-0508-vps-nerf
title: VPS-NeRF (Visual Positioning + Neural Radiance Fields)
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [VPS NeRF, Neural Radiance Fields for Localization]
duplicate_of: none
source_trust_level: A
confidence_score: 0.8
verification_status: applied
tags: [computer-vision, 3d-reconstruction, neural-rendering, localization]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: PyTorch / nerfstudio
---

# VPS-NeRF (Visual Positioning + NeRF)

## 매 한 줄
> **"매 NeRF 의 6-DoF camera pose 의 estimate"**. Mildenhall et al. "NeRF" (2020) → iNeRF (Yen-Chen et al. 2021) inverted NeRF for pose → 2026 의 3D Gaussian Splatting (3DGS) 가 dominant; VPS-NeRF 는 visual positioning system (AR/robotics localization) 의 NeRF/3DGS-based variant.

## 매 핵심

### 매 NeRF 기본
- Function `F(x, d) → (RGB, σ)` parametrized by MLP.
- `x` ∈ ℝ³ (3D point), `d` ∈ S² (view direction).
- Volume rendering: integrate along ray to produce pixel.
- Loss: photometric on rendered vs ground-truth images.

### 매 VPS-NeRF / iNeRF idea
- Forward NeRF: pose → image.
- Inverse: image → pose. Minimize photometric loss w.r.t. camera pose `T ∈ SE(3)`.
- `T* = argmin_T Σ ||I_obs - render_NeRF(T)||²`.
- Optimize via gradient descent on tangent space of SE(3).

### 매 Modern 2026 변형
- **3DGS-based VPS**: replaces NeRF with 3D Gaussians (Kerbl et al. 2023) — 매 100x faster rendering.
- **Splat-based localization**: query image → match against splats → solve PnP + photometric refinement.
- **Foundation model integration**: DINOv2 / SAM-2 features for robust matching.

### 매 응용
1. AR (Niantic Lightship VPS, Google Live View 2024+).
2. Robotics localization in pre-mapped environments.
3. Drone navigation indoor (no GPS).
4. Asset tracking in warehouses.
5. Mixed reality persistent anchors (Apple Vision Pro 2026).

## 💻 패턴

### NeRF training (nerfstudio, 2026)
```bash
# nerfstudio 1.x — standard pipeline
ns-train nerfacto --data path/to/captures
ns-render --load-config outputs/.../config.yml
```

### iNeRF pose refinement
```python
import torch
from torch import nn

def inerf_pose_refine(nerf_model, target_image, T_init, n_iter=300, lr=1e-3):
    """Refine SE(3) pose to match observed image to NeRF render."""
    # Parametrize as twist (se3)
    twist = nn.Parameter(torch.zeros(6))
    optim = torch.optim.Adam([twist], lr=lr)

    for i in range(n_iter):
        T = T_init @ se3_exp(twist)        # SE(3) update
        rays_o, rays_d = pose_to_rays(T, intrinsics, H, W)
        rgb_pred = nerf_model.render(rays_o, rays_d)
        loss = ((rgb_pred - target_image) ** 2).mean()
        optim.zero_grad(); loss.backward(); optim.step()
    return T_init @ se3_exp(twist.detach())
```

### 3D Gaussian Splatting render (2026 dominant)
```python
# gsplat library (UC Berkeley, 2024+)
from gsplat import rasterization

def render_gs(means, quats, scales, opacities, colors, viewmat, K, H, W):
    rendered, alpha, info = rasterization(
        means=means,           # (N, 3)
        quats=quats,           # (N, 4)
        scales=scales,         # (N, 3)
        opacities=opacities,   # (N,)
        colors=colors,         # (N, 3)
        viewmats=viewmat[None],
        Ks=K[None],
        width=W, height=H,
    )
    return rendered[0]
```

### VPS pipeline (query localization)
```python
def localize(query_img, splat_scene, foundation_features):
    # 1. Coarse: match foundation features (DINOv2) to scene
    candidate_poses = retrieve_candidate_poses(query_img, foundation_features, k=5)
    # 2. PnP for initial guess
    T_init = solve_pnp(query_img, candidate_poses)
    # 3. Photometric refinement on splats
    T_refined = photometric_refine(query_img, splat_scene, T_init)
    return T_refined
```

### SE(3) exponential map
```python
import torch

def se3_exp(twist):
    # twist: (6,) = (rotation, translation tangent)
    omega, v = twist[:3], twist[3:]
    theta = omega.norm() + 1e-8
    K = skew(omega / theta)
    R = torch.eye(3) + torch.sin(theta)*K + (1-torch.cos(theta))*(K@K)
    V = torch.eye(3) + ((1-torch.cos(theta))/theta)*K + ((theta-torch.sin(theta))/theta)*(K@K)
    T = torch.eye(4)
    T[:3,:3] = R
    T[:3, 3] = V @ v
    return T

def skew(w):
    return torch.tensor([[0, -w[2], w[1]],
                         [w[2], 0, -w[0]],
                         [-w[1], w[0], 0]])
```

### Photometric loss with masking
```python
def photometric_loss(I_pred, I_obs, mask=None):
    diff = (I_pred - I_obs).abs()
    if mask is not None:
        diff = diff * mask
    return diff.mean()
# Mask out dynamic objects (pedestrians, cars) via SAM-2 segmentation
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Static indoor scene, time budget | 3DGS + photometric refine |
| Outdoor + transient objects | NeRF-W / mask transients |
| Real-time mobile (AR) | Lightweight feature match + PnP only |
| Sparse views | NeRF / NeuS variant; 3DGS struggles |
| Large-scale city | Block-NeRF / hierarchical 3DGS |

**기본값** (2026): 3D Gaussian Splatting + foundation features (DINOv2) for coarse + photometric refine for fine.

## 🔗 Graph
- 부모: [[3D-Reconstruction]]
- 변형: [[NeRF]] · [[3D_Gaussian_Splatting]]
- 응용: [[Mixed-Reality]]
- Adjacent: [[SLAM]]

## 🤖 LLM 활용
**언제**: 6-DoF localization in pre-mapped 3D environment. AR persistent anchors. Robot relocalization. Photo geolocation in known scene.
**언제 X**: Real-time SLAM from scratch (use ORB-SLAM3 / DROID-SLAM). GPS-only outdoor (use RTK). Single-image absolute pose without map (use VPS feature databases).

## ❌ 안티패턴
- **Pure NeRF in 2026**: 매 3DGS / Mip-Splatting 의 standard — 매 vanilla NeRF 의 too slow for VPS.
- **No transient masking**: dynamic objects 의 pollute photometric loss → biased pose.
- **Ignore initialization**: photometric refinement 의 local optimum 의 trap — 매 PnP / feature retrieval init 의 critical.
- **Only photometric**: 매 textureless surfaces 의 fail — 매 feature-based + photometric hybrid.

## 🧪 검증 / 중복
- Verified (NeRF Mildenhall 2020, iNeRF Yen-Chen 2021, 3DGS Kerbl 2023, nerfstudio docs 2026).
- 신뢰도 A-.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — VPS-NeRF with iNeRF, 3DGS, SE(3) optimization, 2026 pipeline |
