---
id: mobile-camera-ar-patterns
title: Mobile Camera / AR — AVFoundation / CameraX / ARKit
category: Coding
status: draft
source_trust_level: B
verification_status: conceptual
created_at: 2026-05-09
updated_at: 2026-05-09
tags: [mobile, camera, ar, vibe-coding]
tech_stack: { language: "Swift / Kotlin", applicable_to: ["iOS", "Android"] }
applied_in: []
aliases: [AVCaptureSession, CameraX, ARKit, ARCore, RealityKit, Sceneform, AR foundation]
---

# Mobile Camera / AR

> Camera = native API. **AVFoundation (iOS), CameraX (Android), ARKit / ARCore for AR**.

## 📖 핵심 개념
- Capture session.
- Frame processing (real-time).
- Permission.
- AR (anchor, plane, light).

## 💻 코드 패턴

### iOS Camera (AVFoundation)
```swift
let session = AVCaptureSession()
session.sessionPreset = .high

guard let device = AVCaptureDevice.default(for: .video),
      let input = try? AVCaptureDeviceInput(device: device) else { return }

session.addInput(input)

let output = AVCapturePhotoOutput()
session.addOutput(output)

let preview = AVCaptureVideoPreviewLayer(session: session)
preview.frame = view.bounds
view.layer.addSublayer(preview)

session.startRunning()

// Capture
output.capturePhoto(with: AVCapturePhotoSettings(), delegate: self)
```

### iOS frame processing
```swift
let videoOutput = AVCaptureVideoDataOutput()
videoOutput.setSampleBufferDelegate(self, queue: DispatchQueue.global(qos: .userInitiated))
session.addOutput(videoOutput)

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }
    
    // 매 frame:
    // - Vision / Core ML inference.
    // - Filter / draw.
}
```

### iOS permission
```xml
<!-- Info.plist -->
<key>NSCameraUsageDescription</key>
<string>Used to take photos.</string>
```

```swift
AVCaptureDevice.requestAccess(for: .video) { granted in
    if granted { setupCamera() }
}
```

### Android CameraX
```kotlin
val cameraProvider = ProcessCameraProvider.getInstance(context).get()

val preview = Preview.Builder().build().also {
    it.setSurfaceProvider(viewFinder.surfaceProvider)
}

val imageCapture = ImageCapture.Builder().build()

val analyzer = ImageAnalysis.Builder().build().also {
    it.setAnalyzer(executor) { imageProxy ->
        // 매 frame.
        imageProxy.close()
    }
}

cameraProvider.bindToLifecycle(
    lifecycleOwner,
    CameraSelector.DEFAULT_BACK_CAMERA,
    preview, imageCapture, analyzer
)
```

→ Modern Android camera API.

### CameraX permission
```kotlin
val launcher = registerForActivityResult(ActivityResultContracts.RequestPermission()) { granted ->
    if (granted) setupCamera()
}

launcher.launch(Manifest.permission.CAMERA)
```

```xml
<uses-permission android:name='android.permission.CAMERA' />
```

### Photo capture
```swift
// iOS
output.capturePhoto(with: AVCapturePhotoSettings(), delegate: self)

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {
    let data = photo.fileDataRepresentation()
    // Save / process.
}
```

```kotlin
// Android
imageCapture.takePicture(
    ContextCompat.getMainExecutor(context),
    object : ImageCapture.OnImageCapturedCallback() {
        override fun onCaptureSuccess(image: ImageProxy) {
            // Process.
            image.close()
        }
    }
)
```

### Video record (iOS)
```swift
let movieOutput = AVCaptureMovieFileOutput()
session.addOutput(movieOutput)

let url = FileManager.default.temporaryDirectory.appendingPathComponent('video.mov')
movieOutput.startRecording(to: url, recordingDelegate: self)

// Stop
movieOutput.stopRecording()
```

### Video record (Android)
```kotlin
val recorder = Recorder.Builder().setQualitySelector(QualitySelector.from(Quality.HD)).build()
val videoCapture = VideoCapture.withOutput(recorder)

cameraProvider.bindToLifecycle(lifecycleOwner, CameraSelector.DEFAULT_BACK_CAMERA, videoCapture)

val recording = videoCapture.output
    .prepareRecording(context, fileOutputOptions)
    .start(executor) { event ->
        // Status.
    }

// Stop
recording.stop()
```

### iOS ARKit
```swift
import ARKit
import RealityKit

let arView = ARView(frame: view.bounds)
view.addSubview(arView)

let config = ARWorldTrackingConfiguration()
config.planeDetection = .horizontal
arView.session.run(config)

// Place 3D object on plane
arView.session.delegate = self

func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
    for anchor in anchors {
        if let plane = anchor as? ARPlaneAnchor {
            // Add 3D model.
        }
    }
}
```

### Android ARCore
```kotlin
// Sceneform 또는 ARCore SDK
val session = Session(context)
val config = Config(session)
config.planeFindingMode = Config.PlaneFindingMode.HORIZONTAL
session.configure(config)

// 매 frame.
val frame = session.update()
val planes = frame.getUpdatedTrackables(Plane::class.java)
```

→ Sceneform 가 deprecated. ARCore 직접.

### Cross-platform AR
```
- Unity AR Foundation: Unity 친화.
- React Native: react-native-arkit / react-native-arcore.
- Flutter: ar_flutter_plugin.

→ Native 가 가장 강력. Cross-platform 가 limit.
```

### ML on frame
```swift
// iOS Vision + Core ML
let request = VNCoreMLRequest(model: model) { request, error in
    // Result.
}

let handler = VNImageRequestHandler(cmSampleBuffer: sampleBuffer)
try? handler.perform([request])
```

```kotlin
// Android ML Kit
val detector = TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)
val image = InputImage.fromMediaImage(imageProxy.image!!, imageProxy.imageInfo.rotationDegrees)

detector.process(image).addOnSuccessListener { result ->
    // text.
}
```

### Performance
```
60 FPS = 16.7 ms / frame.
ML 가 너무 느린 = drop frame.

→ Background thread.
Smaller model.
Skip frame (every 2nd).
```

### Use case
```
- Photo / video capture.
- QR / barcode (ML Kit / Vision).
- Document scan.
- AR (place, measure).
- Filter (Snapchat 식).
- OCR.
- Face detect (Bumble 식).
```

### Camera permission UX
```
사용자 가 deny 후:
- Settings 가 manual 변경.
- Re-prompt 안 됨.

→ "Why need" 명확. 첫 prompt 신중.
```

### Battery / heat
```
Camera 가 큰 battery + heat.
- 사용 안 시 stop session.
- 매 frame ML 가 필요 시만.
- Resolution 가 use case 따라.
```

### iOS LiDAR (modern)
```swift
let config = ARWorldTrackingConfiguration()
config.sceneReconstruction = .mesh

if ARWorldTrackingConfiguration.supportsSceneReconstruction(.mesh) {
    // iPhone Pro 이상.
}
```

→ Depth + 3D mesh.

### Photo metadata
```swift
photo.metadata     // GPS, exif, ...
```

```kotlin
// CameraX 가 자동 set EXIF.
```

→ Privacy 신중.

### Streaming (live broadcast)
```
- HLS / DASH 로 stream.
- WebRTC (real-time).

→ 큰 bandwidth.
```

### Filter / effect
```swift
// Core Image
let context = CIContext()
let filter = CIFilter.gaussianBlur()
filter.setValue(ciImage, forKey: kCIInputImageKey)
let output = filter.outputImage
```

→ Real-time gauss / sharpen / etc.

### React Native
```
- react-native-vision-camera (modern).
- expo-camera.
- frame processor (worklet for ML).
```

### Flutter
```
- camera plugin.
- image_picker.
- ML 가 별 plugin.
```

### Production tips
```
1. Permission flow 신중.
2. Resolution = use case 따라.
3. Background thread (ML).
4. Battery / heat 인지.
5. Privacy (metadata strip).
6. Test 가 매 device (low-end).
```

### 함정
```
- Permission deny: re-prompt 안 됨.
- High resolution = slow + 큰 file.
- ML 이 main thread: jank.
- Background = stop session.
- AR 가 모든 device 안 됨 (LiDAR / GPU).
```

## 🤔 의사결정 기준
| 작업 | 추천 |
|---|---|
| iOS native | AVFoundation |
| Android native | CameraX |
| AR iOS | ARKit + RealityKit |
| AR Android | ARCore |
| ML detect | Vision (iOS) / ML Kit (Android) |
| Cross-platform | RN Vision Camera / Flutter camera |
| Document scan | VisionKit (iOS) / ML Kit (Android) |

## ❌ 안티패턴
- **Permission deny + no fallback**: bad UX.
- **High res + ML: drop frame.
- **Background + session running**: battery.
- **Metadata 가 store**: privacy leak.
- **AR 가 매 device 가정**: 안 됨.

## 🤖 LLM 활용 힌트
- AVFoundation (iOS) / CameraX (Android) 가 modern.
- ARKit / ARCore 가 native AR.
- ML 가 Vision / ML Kit.
- Resolution + permission + privacy.

## 🔗 관련 문서
- [[Android_CameraX_Patterns]]
- [[iOS_Audio_Video_AVKit]]
- [[Android_ML_Kit_Health]]