[G1-Sync] Manual knowledge update
This commit is contained in:
Antigravity Agent
@@ -0,0 +1,412 @@
|
||||
---
|
||||
id: android-ml-kit-health
|
||||
title: Android ML Kit / Health Connect / On-device AI
|
||||
category: Coding
|
||||
status: draft
|
||||
source_trust_level: B
|
||||
verification_status: conceptual
|
||||
created_at: 2026-05-09
|
||||
updated_at: 2026-05-09
|
||||
tags: [android, mlkit, health, on-device, vibe-coding]
|
||||
tech_stack: { language: "Kotlin", applicable_to: ["Android"] }
|
||||
applied_in: []
|
||||
aliases: [ML Kit, Health Connect, MediaPipe, on-device ML, AICore, Gemini Nano]
|
||||
---
|
||||
|
||||
# Android ML Kit / Health Connect / On-device AI
|
||||
|
||||
> Google ML Kit (built-in ML), Health Connect (cross-app health), MediaPipe (advanced ML), Gemini Nano (on-device LLM, Pixel 9+).
|
||||
|
||||
## 📖 핵심 개념
|
||||
- ML Kit: 일반 ML task 빠른 사용.
|
||||
- Health Connect: data 통합 + permissions.
|
||||
- MediaPipe: vision / LLM 자체 모델.
|
||||
- Gemini Nano: AICore — on-device LLM.
|
||||
|
||||
## 💻 코드 패턴
|
||||
|
||||
### ML Kit — Text recognition
|
||||
```kotlin
|
||||
implementation("com.google.mlkit:text-recognition:16.0.0")
|
||||
```
|
||||
|
||||
```kotlin
|
||||
import com.google.mlkit.vision.text.TextRecognition
|
||||
import com.google.mlkit.vision.text.latin.TextRecognizerOptions
|
||||
|
||||
val recognizer = TextRecognition.getClient(TextRecognizerOptions.DEFAULT_OPTIONS)
|
||||
|
||||
val image = InputImage.fromBitmap(bitmap, rotation)
|
||||
val result = recognizer.process(image).await() // suspend extension
|
||||
|
||||
for (block in result.textBlocks) {
|
||||
for (line in block.lines) {
|
||||
println(line.text)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
→ OCR. 영수증 / 명함.
|
||||
|
||||
### ML Kit — Barcode
|
||||
```kotlin
|
||||
implementation("com.google.mlkit:barcode-scanning:17.2.0")
|
||||
|
||||
val scanner = BarcodeScanning.getClient()
|
||||
val barcodes = scanner.process(image).await()
|
||||
|
||||
for (barcode in barcodes) {
|
||||
when (barcode.valueType) {
|
||||
Barcode.TYPE_URL -> println("URL: ${barcode.url?.url}")
|
||||
Barcode.TYPE_WIFI -> println("WiFi: ${barcode.wifi?.ssid}")
|
||||
Barcode.TYPE_TEXT -> println(barcode.rawValue)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### ML Kit — Face detection
|
||||
```kotlin
|
||||
val options = FaceDetectorOptions.Builder()
|
||||
.setPerformanceMode(FaceDetectorOptions.PERFORMANCE_MODE_FAST)
|
||||
.setLandmarkMode(FaceDetectorOptions.LANDMARK_MODE_ALL)
|
||||
.setClassificationMode(FaceDetectorOptions.CLASSIFICATION_MODE_ALL)
|
||||
.build()
|
||||
|
||||
val detector = FaceDetection.getClient(options)
|
||||
val faces = detector.process(image).await()
|
||||
|
||||
for (face in faces) {
|
||||
val bounds = face.boundingBox
|
||||
val smilingProb = face.smilingProbability ?: 0f
|
||||
val leftEye = face.getLandmark(FaceLandmark.LEFT_EYE)?.position
|
||||
}
|
||||
```
|
||||
|
||||
### ML Kit — Translation
|
||||
```kotlin
|
||||
implementation("com.google.mlkit:translate:17.0.2")
|
||||
|
||||
val options = TranslatorOptions.Builder()
|
||||
.setSourceLanguage(TranslateLanguage.KOREAN)
|
||||
.setTargetLanguage(TranslateLanguage.ENGLISH)
|
||||
.build()
|
||||
|
||||
val translator = Translation.getClient(options)
|
||||
translator.downloadModelIfNeeded().await()
|
||||
|
||||
val translation = translator.translate("안녕").await()
|
||||
// → "Hello"
|
||||
```
|
||||
|
||||
### ML Kit — Pose / Body
|
||||
```kotlin
|
||||
implementation("com.google.mlkit:pose-detection:18.0.0-beta3")
|
||||
|
||||
val options = PoseDetectorOptions.Builder()
|
||||
.setDetectorMode(PoseDetectorOptions.STREAM_MODE)
|
||||
.build()
|
||||
|
||||
val detector = PoseDetection.getClient(options)
|
||||
val pose = detector.process(image).await()
|
||||
|
||||
val nose = pose.getPoseLandmark(PoseLandmark.NOSE)?.position
|
||||
val leftWrist = pose.getPoseLandmark(PoseLandmark.LEFT_WRIST)?.position
|
||||
```
|
||||
|
||||
→ Fitness app.
|
||||
|
||||
### MediaPipe (advanced)
|
||||
```kotlin
|
||||
implementation("com.google.mediapipe:tasks-vision:0.10.20")
|
||||
|
||||
val options = ImageClassifier.ImageClassifierOptions.builder()
|
||||
.setBaseOptions(BaseOptions.builder().setModelAssetPath("model.tflite").build())
|
||||
.setMaxResults(5)
|
||||
.build()
|
||||
|
||||
val classifier = ImageClassifier.createFromOptions(context, options)
|
||||
val result = classifier.classify(MPImage.fromBitmap(bitmap))
|
||||
|
||||
for (cat in result.classifications()[0].categories()) {
|
||||
println("${cat.categoryName()}: ${cat.score()}")
|
||||
}
|
||||
```
|
||||
|
||||
→ Custom TFLite 모델.
|
||||
|
||||
### Gemini Nano (on-device, Pixel 9+)
|
||||
```kotlin
|
||||
implementation("com.google.ai.edge.aicore:aicore:0.0.1-exp01")
|
||||
|
||||
val options = generationConfig {
|
||||
context = context // Activity / Application
|
||||
temperature = 0.2f
|
||||
topK = 16
|
||||
maxOutputTokens = 256
|
||||
}
|
||||
|
||||
val generativeModel = GenerativeModel(generationConfig = options)
|
||||
|
||||
val response = generativeModel.generateContent("Summarize this article: ...").text
|
||||
```
|
||||
|
||||
→ Cloud 호출 없이. Privacy + offline + free.
|
||||
|
||||
→ ⚠️ Pixel 9 / 일부 device 만. Compatibility check.
|
||||
|
||||
### Health Connect setup
|
||||
```kotlin
|
||||
implementation("androidx.health.connect:connect-client:1.1.0-alpha07")
|
||||
```
|
||||
|
||||
```xml
|
||||
<!-- AndroidManifest.xml -->
|
||||
<uses-permission android:name="android.permission.health.READ_STEPS" />
|
||||
<uses-permission android:name="android.permission.health.WRITE_STEPS" />
|
||||
<uses-permission android:name="android.permission.health.READ_HEART_RATE" />
|
||||
|
||||
<queries>
|
||||
<package android:name="com.google.android.apps.healthdata" />
|
||||
</queries>
|
||||
```
|
||||
|
||||
```kotlin
|
||||
import androidx.health.connect.client.HealthConnectClient
|
||||
import androidx.health.connect.client.records.StepsRecord
|
||||
import androidx.health.connect.client.permission.HealthPermission
|
||||
|
||||
val healthConnectClient = HealthConnectClient.getOrCreate(context)
|
||||
|
||||
val permissions = setOf(
|
||||
HealthPermission.getReadPermission(StepsRecord::class),
|
||||
HealthPermission.getWritePermission(StepsRecord::class),
|
||||
)
|
||||
|
||||
// Request
|
||||
val launcher = registerForActivityResult(
|
||||
PermissionController.createRequestPermissionResultContract()
|
||||
) { granted -> /* ... */ }
|
||||
|
||||
launcher.launch(permissions)
|
||||
```
|
||||
|
||||
### Read steps
|
||||
```kotlin
|
||||
val response = healthConnectClient.readRecords(
|
||||
ReadRecordsRequest(
|
||||
recordType = StepsRecord::class,
|
||||
timeRangeFilter = TimeRangeFilter.between(
|
||||
Instant.now().minusSeconds(3600 * 24),
|
||||
Instant.now()
|
||||
)
|
||||
)
|
||||
)
|
||||
|
||||
val totalSteps = response.records.sumOf { it.count }
|
||||
```
|
||||
|
||||
### Write steps
|
||||
```kotlin
|
||||
healthConnectClient.insertRecords(listOf(
|
||||
StepsRecord(
|
||||
count = 5000,
|
||||
startTime = Instant.now().minusSeconds(3600),
|
||||
endTime = Instant.now(),
|
||||
startZoneOffset = ZoneOffset.UTC,
|
||||
endZoneOffset = ZoneOffset.UTC,
|
||||
)
|
||||
))
|
||||
```
|
||||
|
||||
### Aggregation
|
||||
```kotlin
|
||||
val agg = healthConnectClient.aggregate(
|
||||
AggregateRequest(
|
||||
metrics = setOf(StepsRecord.COUNT_TOTAL),
|
||||
timeRangeFilter = TimeRangeFilter.between(start, end)
|
||||
)
|
||||
)
|
||||
|
||||
val total = agg[StepsRecord.COUNT_TOTAL] ?: 0L
|
||||
```
|
||||
|
||||
### Background sync
|
||||
```kotlin
|
||||
class HealthSyncWorker(ctx: Context, params: WorkerParameters) : CoroutineWorker(ctx, params) {
|
||||
override suspend fun doWork(): Result {
|
||||
val steps = readSteps()
|
||||
syncToServer(steps)
|
||||
return Result.success()
|
||||
}
|
||||
}
|
||||
|
||||
// Schedule
|
||||
val request = PeriodicWorkRequestBuilder<HealthSyncWorker>(15, TimeUnit.MINUTES).build()
|
||||
WorkManager.getInstance(ctx).enqueueUniquePeriodicWork("health-sync", KEEP, request)
|
||||
```
|
||||
|
||||
### Privacy
|
||||
```
|
||||
- 명시적 사용자 consent
|
||||
- Data minimization (read only what needed)
|
||||
- 사용자 가 access / delete 가능
|
||||
- GDPR / HIPAA compliance (US)
|
||||
```
|
||||
|
||||
### CameraX + ML Kit
|
||||
```kotlin
|
||||
val analyzer = ImageAnalysis.Analyzer { imageProxy ->
|
||||
val mediaImage = imageProxy.image ?: return@Analyzer
|
||||
val image = InputImage.fromMediaImage(mediaImage, imageProxy.imageInfo.rotationDegrees)
|
||||
|
||||
barcodeScanner.process(image)
|
||||
.addOnSuccessListener { barcodes -> /* ... */ }
|
||||
.addOnCompleteListener { imageProxy.close() }
|
||||
}
|
||||
```
|
||||
|
||||
→ Real-time camera + ML.
|
||||
|
||||
### TFLite (custom 모델)
|
||||
```kotlin
|
||||
implementation("org.tensorflow:tensorflow-lite:2.16.1")
|
||||
|
||||
val interpreter = Interpreter(loadModelFile())
|
||||
|
||||
val input = ByteBuffer.allocateDirect(...)
|
||||
val output = ByteBuffer.allocateDirect(...)
|
||||
|
||||
interpreter.run(input, output)
|
||||
```
|
||||
|
||||
→ 자체 모델 (TF / PyTorch → TFLite).
|
||||
|
||||
### Audio classification
|
||||
```kotlin
|
||||
implementation("com.google.mediapipe:tasks-audio:0.10.20")
|
||||
|
||||
val options = AudioClassifier.AudioClassifierOptions.builder()
|
||||
.setBaseOptions(BaseOptions.builder().setModelAssetPath("yamnet.tflite").build())
|
||||
.build()
|
||||
|
||||
val classifier = AudioClassifier.createFromOptions(context, options)
|
||||
val result = classifier.classify(MPAudioData.create(buffer, sampleRate))
|
||||
|
||||
for (cat in result.classifications()[0].categories()) {
|
||||
println("${cat.categoryName()}: ${cat.score()}")
|
||||
// "Music", "Speech", "Bark", ...
|
||||
}
|
||||
```
|
||||
|
||||
### Subject segmentation (BG removal)
|
||||
```kotlin
|
||||
implementation("com.google.mlkit:subject-segmentation:16.0.0-beta1")
|
||||
|
||||
val segmenter = SubjectSegmentation.getClient()
|
||||
val result = segmenter.process(image).await()
|
||||
|
||||
val foregroundBitmap = result.foregroundBitmap
|
||||
// 배경 X — 사용자 / 사람 만
|
||||
```
|
||||
|
||||
### Document scanner
|
||||
```kotlin
|
||||
implementation("com.google.android.gms:play-services-mlkit-document-scanner:16.0.0-beta1")
|
||||
|
||||
val options = GmsDocumentScannerOptions.Builder()
|
||||
.setGalleryImportAllowed(true)
|
||||
.setPageLimit(5)
|
||||
.setResultFormats(RESULT_FORMAT_PDF, RESULT_FORMAT_JPEG)
|
||||
.build()
|
||||
|
||||
GmsDocumentScanning.getClient(options)
|
||||
.getStartScanIntent(activity)
|
||||
.addOnSuccessListener { intent -> startActivityForResult(intent, ...) }
|
||||
```
|
||||
|
||||
→ Document scan + auto crop + PDF.
|
||||
|
||||
### Smart reply (chat)
|
||||
```kotlin
|
||||
implementation("com.google.mlkit:smart-reply:17.0.4")
|
||||
|
||||
val smartReply = SmartReply.getClient()
|
||||
val conversation = listOf(
|
||||
TextMessage.createForRemoteUser("Hi", System.currentTimeMillis(), "user_1"),
|
||||
)
|
||||
|
||||
val result = smartReply.suggestReplies(conversation).await()
|
||||
|
||||
for (suggestion in result.suggestions) {
|
||||
println(suggestion.text)
|
||||
// "Hi!", "Hello", "Hey there"
|
||||
}
|
||||
```
|
||||
|
||||
### Battery / performance
|
||||
```
|
||||
On-device ML = 빠름 + free + private.
|
||||
But:
|
||||
- Battery 사용
|
||||
- Memory
|
||||
- Model size (10-100 MB)
|
||||
|
||||
→ 측정 + throttle.
|
||||
```
|
||||
|
||||
### Cloud vs on-device
|
||||
```
|
||||
On-device:
|
||||
+ Free (no API cost)
|
||||
+ Private (no upload)
|
||||
+ Offline
|
||||
+ Low latency
|
||||
- Limited model size
|
||||
- Battery / memory
|
||||
|
||||
Cloud (Vertex AI / Gemini API):
|
||||
+ Bigger / better model
|
||||
+ Always updated
|
||||
- Cost
|
||||
- Privacy
|
||||
- Latency
|
||||
```
|
||||
|
||||
→ 일반 task = on-device. Complex / accuracy critical = cloud.
|
||||
|
||||
### Edge AI (modern stack)
|
||||
```
|
||||
1. Quick task: ML Kit (built-in)
|
||||
2. Custom: MediaPipe + TFLite
|
||||
3. LLM: Gemini Nano (Pixel 9+)
|
||||
4. Cloud fallback: Gemini API
|
||||
```
|
||||
|
||||
## 🤔 의사결정 기준
|
||||
| 작업 | 추천 |
|
||||
|---|---|
|
||||
| OCR / barcode / face | ML Kit |
|
||||
| 자체 모델 | MediaPipe / TFLite |
|
||||
| On-device LLM | Gemini Nano (Pixel 9+) |
|
||||
| Health data | Health Connect |
|
||||
| 일반 LLM | Cloud (Gemini API) |
|
||||
| Real-time | CameraX + ML Kit |
|
||||
|
||||
## ❌ 안티패턴
|
||||
- **모든 거 cloud LLM**: cost / privacy.
|
||||
- **Health Connect 권한 한 번 + 모든 거**: minimum access.
|
||||
- **PII model 학습 외부 send**: privacy violation.
|
||||
- **Gemini Nano + 모든 device**: compatibility check.
|
||||
- **Battery 무시**: 사용자 끄기.
|
||||
- **모델 download 큰 + first launch**: progressive.
|
||||
|
||||
## 🤖 LLM 활용 힌트
|
||||
- ML Kit = 가장 단순 + 빠른 시작.
|
||||
- Health Connect = cross-app data.
|
||||
- MediaPipe = custom + advanced.
|
||||
- Gemini Nano = privacy-friendly LLM.
|
||||
|
||||
## 🔗 관련 문서
|
||||
- [[Android_CameraX_Patterns]]
|
||||
- [[AI_Local_LLM_Inference]]
|
||||
- [[Mobile_Push_Deep]]
|
||||
Reference in New Issue
Block a user