---
id: wiki-2026-0508-llm-ops-and-tuning
title: LLM Ops and Tuning
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [LLMOps, LLM Operations, Prompt Ops]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [llmops, eval, monitoring, prompt-mgmt, langsmith, helicone, langfuse]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack: { language: python/ts, framework: langsmith/langfuse/helicone }
---

# LLM Ops and Tuning

## 매 한 줄
> **"매 prompt 도 코드, eval 도 CI"**. Version, eval, monitor, alert — 일반 SW lifecycle 그대로 LLM 에 적용.

## 매 핵심
### 매 4 pillar
- **Versioning**: prompt + model 둘 다 version. Git 또는 LangSmith/Langfuse
- **Eval**: golden set + LLM-judge + human spot check, regression CI 에서 차단
- **Observability**: trace (token, latency, cost), Helicone/Langfuse
- **Drift**: input/output 분포 변화 감지, alerting

### 매 응용
1. **Prompt registry**: 중앙 저장, A/B
2. **Eval pipeline**: PR 마다 golden set 자동 실행
3. **Cost dashboard**: per-feature, per-user
4. **Hallucination detector**: claim-source mismatch
5. **Fallback chain**: primary fail → cheaper model

## 💻 패턴

### Pattern 1: Prompt versioning (Langfuse)
```python
from langfuse import Langfuse
lf = Langfuse()

# Pull versioned prompt
prompt = lf.get_prompt("summarize", label="production")
compiled = prompt.compile(input=text)

# Trace usage
with lf.trace(name="summarize") as t:
    out = client.messages.create(model="claude-opus-4-7", messages=[...])
    t.score(name="quality", value=judge(out))
```

### Pattern 2: Golden eval set
```python
golden = [
    {"input": "...", "expected_keywords": ["X", "Y"], "must_not": ["hallucination_token"]},
    ...
]

def eval_prompt(prompt_version):
    results = []
    for case in golden:
        out = run(prompt_version, case["input"])
        score = all(k in out for k in case["expected_keywords"]) \
                and not any(b in out for b in case["must_not"])
        results.append(score)
    return sum(results) / len(results)

assert eval_prompt("v2") >= eval_prompt("v1") - 0.02  # no regression
```

### Pattern 3: LLM-as-judge
```python
def judge(output, criteria):
    resp = client.messages.create(
        model="claude-haiku-4-7",  # cheap judge
        messages=[{"role": "user",
                   "content": f"Rate 1-5 on {criteria}:\n{output}\nReturn JSON {{score:n,reason:s}}"}],
    )
    return json.loads(resp.content[0].text)
```

### Pattern 4: Helicone proxy for observability
```python
client = anthropic.Anthropic(
    base_url="https://anthropic.helicone.ai",
    default_headers={"Helicone-Auth": f"Bearer {HELICONE_KEY}"},
)
# All calls now logged: latency, cost, prompt, response
```

### Pattern 5: Cost / latency budget alert
```python
TRACE_BUDGET = {"p99_ms": 3000, "cost_per_call_usd": 0.05}

def check_budget(traces):
    p99 = np.percentile([t.latency_ms for t in traces], 99)
    avg_cost = np.mean([t.cost for t in traces])
    if p99 > TRACE_BUDGET["p99_ms"]:
        alert(f"p99 latency {p99}ms exceeds budget")
```

### Pattern 6: Drift detection
```python
# Compare today's input embedding distribution to last week
from scipy.stats import ks_2samp
stat, p = ks_2samp(today_embeddings_norm, baseline_embeddings_norm)
if p < 0.01:
    alert("Input distribution drift detected")
```

### Pattern 7: Fallback chain
```python
def call_with_fallback(prompt):
    for model in ["claude-opus-4-7", "claude-sonnet-4-5", "claude-haiku-4-7"]:
        try:
            return client.messages.create(model=model, messages=[...], timeout=10)
        except (RateLimitError, APITimeoutError):
            continue
    raise RuntimeError("All models failed")
```

### Pattern 8: Prompt caching for cost
```python
# Cache long system prompt
client.messages.create(
    model="claude-opus-4-7",
    system=[{"type": "text", "text": LONG_SYS, "cache_control": {"type": "ephemeral"}}],
    messages=[...],
)
# 5min TTL, ~10x cheaper on cache hit
```

## 매 결정 기준

| 상황 | Tool |
|---|---|
| Prompt registry + eval + trace | LangSmith / Langfuse |
| Pure observability proxy | Helicone |
| Self-host OSS | Langfuse (Apache 2.0) |
| Tight budget control | Helicone (cost cap rules) |
| Multi-model A/B | LangSmith |

**기본값**: Langfuse self-hosted + golden eval in CI + prompt cache.

## 🔗 Graph
- 부모: [[MLOps]], [[DevOps]]
- 변형: [[LangSmith]], [[Helicone]], [[Langfuse]]
- 응용: [[RAG]], [[LLM-based_Code_Analysis]]
- Adjacent: [[Prompt_Engineering]], [[Evals]]

## 🤖 LLM 활용
**언제**: 모든 production LLM app — eval, trace, cost guard 필수.
**언제 X**: prototype 단계 (overkill), 단발성 batch job.

## ❌ 안티패턴
- Prompt 하드코딩 → 변경 시 deploy 필요
- Eval 없이 prompt 변경 → silent regression
- Trace 없이 production → debugging 불가
- Cost cap 없음 → bill shock
- Golden set 1년 안 갱신 → drift 못 잡음

## 🧪 검증 / 중복
- Verified (LangSmith docs, Langfuse docs, Helicone docs, Anthropic prompt caching). 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — 4 pillar, eval/trace/drift/fallback patterns |
