2nd/10_Wiki/Topics/AI_and_ML/Text-Mining.md

---
id: wiki-2026-0508-text-mining
title: Text Mining
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Text Analytics, Information Extraction]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [nlp, text-mining, information-extraction]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: Python
  framework: spaCy / LLM-based
---

# Text Mining

## 매 한 줄
> **"매 unstructured text → structured signal"**. Text mining 매 large text corpora 에서 patterns / entities / relationships / sentiment 의 extract 하는 분야. 매 traditional (TF-IDF, NER models) 에서 매 LLM-based extraction (structured output, function calling) 으로 매 paradigm shift.

## 매 핵심

### 매 traditional pipeline
- **Tokenization** → **POS tagging** → **NER** → **dependency parsing** → **sentiment** → **topic modeling**.
- Tools: spaCy, NLTK, scikit-learn (TF-IDF + classifiers), gensim (LDA).
- 매 production NER: spaCy `transformers` pipeline or fine-tuned BERT.

### 매 modern (LLM-based)
- **Structured output** — 매 LLM 이 JSON schema 의 fill (Claude tool use, OpenAI structured output, Outlines).
- **Few-shot extraction** — 매 fine-tune 없이 매 5 examples 만으로 task 의 정의.
- **Long-context** — 매 200k+ token document 의 single-shot processing.
- 매 cost trade-off: spaCy NER ~$0.0001/doc vs LLM ~$0.01/doc — 매 batch + small model (Haiku, gpt-4o-mini) 으로 reduce.

### 매 응용
1. Resume parsing, contract analysis (entity + clause extraction).
2. Customer feedback aggregation (sentiment + topic).
3. Biomedical literature mining (gene/protein/disease NER).

## 💻 패턴

### spaCy NER (traditional)
```python
import spacy
nlp = spacy.load("en_core_web_trf")
doc = nlp("Apple acquired Anthropic for $50B in March 2025.")
for ent in doc.ents:
    print(ent.text, ent.label_)
# Apple ORG, Anthropic ORG, $50B MONEY, March 2025 DATE
```

### TF-IDF + classifier
```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=20000)),
    ("clf", LogisticRegression(max_iter=1000)),
])
pipe.fit(X_train, y_train)
```

### LLM structured extraction (Claude)
```python
from anthropic import Anthropic
from pydantic import BaseModel

class Contract(BaseModel):
    parties: list[str]
    effective_date: str
    total_value_usd: float | None
    governing_law: str | None

client = Anthropic()
resp = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=2000,
    tools=[{
        "name": "extract_contract",
        "input_schema": Contract.model_json_schema(),
    }],
    tool_choice={"type": "tool", "name": "extract_contract"},
    messages=[{"role": "user", "content": contract_text}],
)
data = Contract(**resp.content[0].input)
```

### Topic modeling (BERTopic)
```python
from bertopic import BERTopic
from sentence_transformers import SentenceTransformer

embed = SentenceTransformer("BAAI/bge-large-en-v1.5")
topic_model = BERTopic(embedding_model=embed, min_topic_size=10)
topics, probs = topic_model.fit_transform(docs)
topic_model.get_topic_info()
```

### Long-context document QA
```python
# 200k token contract → single LLM call
resp = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=4000,
    messages=[{"role": "user", "content": [
        {"type": "text", "text": contract_full_text,
         "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": "Extract all change-of-control provisions."},
    ]}],
)
```

### Hybrid (LLM + regex precheck)
```python
import re
DATE_RE = re.compile(r"\b\d{4}-\d{2}-\d{2}\b")
candidates = DATE_RE.findall(text)
# 매 LLM 의 candidate 만 disambiguate — 매 cost reduce
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| High-volume (M docs/day) NER | spaCy / fine-tuned BERT |
| Complex schema, low volume | LLM structured output |
| Topic discovery | BERTopic / embeddings + clustering |
| Sentiment | Fine-tuned RoBERTa or LLM |
| Long documents (>50k tokens) | LLM with caching |
| Domain-specific (legal, medical) | Fine-tune + LLM hybrid |

**기본값**: 매 prototype LLM, 매 production 은 LLM (low volume) or distilled fine-tuned model (high volume).

## 🔗 Graph
- 부모: [[Information Retrieval]]
- 변형: [[Named-Entity-Recognition]] · [[Sentiment-Analysis]]
- 응용: [[RAG]] · [[Search]]
- Adjacent: [[Embeddings]]

## 🤖 LLM 활용
**언제**: 매 unstructured text corpus 의 query / extract / classify, schema-driven extraction, low-to-medium volume.
**언제 X**: 매 milli-second latency 의 필요 (real-time chat moderation) — 매 small distilled model.

## ❌ 안티패턴
- **Regex-only complex extraction**: 매 brittle — 매 LLM hybrid 로 graceful.
- **No evaluation set**: 매 LLM 매 hallucinate — 매 ground-truth eval 의 maintain.
- **Full-document LLM 의 every query**: 매 cache or pre-extract structured DB.
- **Unicode normalization 의 skip**: 매 Korean/CJK text 매 NFC normalize 필수.

## 🧪 검증 / 중복
- Verified (spaCy 3.x docs, Anthropic structured output guide, BERTopic, 2024-2026 NLP practice).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — traditional + LLM-based extraction patterns |