2nd/10_Wiki/Topics/Domain_Programming/AI_and_ML/Text-Mining.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-text-mining	Text Mining	10_Wiki/Topics	verified	self	Text Analytics Information Extraction	none	A	0.9	applied	nlp text-mining information-extraction		2026-05-10	pending	language framework Python spaCy / LLM-based

Text Mining

매 한 줄

"매 unstructured text → structured signal". Text mining 매 large text corpora 에서 patterns / entities / relationships / sentiment 의 extract 하는 분야. 매 traditional (TF-IDF, NER models) 에서 매 LLM-based extraction (structured output, function calling) 으로 매 paradigm shift.

매 핵심

매 traditional pipeline

• Tokenization → POS tagging → NER → dependency parsing → sentiment → topic modeling.
• Tools: spaCy, NLTK, scikit-learn (TF-IDF + classifiers), gensim (LDA).
• 매 production NER: spaCy transformers pipeline or fine-tuned BERT.

매 modern (LLM-based)

• Structured output — 매 LLM 이 JSON schema 의 fill (Claude tool use, OpenAI structured output, Outlines).
• Few-shot extraction — 매 fine-tune 없이 매 5 examples 만으로 task 의 정의.
• Long-context — 매 200k+ token document 의 single-shot processing.
• 매 cost trade-off: spaCy NER ~$0.0001/doc vs LLM ~$0.01/doc — 매 batch + small model (Haiku, gpt-4o-mini) 으로 reduce.

매 응용

1. Resume parsing, contract analysis (entity + clause extraction).
2. Customer feedback aggregation (sentiment + topic).
3. Biomedical literature mining (gene/protein/disease NER).

💻 패턴

spaCy NER (traditional)

import spacy
nlp = spacy.load("en_core_web_trf")
doc = nlp("Apple acquired Anthropic for $50B in March 2025.")
for ent in doc.ents:
    print(ent.text, ent.label_)
# Apple ORG, Anthropic ORG, $50B MONEY, March 2025 DATE

TF-IDF + classifier

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

pipe = Pipeline([
    ("tfidf", TfidfVectorizer(ngram_range=(1, 2), max_features=20000)),
    ("clf", LogisticRegression(max_iter=1000)),
])
pipe.fit(X_train, y_train)

LLM structured extraction (Claude)

from anthropic import Anthropic
from pydantic import BaseModel

class Contract(BaseModel):
    parties: list[str]
    effective_date: str
    total_value_usd: float | None
    governing_law: str | None

client = Anthropic()
resp = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=2000,
    tools=[{
        "name": "extract_contract",
        "input_schema": Contract.model_json_schema(),
    }],
    tool_choice={"type": "tool", "name": "extract_contract"},
    messages=[{"role": "user", "content": contract_text}],
)
data = Contract(**resp.content[0].input)

Topic modeling (BERTopic)

from bertopic import BERTopic
from sentence_transformers import SentenceTransformer

embed = SentenceTransformer("BAAI/bge-large-en-v1.5")
topic_model = BERTopic(embedding_model=embed, min_topic_size=10)
topics, probs = topic_model.fit_transform(docs)
topic_model.get_topic_info()

Long-context document QA

# 200k token contract → single LLM call
resp = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=4000,
    messages=[{"role": "user", "content": [
        {"type": "text", "text": contract_full_text,
         "cache_control": {"type": "ephemeral"}},
        {"type": "text", "text": "Extract all change-of-control provisions."},
    ]}],
)

Hybrid (LLM + regex precheck)

import re
DATE_RE = re.compile(r"\b\d{4}-\d{2}-\d{2}\b")
candidates = DATE_RE.findall(text)
# 매 LLM 의 candidate 만 disambiguate — 매 cost reduce

매 결정 기준

상황	Approach
High-volume (M docs/day) NER	spaCy / fine-tuned BERT
Complex schema, low volume	LLM structured output
Topic discovery	BERTopic / embeddings + clustering
Sentiment	Fine-tuned RoBERTa or LLM
Long documents (>50k tokens)	LLM with caching
Domain-specific (legal, medical)	Fine-tune + LLM hybrid

기본값: 매 prototype LLM, 매 production 은 LLM (low volume) or distilled fine-tuned model (high volume).

🔗 Graph

• 부모: Information Retrieval
• 변형: Named-Entity-Recognition · Sentiment-Analysis
• 응용: RAG · Search
• Adjacent: Embeddings

🤖 LLM 활용

언제: 매 unstructured text corpus 의 query / extract / classify, schema-driven extraction, low-to-medium volume. 언제 X: 매 milli-second latency 의 필요 (real-time chat moderation) — 매 small distilled model.

❌ 안티패턴

• Regex-only complex extraction: 매 brittle — 매 LLM hybrid 로 graceful.
• No evaluation set: 매 LLM 매 hallucinate — 매 ground-truth eval 의 maintain.
• Full-document LLM 의 every query: 매 cache or pre-extract structured DB.
• Unicode normalization 의 skip: 매 Korean/CJK text 매 NFC normalize 필수.

🧪 검증 / 중복

• Verified (spaCy 3.x docs, Anthropic structured output guide, BERTopic, 2024-2026 NLP practice).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — traditional + LLM-based extraction patterns