2nd/10_Wiki/Topic_Programming/AI_and_ML/Stem-Analysis.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-stem-analysis	Stem Analysis	10_Wiki/Topics	verified	self	Stemming Word Stemming 형태소 어간추출	none	A	0.9	applied	nlp stemming lemmatization text-preprocessing		2026-05-10	pending	language framework python nltk-spacy-stanza

Stem Analysis

매 한 줄

"매 stem = word 의 morphological root, 매 prefix/suffix 제거". Porter (1980) → Snowball → Lancaster 의 evolution. 2026 LLM era 에선 매 BPE/SentencePiece tokenizer 가 자연스럽게 흡수 — stemming explicit 사용 매 classical IR/sparse search 만.

매 핵심

매 stem vs lemma

• Stem: 매 surface-level chop ("running" → "run", "studies" → "studi"). Heuristic, 매 invalid word OK.
• Lemma: 매 dictionary form ("ran" → "run", "better" → "good"). POS-aware, 매 valid word.
• Tokenization (BPE/WordPiece/SentencePiece): 매 LLM era 의 default — sub-word 의 학습된 segmentation.

매 algorithms

• Porter Stemmer (1980, Martin Porter): 매 5-step rule cascade — English 의 de-facto.
• Snowball / Porter2 (2002): 매 multi-language framework — Porter 의 cleaner re-design.
• Lancaster (Paice/Husk): 매 aggressive — 매 over-stemming 위험.
• Lovins (1968): 매 first published, 매 longest-match 의 single pass.
• Korean (Hannanum/Mecab/Khaiii): 매 morphological analyzer — 매 stemming 보다 morpheme segmentation 이 적합.

매 응용

1. Classical IR / BM25: Elasticsearch / OpenSearch analyzer chain — 매 Snowball default.
2. Sparse retrieval (SPLADE): 매 hybrid 의 sparse leg.
3. Topic modeling (LDA): 매 vocabulary 의 collapse.
4. Feature engineering (legacy ML): TF-IDF + classifier.
5. Domain search 의 keyword expansion: 매 e-commerce, legal, medical.

💻 패턴

Porter / Snowball (NLTK)

from nltk.stem import PorterStemmer, SnowballStemmer

porter = PorterStemmer()
snow = SnowballStemmer("english")

words = ["running", "ran", "runs", "studies", "studying", "easily", "fairly"]
for w in words:
    print(w, "->", porter.stem(w), "/", snow.stem(w))
# running -> run / run
# studies -> studi / studi
# easily -> easili / easili   (heuristic, not a real word)

Lemmatization (spaCy, POS-aware)

import spacy
nlp = spacy.load("en_core_web_lg")  # 2026: spacy 4.x

doc = nlp("The runners ran more easily than the studies suggested.")
for tok in doc:
    print(tok.text, tok.lemma_, tok.pos_)
# runners runner NOUN
# ran     run    VERB
# easily  easily ADV
# studies study  NOUN

Korean morphological analysis (Mecab)

# 매 Korean: stemming X — morpheme segmentation 적합
from konlpy.tag import Mecab
mecab = Mecab()

text = "어제 친구들과 영화관에 갔었습니다"
print(mecab.morphs(text))
# ['어제', '친구', '들', '과', '영화관', '에', '가', '었', '습니다']
print(mecab.pos(text))
# [('어제', 'MAG'), ('친구', 'NNG'), ('들', 'XSN'), ...]

Elasticsearch analyzer chain (2026 OpenSearch 3.x)

PUT /products
{
  "settings": {
    "analysis": {
      "analyzer": {
        "english_stem": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": ["lowercase", "english_stop", "english_stemmer"]
        }
      },
      "filter": {
        "english_stemmer": { "type": "stemmer", "language": "english" }
      }
    }
  }
}

Stemming + BM25 retrieval

from rank_bm25 import BM25Okapi
from nltk.stem import SnowballStemmer
import re

stem = SnowballStemmer("english").stem
def tokenize(text):
    return [stem(t) for t in re.findall(r"\w+", text.lower())]

corpus = ["The cat is running", "Cats run fast", "Dogs bark loudly"]
tokenized = [tokenize(d) for d in corpus]
bm25 = BM25Okapi(tokenized)

query = tokenize("running cats")
print(bm25.get_scores(query))
# Cat/cats + run/running stems collapse → relevant docs score higher

Modern hybrid: BPE token (LLM-era default)

from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3.3-70B-Instruct")

print(tok.tokenize("running runners studies"))
# ['Ġrunning', 'Ġrunners', 'Ġstudies']  — 매 sub-word, 매 stemming 의 implicit
# 매 LLM 은 매 surface form 의 학습 — explicit stemming 의 X 필요.

Over-stemming detection

from nltk.stem import LancasterStemmer
lan = LancasterStemmer()
# 매 Lancaster aggressive 의 example
print(lan.stem("organization"))  # 'org' — 매 too aggressive
print(lan.stem("organic"))       # 'org' — 매 false conflation
# 매 Snowball preferred over Lancaster 의 production.

매 결정 기준

상황	Approach
매 LLM-based search (dense / RAG)	매 stemming X — embedding 이 흡수
BM25 / lexical search	Snowball stemmer 기본
Topic modeling / 매 sparse features	Snowball + 매 stopword
매 한국어 / 일본어 / 중국어	매 morphological analyzer (Mecab, Khaiii)
Aggressive 의 conflation 필요	Lancaster — 매 over-stem 주의

기본값: 매 Snowball (English/multi-lang) + spaCy lemmatizer (POS 가 중요한 경우). 매 dense retrieval 시 stemming X.

🔗 Graph

• 부모: Information Retrieval
• 변형: Tokenization · BPE · SentencePiece
• 응용: BM25 · Elasticsearch · TF-IDF
• Adjacent: Morphological-Analysis

🤖 LLM 활용

언제: 매 legacy IR pipeline, hybrid (sparse+dense) retrieval 의 sparse leg, domain-specific keyword expansion (legal/medical). 언제 X: 매 modern dense retrieval (E5, BGE, Voyage 3), LLM RAG — 매 BPE tokenizer 가 흡수. 매 explicit stemming 의 over-engineering.

❌ 안티패턴

• Stemming dense embeddings 전: 매 embedding model 이 surface form 의 학습 — stemming 이 정보 파괴.
• Lancaster 의 production: 매 over-stemming 의 false-positive 폭증.
• English stemmer 매 한국어 적용: 매 nonsense 결과 — language-specific analyzer 필수.
• Lemma 의 POS X: "saw" (verb→see / noun→saw) 의 ambiguity — POS tag 없이 lemmatize 의 부정확.

🧪 검증 / 중복

• Verified (Porter 1980 paper, Snowball stemmer.tartarus.org, NLTK/spaCy/Stanza 2026 docs).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — stemming algorithms + LLM era 의 위치 정리