2nd/10_Wiki/Topics/AI_and_ML/Inductive-Bias.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-inductive-bias	Inductive Bias	10_Wiki/Topics	verified	self	inductive bias prior model assumption no free lunch	none	A	0.94	applied	machine-learning inductive-bias prior no-free-lunch deep-learning		2026-05-10	pending	language applicable_to ML Theory Model Design Architecture

Inductive Bias

매 한 줄

"매 model 의 의 의 의 의 assumption 의 의 의 generalize". 매 No Free Lunch theorem (Wolpert): 매 매 algorithm 의 model 의 의 의 매 task 의 best. 매 modern: 매 transformer 의 inductive bias 의 의 의 weak — 매 scale 의 의 의 의 compensate.

매 핵심

매 examples

• CNN: 매 translation invariance, locality.
• RNN: 매 sequence, temporal.
• Transformer: 매 weak — 매 매 sufficient data 의 의 의 learn.
• GNN: 매 graph structure, permutation invariance.
• Tree: 매 axis-aligned splits.
• kNN: 매 local smoothness.

매 trade-off

• Strong bias: 매 small data 의 효율, 매 wrong domain 매 fail.
• Weak bias (transformer): 매 large data 매 win, 매 small data 매 lose.

매 No Free Lunch

• 매 average performance over 매 모든 problems 의 same.
• 매 매 task 의 의 의 의 best algorithm.
• 매 → 매 inductive bias 매 critical.

매 응용

1. Architecture choice.
2. Transfer learning.
3. Few-shot learning.
4. Domain adaptation.

💻 패턴

CNN inductive bias (translation)

import torch.nn as nn
# 매 same kernel 의 매 location 의 적용
conv = nn.Conv2d(3, 64, kernel_size=3, padding=1)
# 매 translation invariance + locality

Transformer (weak bias)

# 매 매 token pair 의 attention — 매 specific structure 의 X
# 매 → 매 large pretrain data 의 의 의 compensate
class Attention(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.q = nn.Linear(dim, dim)
        self.k = nn.Linear(dim, dim)
        self.v = nn.Linear(dim, dim)
    
    def forward(self, x):
        return torch.softmax(self.q(x) @ self.k(x).T, -1) @ self.v(x)

GNN (graph + permutation invariance)

# 매 message passing — 매 node order independent
class GNNLayer:
    def forward(self, x, edge_index):
        # 매 매 node 의 매 neighbor 의 aggregate (sum/mean)
        return aggregate_neighbors(x, edge_index)

Strong vs weak bias (data regime)

def model_for_data_size(n):
    if n < 1000:
        return 'tree-based'  # 매 strong inductive bias
    if n < 100000:
        return 'classical ML or small CNN'
    if n < 10_000_000:
        return 'CNN / RNN'
    return 'transformer'  # 매 매 weak bias 의 의 의 의 의 의 win

Transfer learning (use pretrained bias)

from torchvision.models import resnet50
model = resnet50(pretrained=True)
# 매 freeze backbone (use ImageNet bias)
for p in model.parameters(): p.requires_grad = False
# 매 fine-tune classifier
model.fc = nn.Linear(2048, n_classes)

Encode bias as constraint

# 매 monotonic prediction (e.g., income ↑ → spend ↑)
class MonotonicNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.weights = nn.Parameter(torch.randn(...))
    
    def forward(self, x):
        # 매 abs of weights → 매 monotonic
        return x @ torch.abs(self.weights)

Symmetry-aware network

# 매 E(n)-equivariant for molecule
class EGNN(nn.Module):
    def forward(self, x, pos, edge_index):
        # 매 rotation / translation invariance built in
        ...

Probing inductive bias

def small_data_eval(model, train_sizes=[100, 1000, 10000]):
    """매 small data 의 의 의 model 의 generalize?"""
    perf = {}
    for n in train_sizes:
        m = train(model, X_train[:n], y_train[:n])
        perf[n] = evaluate(m, X_test, y_test)
    return perf  # 매 strong bias model 매 small n 의 win

Architecture search (inductive bias picker)

def pick_architecture(data_features):
    if data_features['has_spatial_structure']: return 'CNN'
    if data_features['is_graph']: return 'GNN'
    if data_features['is_sequence_long']: return 'Transformer'
    if data_features['is_tabular']: return 'XGBoost'

Synthetic + real (mix biases)

def hybrid_train(synthetic_data, real_data):
    """매 synthetic data 매 strong bias, 매 real data 매 fine-tune."""
    pretrain(model, synthetic_data)
    finetune(model, real_data)

매 결정 기준

상황	Bias
Small data	Strong (tree, kNN)
Image	CNN (translation)
Sequence	RNN / Transformer
Graph	GNN
Tabular	Tree
Big data + flexible	Transformer
Symmetry-aware	Equivariant NN

기본값: 매 task structure 의 match 의 inductive bias + 매 large data 의 의 의 weak bias 의 OK + 매 small data 의 의 strong bias 의 critical.

🔗 Graph

• 변형: No-Free-Lunch
• Adjacent: Generalization-in-AI · Foundation-Models

🤖 LLM 활용

언제: 매 model design. 매 small data. 매 transfer. 언제 X: 매 toy.

❌ 안티패턴

• Wrong bias for task: 매 image 매 dense MLP.
• Always weak bias: 매 small data 의 fail.
• Ignore symmetry: 매 sample efficiency lose.
• No transfer for small data: 매 reinvent.

🧪 검증 / 중복

• Verified (Wolpert NFL, Mitchell ML).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — bias + 매 CNN/Transformer/GNN / NFL / arch picker code