2nd/10_Wiki/Topics/AI_and_ML/Explainable-AI-XAI.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-explainable-ai-xai	Explainable AI (XAI)	10_Wiki/Topics	verified	self	XAI interpretability SHAP LIME attention mechanistic interpretability saliency	none	A	0.95	applied	ai interpretability xai shap lime attention mechanistic transparency		2026-05-10	pending	language framework Python SHAP / LIME / Captum / TransformerLens

Explainable AI (XAI)

매 한 줄

"매 black-box 의 decision 의 understand". 매 SHAP / LIME (post-hoc), 매 attention / saliency, 매 mechanistic interpretability (modern). 매 EU AI Act 의 high-risk 의 require. 매 trade-off: 매 accuracy ↔ interpretability (often false dichotomy).

매 핵심

매 dimension

• Local vs global: 매 single prediction vs overall model.
• Model-specific vs agnostic: 매 internals vs black-box.
• Post-hoc vs intrinsic: 매 after training vs by design.

매 method

• Feature importance: SHAP, LIME, permutation.
• Saliency: Grad-CAM, Integrated Gradients.
• Attention: 매 transformer.
• Counterfactual: 매 minimal change.
• Mechanistic: 매 circuit, SAE, attribution.
• Concept-based: TCAV.

매 modern (mechanistic)

• TransformerLens (Anthropic).
• Sparse Autoencoders (SAE).
• Activation Patching.
• Probing.
• Anthropic Circuits: 매 Towards Monosemanticity.

매 응용

1. Healthcare: 매 diagnostic explain.
2. Finance: 매 credit decision.
3. Justice: 매 risk score.
4. Debugging: 매 model failure.
5. Compliance: 매 EU AI Act.
6. AI safety: 매 alignment audit.

💻 패턴

SHAP (TreeExplainer)

import shap
import xgboost as xgb

model = xgb.XGBClassifier().fit(X, y)
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)
shap.force_plot(explainer.expected_value, shap_values[0], X_test.iloc[0])

LIME

from lime.lime_tabular import LimeTabularExplainer
explainer = LimeTabularExplainer(X_train, feature_names=cols, class_names=['neg', 'pos'])
exp = explainer.explain_instance(X_test[0], model.predict_proba, num_features=10)
exp.show_in_notebook()

Integrated Gradients (Captum)

import captum
from captum.attr import IntegratedGradients

ig = IntegratedGradients(model)
attributions = ig.attribute(input_tensor, target=label_idx, n_steps=50)

Grad-CAM (vision)

from captum.attr import LayerGradCam
import torchvision.models as models

model = models.resnet50(pretrained=True).eval()
gc = LayerGradCam(model, model.layer4[-1])
attribution = gc.attribute(input_image, target=class_idx)

Attention visualization (transformer)

import torch
def attention_rollout(attentions, layer):
    """매 average heads, multiply layers."""
    A = attentions[layer]  # 매 [B, H, T, T]
    A = A.mean(dim=1)  # 매 average heads
    return A

Counterfactual explanation

def counterfactual(model, x, target_class, max_iter=100):
    """매 minimum change to flip prediction."""
    x_cf = x.clone().detach().requires_grad_(True)
    optim = torch.optim.Adam([x_cf], lr=0.01)
    for _ in range(max_iter):
        pred = model(x_cf)
        if pred.argmax() == target_class: return x_cf
        loss = F.cross_entropy(pred, torch.tensor([target_class])) + 0.1 * (x_cf - x).norm()
        optim.zero_grad(); loss.backward(); optim.step()
    return x_cf

Permutation importance

from sklearn.inspection import permutation_importance
result = permutation_importance(model, X_val, y_val, n_repeats=10, random_state=0)
sorted_idx = result.importances_mean.argsort()[::-1]
for i in sorted_idx[:10]:
    print(f'{cols[i]}: {result.importances_mean[i]:.4f}')

TCAV (concept-based)

def tcav_score(model, layer, concept_examples, random_examples, target_class):
    cav = train_cav(layer.activations(concept_examples), layer.activations(random_examples))
    sensitivities = []
    for x in target_class_examples:
        grad = compute_gradient_at_layer(model, x, target_class, layer)
        sensitivities.append((grad @ cav) > 0)
    return np.mean(sensitivities)

Mechanistic — activation patching

import transformer_lens as tl

model = tl.HookedTransformer.from_pretrained('gpt2')
def patched_forward(prompt_clean, prompt_corrupt, layer):
    _, clean_cache = model.run_with_cache(prompt_clean)
    
    def patch_hook(activation, hook):
        return clean_cache[hook.name]
    
    return model.run_with_hooks(prompt_corrupt, fwd_hooks=[(f'blocks.{layer}.attn.hook_z', patch_hook)])

Sparse Autoencoder (SAE)

class SAE(nn.Module):
    def __init__(self, d_model, d_sae=32768, l1_coef=1e-3):
        super().__init__()
        self.W_enc = nn.Linear(d_model, d_sae)
        self.W_dec = nn.Linear(d_sae, d_model, bias=False)
        self.l1_coef = l1_coef
    
    def forward(self, x):
        z = F.relu(self.W_enc(x))
        x_recon = self.W_dec(z)
        recon_loss = F.mse_loss(x_recon, x)
        sparsity_loss = self.l1_coef * z.abs().sum(-1).mean()
        return x_recon, recon_loss + sparsity_loss

Probing classifier

def probe(activations, labels, layer):
    X = activations[layer].detach().numpy()
    clf = LogisticRegression(max_iter=1000).fit(X, labels)
    return clf.score(X_val, y_val)

Model card explainability

explainability:
  method: SHAP TreeExplainer
  global_top_features:
    - credit_score: 0.34
    - debt_to_income: 0.18
    - employment_years: 0.12
  per_decision_explanation: available_in_ui
  counterfactual_offered: yes

LLM explanation prompt

def llm_explain_decision(prediction, features):
    prompt = f"""You are explaining an ML decision. Use ONLY the features given.
Prediction: {prediction}
Features: {features}

Output:
1. Top 3 reasons
2. What change would flip the decision
3. Limitations of this explanation"""
    return llm.generate(prompt)

매 결정 기준

상황	Method
Tabular ML	SHAP TreeExplainer
Black-box agnostic	LIME / SHAP KernelExplainer
Vision	Grad-CAM / Integrated Gradients
NLP	Attention + token attribution
LLM internals	TransformerLens / SAE
User-facing	Counterfactual + plain language
Compliance	Model card + global + local

기본값: 매 SHAP global + local + 매 counterfactual + 매 model card. 매 LLM internals = SAE + activation patching.

🔗 Graph

• 부모: AI · Interpretability
• 변형: SHAP · LIME · Mechanistic-Interpretability
• 응용: Model-Card · EU-AI-Act · AI Safety
• Adjacent: Sparse-Autoencoder · Ethics & AI

🤖 LLM 활용

언제: 매 high-risk EU. 매 healthcare / finance. 매 model debug. 언제 X: 매 low-stakes. 매 explanation 의 misleading 의 risk.

❌ 안티패턴

• Single number trust: 매 SHAP 의 misuse.
• Saliency as causal: 매 correlation only.
• Attention = explanation: 매 not always (Jain & Wallace 2019).
• Post-hoc only: 매 design 의 ignore.
• Explanation 의 misleading: 매 false confidence.

🧪 검증 / 중복

• Verified (Lundberg SHAP 2017, Ribeiro LIME 2016, Olah Anthropic 2024).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — XAI methods + 매 SHAP / LIME / IG / counterfactual / SAE / probing code