---
id: wiki-2026-0508-lstm-long-short-term-memory
title: LSTM (Long Short-Term Memory)
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [LSTM, Long Short-Term Memory Network]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [neural-network, sequence-modeling, rnn, deep-learning]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: pytorch
---

# LSTM (Long Short-Term Memory)

## 매 한 줄
> **"매 gated cell state 로 vanishing gradient 회피"**. Hochreiter & Schmidhuber (1997) — RNN 의 long-range dependency 학습 문제 의 forget/input/output gate 의 cell state 보호. 2026 에서는 Transformer 가 sequence modeling 의 dominant 이지만 LSTM 은 매 small-data, low-latency, streaming, edge inference 에서 여전히 strong choice (Mamba/SSM 의 부상에도 production 의 niche 유지).

## 매 핵심

### 매 RNN 의 한계
- Vanilla RNN: `h_t = tanh(W·[h_{t-1}, x_t])` 의 매 step 마다 gradient multiplication.
- Vanishing gradient: 매 long sequence 에서 `∂L/∂h_0 ≈ 0` — early step 의 학습 불가.
- Exploding gradient: 반대 방향, gradient clipping 으로 mitigate 가능.

### 매 LSTM cell 구조
- **Cell state C_t**: 매 highway — gradient 가 매 multiplicative interaction 없이 flow.
- **Forget gate f_t**: 매 어떤 information 을 cell state 에서 drop 할지 결정.
- **Input gate i_t**: 매 새로운 information 을 cell state 에 add.
- **Output gate o_t**: 매 cell state 의 어떤 부분을 hidden state 로 expose.
- 매 sigmoid gate (0~1) 의 element-wise gating.

### 매 응용 (2026)
1. Time-series forecasting (small/medium scale, transformer overkill 인 경우).
2. Streaming ASR / speech encoder (low-latency causal).
3. Edge device inference (TinyML, mobile keyboard prediction).
4. Anomaly detection on sensor streams.
5. Reinforcement learning policy network (recurrent baselines).

## 💻 패턴

### 1. PyTorch LSTM (basic)
```python
import torch
import torch.nn as nn

class LSTMTagger(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_tags):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, num_tags)

    def forward(self, x):
        # x: [B, T]
        e = self.embed(x)            # [B, T, E]
        out, (h, c) = self.lstm(e)   # out: [B, T, H]
        return self.fc(out)          # [B, T, num_tags]

model = LSTMTagger(vocab_size=10000, embed_dim=128, hidden_dim=256, num_tags=10)
```

### 2. Bidirectional + multi-layer
```python
self.lstm = nn.LSTM(
    input_size=128,
    hidden_size=256,
    num_layers=2,
    bidirectional=True,
    dropout=0.3,
    batch_first=True,
)
# Output: [B, T, 2*256]
```

### 3. Manual LSTM cell (understand internals)
```python
class LSTMCellManual(nn.Module):
    def __init__(self, input_dim, hidden_dim):
        super().__init__()
        self.W = nn.Linear(input_dim + hidden_dim, 4 * hidden_dim)
        self.hidden_dim = hidden_dim

    def forward(self, x, state):
        h, c = state
        z = self.W(torch.cat([x, h], dim=-1))
        i, f, g, o = z.chunk(4, dim=-1)
        i, f, o = torch.sigmoid(i), torch.sigmoid(f), torch.sigmoid(o)
        g = torch.tanh(g)
        c_new = f * c + i * g          # cell state update
        h_new = o * torch.tanh(c_new)  # hidden output
        return h_new, c_new
```

### 4. Streaming inference (stateful)
```python
class StreamingLSTM:
    def __init__(self, model):
        self.model = model
        self.state = None

    def step(self, x_t):
        # x_t: [1, 1, E]
        out, self.state = self.model.lstm(x_t, self.state)
        return self.model.fc(out)

    def reset(self):
        self.state = None
```

### 5. Time-series forecasting
```python
class Forecaster(nn.Module):
    def __init__(self, n_features, hidden=128, horizon=24):
        super().__init__()
        self.lstm = nn.LSTM(n_features, hidden, num_layers=2, batch_first=True)
        self.head = nn.Linear(hidden, horizon)

    def forward(self, x):
        # x: [B, T, F]
        _, (h, _) = self.lstm(x)
        return self.head(h[-1])  # [B, horizon]

# Train with MSE on next-N-step prediction
```

### 6. Gradient clipping (essential for LSTM)
```python
optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
```

### 7. ONNX export for edge
```python
dummy = torch.randn(1, 100, 128)
torch.onnx.export(
    model, dummy, "lstm.onnx",
    input_names=["x"], output_names=["y"],
    dynamic_axes={"x": {0: "batch", 1: "time"}},
    opset_version=17,
)
# Deploy to mobile via ONNX Runtime / Core ML
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Long context (>1k tokens), large data | Transformer (attention). |
| Streaming, causal, low-latency | LSTM 또는 Mamba/SSM. |
| Small data (<10k samples) sequence | LSTM (transformer overfits). |
| Edge / mobile inference | LSTM (small footprint). |
| Multi-modal long-form generation | Transformer / Diffusion. |
| State-space modeling, very long seq | Mamba / S4 (post-2023 alternative). |

**기본값**: 2026 에서 새 sequence task 는 매 Transformer 부터. LSTM 은 매 streaming + edge + small-data 의 specific niche.

## 🔗 Graph
- 부모: [[Sequence Modeling]]
- 변형: [[데이터 사이언스 및 ML 엔지니어링|GRU]]
- 응용: [[Anomaly Detection]]
- Adjacent: [[Transformer]] · [[Transformer_Architecture_and_LLM_Foundations|Attention Mechanism]]

## 🤖 LLM 활용
**언제**: streaming inference design, edge deployment, small-data baseline, recurrent RL agent.
**언제 X**: large-scale language modeling, long-context (>4k) tasks, 매 modern foundation model approach.

## ❌ 안티패턴
- **No gradient clipping**: 매 exploding gradient 의 NaN loss. 매 clip_grad_norm 필수.
- **Wrong batch_first**: PyTorch default 는 `[T, B, H]`. `batch_first=True` 안 쓰면 매 shape 혼란.
- **Frozen state across batches**: 매 stateful LSTM 에서 batch boundary detach 안 하면 BPTT 가 매 entire history 까지 backprop — OOM.
- **LSTM for long context**: 매 >1k token 의 long-range dependency 는 매 attention 으로.
- **No layer norm**: 매 deep LSTM 의 training instability — LayerNorm-LSTM variant 권장.

## 🧪 검증 / 중복
- Verified (Hochreiter & Schmidhuber 1997, *Neural Computation*; PyTorch nn.LSTM docs).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — LSTM cell internals + 2026 niche positioning vs Transformer/Mamba |
