2nd/10_Wiki/Topics/Architecture/Queue-Management-Systems.md

---
id: wiki-2026-0508-queue-management-systems
title: Queue Management Systems
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Message Queues, Job Queues, Task Queues]
duplicate_of: none
source_trust_level: A
confidence_score: 0.95
verification_status: applied
tags: [queue, messaging, async, distributed-systems]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: rabbitmq-sqs-celery
---

# Queue Management Systems

## 매 한 줄
> **"매 producer 와 consumer 매 decouple — 매 buffered, durable, retried"**. RabbitMQ / SQS / Redis / Celery / Sidekiq / Temporal — 매 async work 매 backbone. 매 2026 매 Temporal-style durable execution 매 mainstream.

## 매 핵심

### 매 queue types
- **FIFO point-to-point**: 매 single consumer per message (SQS, RabbitMQ direct).
- **Pub-sub fanout**: 매 N consumers 모두 받음 (SNS, Redis pub/sub, RabbitMQ fanout).
- **Priority**: 매 weighted consumption (RabbitMQ priority, Redis sorted set).
- **Delayed**: 매 ETA-based (SQS DelaySeconds, Sidekiq scheduled).
- **Dead letter**: 매 failed messages → DLQ.

### 매 delivery guarantees
- **At-most-once**: ack-on-delivery (Redis pub/sub).
- **At-least-once**: ack-on-process — 매 default for most prod queues.
- **Exactly-once**: 매 idempotent consumer + dedup (FIFO SQS, Kafka EOS).

### 매 patterns
- **Work queue**: 매 N workers, load-balanced.
- **Competing consumers**: 매 same.
- **Saga / orchestration**: 매 multi-step workflow (Temporal, Cadence).
- **Outbox**: 매 transactional message dispatch.

### 매 응용
1. Background jobs (email, image resize, PDF gen).
2. Microservice integration (order → fulfillment).
3. Rate limiting / throttling (queue as buffer).
4. Workflow orchestration (Temporal).
5. Batch processing (SQS → Lambda fanout).

## 💻 패턴

### Celery (Python)
```python
from celery import Celery
app = Celery("tasks", broker="redis://localhost:6379/0")

@app.task(bind=True, max_retries=3, retry_backoff=True)
def send_email(self, to: str):
    try:
        smtp.send(to)
    except SMTPException as e:
        raise self.retry(exc=e, countdown=2 ** self.request.retries)

# Producer
send_email.delay("alice@example.com")
```

### RabbitMQ work queue
```python
import pika
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.queue_declare(queue="tasks", durable=True)

# Publisher
ch.basic_publish(
    exchange="", routing_key="tasks", body=b"work",
    properties=pika.BasicProperties(delivery_mode=2),  # persistent
)

# Worker
ch.basic_qos(prefetch_count=1)
def callback(ch, method, props, body):
    process(body)
    ch.basic_ack(delivery_tag=method.delivery_tag)
ch.basic_consume(queue="tasks", on_message_callback=callback)
ch.start_consuming()
```

### AWS SQS (boto3)
```python
import boto3
sqs = boto3.client("sqs")
url = sqs.get_queue_url(QueueName="jobs")["QueueUrl"]

# Send
sqs.send_message(QueueUrl=url, MessageBody=json.dumps(payload),
                 MessageGroupId="orders", MessageDeduplicationId=order_id)  # FIFO

# Receive (long poll)
resp = sqs.receive_message(QueueUrl=url, WaitTimeSeconds=20, MaxNumberOfMessages=10)
for msg in resp.get("Messages", []):
    process(json.loads(msg["Body"]))
    sqs.delete_message(QueueUrl=url, ReceiptHandle=msg["ReceiptHandle"])
```

### Redis Streams (consumer groups)
```python
import redis
r = redis.Redis()
r.xgroup_create("orders", "fulfillment", id="0", mkstream=True)

# Producer
r.xadd("orders", {"id": "42", "total": "99"})

# Consumer
while True:
    msgs = r.xreadgroup("fulfillment", "worker-1", {"orders": ">"}, count=10, block=5000)
    for stream, entries in msgs or []:
        for mid, data in entries:
            process(data)
            r.xack("orders", "fulfillment", mid)
```

### Temporal workflow (durable execution)
```python
from temporalio import workflow, activity
from datetime import timedelta

@activity.defn
async def charge(card: str, amount: int) -> str:
    return payment_api.charge(card, amount)

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order: dict) -> str:
        tx = await workflow.execute_activity(
            charge, order["card"], order["total"],
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=5),
        )
        await workflow.execute_activity(ship, order, ...)
        return tx
```

### Outbox pattern (transactional)
```python
def place_order(db, order):
    with db.transaction():
        db.execute("INSERT INTO orders ...", order)
        db.execute("INSERT INTO outbox (topic, payload) VALUES (?, ?)",
                   "orders.created", json.dumps(order))
    # separate poller relays outbox → broker (at-least-once)
```

### Dead letter queue handling
```python
# RabbitMQ DLX
ch.queue_declare("tasks", arguments={
    "x-dead-letter-exchange": "dlx",
    "x-message-ttl": 60000,
    "x-max-retries": 3,
})
# DLQ consumer logs / alerts / manual replay
```

## 매 결정 기준
| 상황 | Choice |
|---|---|
| 매 simple background jobs | Celery / Sidekiq / BullMQ |
| 매 enterprise messaging | RabbitMQ |
| 매 cloud-managed | SQS / Cloud Tasks |
| 매 ordered + dedup | FIFO SQS |
| 매 multi-step workflow | Temporal / Cadence / AWS Step Functions |
| 매 high throughput log | Kafka (technically not a queue) |
| 매 in-process | asyncio.Queue / channels |

**기본값**: 매 2026 매 durable workflow → Temporal. 매 simple jobs → BullMQ / Celery.

## 🔗 Graph
- 부모: [[Async Programming]]
- 변형: [[Dead Letter Queue]]
- Adjacent: [[RabbitMQ]] · [[SQS]] · [[Temporal]] · [[Sidekiq]]

## 🤖 LLM 활용
**언제**: 매 retry policy design, 매 DLQ analysis (cluster failure modes), 매 workflow code scaffold.
**언제 X**: 매 throughput sizing — load test 직접.

## ❌ 안티패턴
- **No idempotency**: 매 at-least-once + non-idempotent → duplicate side effects. 매 dedup key 필수.
- **Infinite retry**: 매 poison message 매 forever — max attempts + DLQ.
- **Unbounded queue**: 매 producer faster than consumer — OOM. 매 backpressure / drop oldest.
- **Sync wait for queue result**: 매 anti-async — 매 callback / webhook / polling.
- **Long-running task in queue with short visibility timeout**: 매 redelivered while still running — race.

## 🧪 검증 / 중복
- Verified (RabbitMQ docs; AWS SQS docs; Temporal docs 1.20+).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — full queue systems entry with Temporal |