2nd/10_Wiki/Topic_Programming/Architecture/Queue-Management-Systems.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-queue-management-systems	Queue Management Systems	10_Wiki/Topics	verified	self	Message Queues Job Queues Task Queues	none	A	0.95	applied	queue messaging async distributed-systems		2026-05-10	pending	language framework python rabbitmq-sqs-celery

Queue Management Systems

매 한 줄

"매 producer 와 consumer 매 decouple — 매 buffered, durable, retried". RabbitMQ / SQS / Redis / Celery / Sidekiq / Temporal — 매 async work 매 backbone. 매 2026 매 Temporal-style durable execution 매 mainstream.

매 핵심

매 queue types

• FIFO point-to-point: 매 single consumer per message (SQS, RabbitMQ direct).
• Pub-sub fanout: 매 N consumers 모두 받음 (SNS, Redis pub/sub, RabbitMQ fanout).
• Priority: 매 weighted consumption (RabbitMQ priority, Redis sorted set).
• Delayed: 매 ETA-based (SQS DelaySeconds, Sidekiq scheduled).
• Dead letter: 매 failed messages → DLQ.

매 delivery guarantees

• At-most-once: ack-on-delivery (Redis pub/sub).
• At-least-once: ack-on-process — 매 default for most prod queues.
• Exactly-once: 매 idempotent consumer + dedup (FIFO SQS, Kafka EOS).

매 patterns

• Work queue: 매 N workers, load-balanced.
• Competing consumers: 매 same.
• Saga / orchestration: 매 multi-step workflow (Temporal, Cadence).
• Outbox: 매 transactional message dispatch.

매 응용

1. Background jobs (email, image resize, PDF gen).
2. Microservice integration (order → fulfillment).
3. Rate limiting / throttling (queue as buffer).
4. Workflow orchestration (Temporal).
5. Batch processing (SQS → Lambda fanout).

💻 패턴

Celery (Python)

from celery import Celery
app = Celery("tasks", broker="redis://localhost:6379/0")

@app.task(bind=True, max_retries=3, retry_backoff=True)
def send_email(self, to: str):
    try:
        smtp.send(to)
    except SMTPException as e:
        raise self.retry(exc=e, countdown=2 ** self.request.retries)

# Producer
send_email.delay("alice@example.com")

RabbitMQ work queue

import pika
conn = pika.BlockingConnection(pika.ConnectionParameters("localhost"))
ch = conn.channel()
ch.queue_declare(queue="tasks", durable=True)

# Publisher
ch.basic_publish(
    exchange="", routing_key="tasks", body=b"work",
    properties=pika.BasicProperties(delivery_mode=2),  # persistent
)

# Worker
ch.basic_qos(prefetch_count=1)
def callback(ch, method, props, body):
    process(body)
    ch.basic_ack(delivery_tag=method.delivery_tag)
ch.basic_consume(queue="tasks", on_message_callback=callback)
ch.start_consuming()

AWS SQS (boto3)

import boto3
sqs = boto3.client("sqs")
url = sqs.get_queue_url(QueueName="jobs")["QueueUrl"]

# Send
sqs.send_message(QueueUrl=url, MessageBody=json.dumps(payload),
                 MessageGroupId="orders", MessageDeduplicationId=order_id)  # FIFO

# Receive (long poll)
resp = sqs.receive_message(QueueUrl=url, WaitTimeSeconds=20, MaxNumberOfMessages=10)
for msg in resp.get("Messages", []):
    process(json.loads(msg["Body"]))
    sqs.delete_message(QueueUrl=url, ReceiptHandle=msg["ReceiptHandle"])

Redis Streams (consumer groups)

import redis
r = redis.Redis()
r.xgroup_create("orders", "fulfillment", id="0", mkstream=True)

# Producer
r.xadd("orders", {"id": "42", "total": "99"})

# Consumer
while True:
    msgs = r.xreadgroup("fulfillment", "worker-1", {"orders": ">"}, count=10, block=5000)
    for stream, entries in msgs or []:
        for mid, data in entries:
            process(data)
            r.xack("orders", "fulfillment", mid)

Temporal workflow (durable execution)

from temporalio import workflow, activity
from datetime import timedelta

@activity.defn
async def charge(card: str, amount: int) -> str:
    return payment_api.charge(card, amount)

@workflow.defn
class OrderWorkflow:
    @workflow.run
    async def run(self, order: dict) -> str:
        tx = await workflow.execute_activity(
            charge, order["card"], order["total"],
            start_to_close_timeout=timedelta(seconds=30),
            retry_policy=RetryPolicy(maximum_attempts=5),
        )
        await workflow.execute_activity(ship, order, ...)
        return tx

Outbox pattern (transactional)

def place_order(db, order):
    with db.transaction():
        db.execute("INSERT INTO orders ...", order)
        db.execute("INSERT INTO outbox (topic, payload) VALUES (?, ?)",
                   "orders.created", json.dumps(order))
    # separate poller relays outbox → broker (at-least-once)

Dead letter queue handling

# RabbitMQ DLX
ch.queue_declare("tasks", arguments={
    "x-dead-letter-exchange": "dlx",
    "x-message-ttl": 60000,
    "x-max-retries": 3,
})
# DLQ consumer logs / alerts / manual replay

매 결정 기준

상황	Choice
매 simple background jobs	Celery / Sidekiq / BullMQ
매 enterprise messaging	RabbitMQ
매 cloud-managed	SQS / Cloud Tasks
매 ordered + dedup	FIFO SQS
매 multi-step workflow	Temporal / Cadence / AWS Step Functions
매 high throughput log	Kafka (technically not a queue)
매 in-process	asyncio.Queue / channels

기본값: 매 2026 매 durable workflow → Temporal. 매 simple jobs → BullMQ / Celery.

🔗 Graph

• 부모: Async Programming
• 변형: Dead Letter Queue
• Adjacent: RabbitMQ · SQS · Temporal · Sidekiq

🤖 LLM 활용

언제: 매 retry policy design, 매 DLQ analysis (cluster failure modes), 매 workflow code scaffold. 언제 X: 매 throughput sizing — load test 직접.

❌ 안티패턴

• No idempotency: 매 at-least-once + non-idempotent → duplicate side effects. 매 dedup key 필수.
• Infinite retry: 매 poison message 매 forever — max attempts + DLQ.
• Unbounded queue: 매 producer faster than consumer — OOM. 매 backpressure / drop oldest.
• Sync wait for queue result: 매 anti-async — 매 callback / webhook / polling.
• Long-running task in queue with short visibility timeout: 매 redelivered while still running — race.

🧪 검증 / 중복

• Verified (RabbitMQ docs; AWS SQS docs; Temporal docs 1.20+).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — full queue systems entry with Temporal