2nd/10_Wiki/Topics/Architecture/Message-Queues-and-Event-Streams.md

id, title, category, status, canonical_id, aliases, duplicate_of, source_trust_level, confidence_score, verification_status, tags, raw_sources, last_reinforced, github_commit, tech_stack

id	title	category	status	canonical_id	aliases	duplicate_of	source_trust_level	confidence_score	verification_status	tags	raw_sources	last_reinforced	github_commit	tech_stack
wiki-2026-0508-message-queues-and-event-streams	Message Queues and Event Streams	10_Wiki/Topics	verified	self	Queue vs Stream MQ vs Streaming	none	A	0.9	applied	messaging queues streams kafka sqs		2026-05-10	pending	language framework multi kafka-sqs-rabbitmq

Message Queues and Event Streams

매 한 줄

"매 Queue 는 work 를 distribute, 매 Stream 은 history 를 record.". Queue 와 stream 은 매 둘 다 producer-consumer 를 decouple 하지만 매 mental model 이 다르다 — 매 queue 는 "처리할 일" (소비 후 사라짐), 매 stream 은 "발생한 사건" (immutable log, 매 replay 가능). 매 modern system 은 둘을 매 함께 사용.

매 핵심

매 Queue 의 본질

• Consume = delete (또는 ack 후 hidden).
• 매 1 message → 매 1 worker (work distribution).
• 매 short retention (ack 후 사라짐).
• 매 Examples: SQS, RabbitMQ classic, Redis list (BRPOP).

매 Stream 의 본질

• Append-only immutable log.
• 매 모든 consumer 가 매 모든 event 읽음 (각자 offset).
• 매 long retention (daysweeksforever).
• 매 replay 가능.
• 매 Examples: Kafka, Kinesis, Pulsar, Redis Streams.

매 비교

측면	Queue	Stream
Consume	destructive	non-destructive
Retention	짧음 (ack 후)	긺 (time/size 기반)
Replay	X	O
Ordering	약함 (per-queue)	강함 (per-partition)
Throughput	중	매 high (millions/sec)
Use case	task distribution	event sourcing, analytics, ML

매 함께 쓰기

• 매 Stream → consumer 가 매 work item 추출 → 매 Queue 에 push (workflow orchestration).
• 매 Queue 에서 매 ack 후 매 audit event 를 매 stream 에 publish.

💻 패턴

Queue: SQS worker (Node.js)

import { SQSClient, ReceiveMessageCommand, DeleteMessageCommand } from '@aws-sdk/client-sqs';

const sqs = new SQSClient({});
const QueueUrl = process.env.QUEUE_URL;

while (true) {
  const { Messages } = await sqs.send(new ReceiveMessageCommand({
    QueueUrl,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20,  // long polling
    VisibilityTimeout: 60,
  }));
  if (!Messages) continue;

  for (const msg of Messages) {
    try {
      await processJob(JSON.parse(msg.Body));
      await sqs.send(new DeleteMessageCommand({ QueueUrl, ReceiptHandle: msg.ReceiptHandle }));
    } catch (e) {
      // Don't delete → message becomes visible again after VisibilityTimeout → retry → DLQ after maxReceiveCount
    }
  }
}

Stream: Kafka consumer with offset (Node.js)

import { Kafka } from 'kafkajs';

const consumer = new Kafka({ brokers: ['kafka:9092'] }).consumer({
  groupId: 'analytics-v2',
});
await consumer.subscribe({ topic: 'user-events', fromBeginning: true });
// fromBeginning: true → re-read entire history (replay)

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value.toString());
    await projectIntoDB(event);
    // Offset auto-committed (or manual via heartbeat)
  },
});

Stream: replay from specific offset

// Reset consumer group to specific offset (admin operation)
import { Kafka } from 'kafkajs';
const admin = new Kafka({ brokers: ['kafka:9092'] }).admin();
await admin.connect();
await admin.resetOffsets({
  groupId: 'analytics-v2',
  topic: 'user-events',
  earliest: true,  // or specific offset / timestamp
});
// Next consumer.run() will re-read from beginning

Redis Streams (XADD / XREADGROUP)

import Redis from 'ioredis';
const redis = new Redis();

// Producer
await redis.xadd('events', '*', 'type', 'order.created', 'data', JSON.stringify(order));

// Consumer group
await redis.xgroup('CREATE', 'events', 'workers', '$', 'MKSTREAM').catch(() => {});

// Consume
while (true) {
  const res = await redis.xreadgroup(
    'GROUP', 'workers', 'worker-1',
    'COUNT', 10, 'BLOCK', 5000,
    'STREAMS', 'events', '>'
  );
  if (!res) continue;
  for (const [, entries] of res) {
    for (const [id, fields] of entries) {
      await process(fields);
      await redis.xack('events', 'workers', id);
    }
  }
}

Outbox → Stream (CDC pattern)

# Debezium reads DB WAL → publishes to Kafka topic
# → other services subscribe to data changes without polling
connector: debezium-postgres
config:
  database.hostname: db
  database.dbname: orders
  table.include.list: public.orders, public.outbox
  topic.prefix: cdc
# → cdc.public.orders, cdc.public.outbox topics created

Choice: 매 task vs event

Q1: "결과를 매 누가 처리하면 끝나나?" → Queue (1 worker)
Q2: "여러 service 가 매 동일 event 에 react?" → Stream (fan-out)
Q3: "역사를 replay 해야?" → Stream
Q4: "Order 가 strict (per-key)?" → Stream (partition by key)
Q5: "단순 background job?" → Queue

매 결정 기준

요구	선택
Background job (이메일, 썸네일)	Queue (SQS, BullMQ)
Event sourcing / CDC	Stream (Kafka)
Real-time analytics	Stream (Kafka, Kinesis)
매 strict ordering per user	Stream + partition key
Pub/Sub broadcast	Stream 또는 Pub/Sub
Workflow orchestration	Queue + worker (Temporal, Step Functions)
매 audit log	Stream (immutable retention)

기본값: 매 task 면 queue, 매 event 면 stream. 매 둘 다 필요하면 매 함께 사용.

🔗 Graph

• 부모: Message Broker · Distributed Systems
• 변형: Kafka · SQS
• 응용: Event Sourcing · CQRS · CDC
• Adjacent: Dead Letter Queue

🤖 LLM 활용

언제: messaging architecture 결정, queue vs stream 선택, replay 요구사항 평가. 언제 X: 매 simple sync RPC 로 충분한 매 internal call.

❌ 안티패턴

• Stream 을 queue 로: 매 short retention 만 쓰면 매 stream 의 강점 (replay) 못 살림.
• Queue 로 broadcast: 매 fan-out 에 queue → 매 consumer 마다 별도 queue 만들어야 → 매 stream 이 정답.
• Partition key 무시: 매 stream 에서 매 ordering 필요한데 매 random key → 매 race condition.
• Retention infinity: 매 cost 폭주 — 매 compaction / time-based 설정.

🧪 검증 / 중복

• Verified (Kafka docs, AWS SQS/Kinesis docs, Confluent blog, Martin Kleppmann "DDIA" Ch 11).
• 신뢰도 A.

🕓 Changelog

날짜	변경
2026-05-08	Phase 1
2026-05-10	Manual cleanup — queue vs stream 비교 + SQS/Kafka/Redis Streams 패턴