2nd/10_Wiki/Topics/Domain_Programming/Coding/Messaging_Kafka_Patterns.md

id, title, category, status, source_trust_level, verification_status, created_at, updated_at, tags, tech_stack, applied_in, aliases

id	title	category	status	source_trust_level	verification_status	created_at	updated_at	tags	tech_stack	applied_in	aliases
messaging-kafka-patterns	Kafka — Topic / Partition / Consumer Group	Coding	draft	B	conceptual	2026-05-09	2026-05-09	messaging kafka streaming vibe-coding	language applicable_to TS / Kafka Backend		Kafka partitioning consumer group offsets exactly-once Confluent

Kafka

이벤트 stream 의 표준. Topic = 큐, Partition = 병렬 단위, Consumer group = 분산 처리. Ordering 은 partition 안만. exactly-once 는 idempotent producer + transactional.

📖 핵심 개념

• Topic: 분류된 메시지 stream.
• Partition: topic 내 분산 unit. ordering 단위.
• Consumer group: 같은 group 안 partition 분배.
• Offset: consumer 가 어디까지 읽었는지.
• Retention: 시간/크기 기반 (vs 일반 큐는 ack 후 삭제).

💻 코드 패턴

Producer (KafkaJS)

import { Kafka, CompressionTypes } from 'kafkajs';

const kafka = new Kafka({ clientId: 'api', brokers: ['kafka:9092'] });
const producer = kafka.producer({
  idempotent: true, // Exactly-once 첫 step
  maxInFlightRequests: 5,
});
await producer.connect();

await producer.send({
  topic: 'orders',
  compression: CompressionTypes.GZIP,
  messages: [
    {
      key: order.userId, // partition key — 같은 user = 같은 partition (ordering)
      value: JSON.stringify(order),
      headers: { 'x-event-id': eventId, 'content-type': 'application/json' },
    },
  ],
});

Consumer

const consumer = kafka.consumer({ groupId: 'order-projector' });
await consumer.connect();
await consumer.subscribe({ topic: 'orders', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const body = JSON.parse(message.value!.toString());
    const eventId = message.headers!['x-event-id']!.toString();

    // Idempotency
    if (await db.processed.exists(eventId)) return;

    await handleOrder(body);
    await db.processed.insert({ id: eventId });
    // offset 자동 commit (auto)
  },
});

Manual offset commit (exactly-once style)

await consumer.run({
  autoCommit: false,
  eachBatch: async ({ batch, resolveOffset, heartbeat, commitOffsetsIfNecessary }) => {
    for (const msg of batch.messages) {
      await db.transaction(async (tx) => {
        await handleOrderInTx(tx, JSON.parse(msg.value!.toString()));
        // tx commit + offset commit 분리 — outbox/idempotency 로 보완
      });
      resolveOffset(msg.offset);
      await heartbeat();
    }
    await commitOffsetsIfNecessary();
  },
});

Topic 설정

kafka-topics --create --topic orders --partitions 12 --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=604800000 \   # 7일
  --config compression.type=gzip

Schema Registry + Avro/Protobuf

import { SchemaRegistry } from '@kafkajs/confluent-schema-registry';

const registry = new SchemaRegistry({ host: 'http://schema-registry:8081' });
const id = await registry.getRegistryIdBySubject({ subject: 'orders-value' });

await producer.send({
  topic: 'orders',
  messages: [{ value: await registry.encode(id, order) }],
});

호환성 정책: BACKWARD (구 consumer 가 새 메시지 OK).

Dead-letter topic

async function handleWithDLQ(msg: KafkaMessage) {
  try {
    await handle(msg);
  } catch (e) {
    await producer.send({
      topic: 'orders.dlq',
      messages: [{ key: msg.key, value: msg.value, headers: { ...msg.headers, 'x-error': String(e) } }],
    });
  }
}

Compaction (key 별 최신만 보존)

--config cleanup.policy=compact

key 별로 latest value 만 — user_state 같은 use case.

Streams (Kafka Streams 또는 Faust / KSQL)

• Topic → Topic 변환 / 집계.
• Stateful (window aggregations).
• Java/Scala 가 1급, TS 는 제한적.

🤔 의사결정 기준

상황	추천
큰 throughput + 영속	Kafka
단순 큐	RabbitMQ / SQS / NATS
작은 팀	NATS JetStream (가벼움)
Self-host 어려움	Confluent Cloud / AWS MSK / Redpanda
Order 강 보장	partition key
Replay 필요	Kafka 자연 (retention)
Schema 진화	Schema Registry

❌ 안티패턴

• Partition 없이 1개: 병렬 X.
• 너무 많은 partition (1000+): open file 폭발.
• Key 무작위: ordering 깨짐.
• autoCommit 만 + 처리 실패: offset commit 됐는데 처리 안 됨.
• Idempotency 없는 consumer: at-least-once 가 중복.
• Replication factor 1: 노드 죽으면 데이터 잃음.
• min.insync.replicas 없음: split-brain 시 데이터 손실.
• Long processing in eachMessage: heartbeat 끊겨 rebalance.

🤖 LLM 활용 힌트

• Partition key = ordering 단위.
• Idempotency 헤더 + dedupe table.
• Schema registry + DLQ 표준.

🔗 관련 문서

• Messaging_NATS_RabbitMQ_Comparison
• Messaging_Exactly_Once