---
id: wiki-2026-0508-datacollector-engine-stalled-loop-guard
title: Datacollector Engine Processed Count and Stalled Loop Guard
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Datacollector Stalled Loop, Engine Processed Counter Guard]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [skybound, datacollector, engine, watchdog, observability]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: typescript
  framework: node
---

# Datacollector Engine Processed Count and Stalled Loop Guard

## 매 한 줄
> **"매 long-running engine loop 의 stalled 상태 detection — processed counter delta + heartbeat watchdog"**. Skybound datacollector 의 processed 카운터 정체 issue 가 silent failure 로 이어졌고, watchdog 패턴 + structured heartbeat metric 으로 fix. 매 production-grade ingestion service 의 default pattern.

## 매 핵심

### 매 문제
- Engine main loop 이 exception 없이 중단 (외부 API stall, lock contention, GC pause).
- `processed_count` metric 이 freeze 됨에도 process alive → health-check pass.
- 결과: hours-long silent stall, downstream backlog explosion.

### 매 해결 axes
- **Counter delta watchdog**: N seconds 마다 delta == 0 면 alarm.
- **Heartbeat token**: loop iteration 마다 last_tick = now() update.
- **Bounded queue + timeout**: 매 await 에 timeout, fail-fast.
- **Structured log per iteration**: tracing 의 base.

### 매 응용
1. Skybound telemetry ingestion engine.
2. Game session replay processor.
3. Generic ETL pipeline worker.

## 💻 패턴

### Counter delta watchdog
```typescript
class StalledLoopGuard {
  private lastProcessed = 0;
  private lastChange = Date.now();
  constructor(
    private getCount: () => number,
    private maxStallMs = 30_000,
    private onStall: (info: { sinceMs: number; count: number }) => void,
  ) {}
  tick() {
    const cur = this.getCount();
    if (cur !== this.lastProcessed) {
      this.lastProcessed = cur;
      this.lastChange = Date.now();
      return;
    }
    const sinceMs = Date.now() - this.lastChange;
    if (sinceMs > this.maxStallMs) {
      this.onStall({ sinceMs, count: cur });
    }
  }
}
```

### Heartbeat token in main loop
```typescript
let lastTick = Date.now();
const heartbeat = setInterval(() => {
  if (Date.now() - lastTick > 15_000) {
    logger.error({ stalledMs: Date.now() - lastTick }, "engine loop stalled");
    metrics.increment("engine.stall");
  }
}, 5_000);

while (running) {
  lastTick = Date.now();
  const batch = await source.next({ timeoutMs: 10_000 });
  if (!batch.length) { await sleep(250); continue; }
  await Promise.all(batch.map(processOne));
  processedCount += batch.length;
  metrics.gauge("engine.processed", processedCount);
}
```

### Awaited timeout wrapper
```typescript
async function withTimeout<T>(p: Promise<T>, ms: number, label: string): Promise<T> {
  return await Promise.race([
    p,
    new Promise<T>((_, rej) => setTimeout(() => rej(new Error(`timeout:${label}`)), ms)),
  ]);
}
```

### Stall-aware health endpoint
```typescript
app.get("/healthz", (_req, res) => {
  const stalledMs = Date.now() - lastTick;
  if (stalledMs > 30_000) return res.status(503).json({ ok: false, stalledMs });
  res.json({ ok: true, processedCount, stalledMs });
});
```

### Auto-restart supervisor
```typescript
guard = new StalledLoopGuard(
  () => processedCount,
  60_000,
  ({ sinceMs }) => {
    logger.fatal({ sinceMs }, "stall threshold exceeded — exiting for systemd restart");
    process.exit(2);
  },
);
setInterval(() => guard.tick(), 5_000);
```

### Stall metric for alerting (Prometheus)
```typescript
// engine_stall_seconds gauge
const stallGauge = new Gauge({ name: "engine_stall_seconds", help: "..." });
setInterval(() => {
  stallGauge.set((Date.now() - lastTick) / 1000);
}, 1_000);
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Single-process worker | heartbeat + exit(2), supervisor restart |
| Multi-tenant scheduler | per-tenant counter delta guard |
| External API source | per-call timeout + circuit breaker |
| Brief idle expected | exempt idle from stall (separate metric) |

**기본값**: heartbeat token + counter delta watchdog + supervisor restart.

## 🔗 Graph
- 부모: [[Observability]]
- 변형: [[Circuit Breaker]] · [[Liveness Probe]]
- 응용: [[Skybound Knowledge Hub]]
- Adjacent: [[Heartbeat]]

## 🤖 LLM 활용
**언제**: long-running ingestion / engine loop 의 silent stall detection 설계 시.
**언제 X**: short-lived request handler, batch jobs with natural exit.

## ❌ 안티패턴
- **Process alive == healthy**: liveness != progress. counter delta 가 truth.
- **Exception-only error path**: silent stall 은 throw 안 함.
- **Unbounded await**: timeout 없는 외부 call 은 무한 stall vector.
- **Restart without diagnostics**: stall 의 root cause 안 잡고 restart 만 하면 reoccur.

## 🧪 검증 / 중복
- Verified (Skybound production incident 2026-04-25 + watchdog pattern 의 standard literature).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-04-25 | Original incident note |
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — full FULL spec rewrite with watchdog patterns |
