[G1-Sync] Manual knowledge update
This commit is contained in:
Antigravity Agent
@@ -0,0 +1,455 @@
|
||||
---
|
||||
id: db-connection-pooling-patterns
|
||||
title: Connection Pooling — PgBouncer / Pool / Statement
|
||||
category: Coding
|
||||
status: draft
|
||||
source_trust_level: B
|
||||
verification_status: conceptual
|
||||
created_at: 2026-05-09
|
||||
updated_at: 2026-05-09
|
||||
tags: [database, pool, vibe-coding]
|
||||
tech_stack: { language: "TS / Postgres", applicable_to: ["Backend"] }
|
||||
applied_in: []
|
||||
aliases: [PgBouncer, connection pool, pool mode, statement pool, transaction pool, RDS Proxy]
|
||||
---
|
||||
|
||||
# Connection Pooling Patterns
|
||||
|
||||
> Postgres connection 가 expensive. **App pool (small) + PgBouncer (transaction pool, 1000+ client)**. Lambda / serverless = HTTP driver / RDS Proxy.
|
||||
|
||||
## 📖 핵심 개념
|
||||
- App pool: 매 process 가 N connection.
|
||||
- External pool: PgBouncer 가 multiplex.
|
||||
- Pool mode: session / transaction / statement.
|
||||
- Limit: Postgres 의 max_connections.
|
||||
|
||||
## 💻 코드 패턴
|
||||
|
||||
### App pool (간단)
|
||||
```ts
|
||||
import { Pool } from 'pg';
|
||||
|
||||
const pool = new Pool({
|
||||
connectionString,
|
||||
max: 20,
|
||||
idleTimeoutMillis: 30_000,
|
||||
connectionTimeoutMillis: 5_000,
|
||||
});
|
||||
|
||||
// 모든 query 가 pool 사용
|
||||
await pool.query('SELECT * FROM users');
|
||||
```
|
||||
|
||||
→ App instance 당 N. 큰 traffic = 큰 N — Postgres 한계.
|
||||
|
||||
### Postgres max_connections
|
||||
```sql
|
||||
SHOW max_connections;
|
||||
-- Default: 100. 매 connection ~= 10 MB RAM.
|
||||
|
||||
-- Heavy production:
|
||||
ALTER SYSTEM SET max_connections = 200;
|
||||
```
|
||||
|
||||
→ 매 connection 의 cost (memory + process). Limit 있음.
|
||||
|
||||
### Pool 크기 결정
|
||||
```
|
||||
규칙 (basic):
|
||||
max = (CPU 코어 × 2) + effective_spindle
|
||||
|
||||
DB 4 core SSD:
|
||||
- 작은: 5-10 per app instance
|
||||
- 일반: 20-30
|
||||
- 큰: 50
|
||||
|
||||
App instance × pool max < Postgres max_connections.
|
||||
e.g. 10 instance × 20 = 200 < 250.
|
||||
```
|
||||
|
||||
### PgBouncer (외부 pool)
|
||||
```ini
|
||||
# pgbouncer.ini
|
||||
[databases]
|
||||
app = host=primary-db port=5432 dbname=app
|
||||
|
||||
[pgbouncer]
|
||||
listen_port = 6432
|
||||
auth_type = md5
|
||||
auth_file = /etc/pgbouncer/userlist.txt
|
||||
|
||||
pool_mode = transaction # session / transaction / statement
|
||||
max_client_conn = 1000 # client → pgbouncer
|
||||
default_pool_size = 25 # pgbouncer → Postgres
|
||||
reserve_pool_size = 5
|
||||
server_idle_timeout = 600
|
||||
```
|
||||
|
||||
→ App 가 PgBouncer (port 6432) 호출. PgBouncer 가 Postgres connection multiplex.
|
||||
|
||||
### Pool modes
|
||||
```
|
||||
Session:
|
||||
- Client 가 connection 점유 (release 까지)
|
||||
- 모든 feature OK
|
||||
- Multiplex 안 됨
|
||||
|
||||
Transaction:
|
||||
- Transaction 끝 마다 release
|
||||
- ~10x more efficient
|
||||
- 일부 feature X (prepared statements, advisory locks)
|
||||
|
||||
Statement:
|
||||
- 매 statement 끝 release
|
||||
- 가장 efficient
|
||||
- 더 많은 feature X (transactions X)
|
||||
|
||||
→ 보통 transaction.
|
||||
```
|
||||
|
||||
### Transaction mode 의 함정
|
||||
```
|
||||
Session-bound features 안 됨:
|
||||
- SET (variable)
|
||||
- LISTEN / NOTIFY
|
||||
- Prepared statements (자체 prepare)
|
||||
- Advisory lock (xact 만 OK)
|
||||
- Cursor (WITH HOLD)
|
||||
- Temporary table (보통)
|
||||
|
||||
→ 일반 query 는 OK.
|
||||
```
|
||||
|
||||
```ts
|
||||
// Workaround: prepared statements off
|
||||
const pool = new Pool({
|
||||
connectionString: 'postgres://user:pw@pgbouncer:6432/app',
|
||||
// 자체 prepare 비활성
|
||||
});
|
||||
|
||||
// node-postgres
|
||||
client.query({ name: 'q', text: '...' }); // pgbouncer transaction mode 깨짐 가능
|
||||
|
||||
// Use raw query
|
||||
client.query('...');
|
||||
```
|
||||
|
||||
### node-postgres + PgBouncer
|
||||
```ts
|
||||
import postgres from 'postgres';
|
||||
|
||||
const sql = postgres(url, {
|
||||
max: 10,
|
||||
prepare: false, // pgbouncer transaction mode
|
||||
});
|
||||
```
|
||||
|
||||
```ts
|
||||
// pg
|
||||
const pool = new Pool({
|
||||
connectionString,
|
||||
// statement_timeout 매 connection
|
||||
statement_timeout: 30_000,
|
||||
});
|
||||
|
||||
pool.on('connect', (client) => {
|
||||
client.query('SET application_name = "my-app"'); // session 별
|
||||
});
|
||||
```
|
||||
|
||||
### RDS Proxy (AWS)
|
||||
```ts
|
||||
// Lambda → RDS Proxy → RDS
|
||||
// 자동 connection pool + auth + IAM
|
||||
|
||||
// Same code 가 그냥 endpoint 변경
|
||||
const pool = new Pool({
|
||||
host: 'my-proxy.proxy-xxx.rds.amazonaws.com',
|
||||
// ...
|
||||
});
|
||||
```
|
||||
|
||||
→ Lambda + Postgres 의 답.
|
||||
|
||||
### Hyperdrive (Cloudflare)
|
||||
```ts
|
||||
// wrangler.toml
|
||||
[[hyperdrive]]
|
||||
binding = "HYPERDRIVE"
|
||||
id = "..."
|
||||
```
|
||||
|
||||
```ts
|
||||
import postgres from 'postgres';
|
||||
|
||||
export default {
|
||||
async fetch(req: Request, env: Env) {
|
||||
const sql = postgres(env.HYPERDRIVE.connectionString);
|
||||
const r = await sql`SELECT * FROM users WHERE id = ${id}`;
|
||||
return Response.json(r);
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
→ Hyperdrive = pool + cache. CF Workers 에서 일반 Postgres.
|
||||
|
||||
### Neon HTTP driver
|
||||
```ts
|
||||
import { neon } from '@neondatabase/serverless';
|
||||
const sql = neon(url);
|
||||
|
||||
const users = await sql`SELECT * FROM users`;
|
||||
```
|
||||
|
||||
→ Connection 없음. HTTP request 만. Edge / Lambda 친화.
|
||||
|
||||
### Supabase pooler
|
||||
```
|
||||
Supabase 가 PgBouncer 자체 host.
|
||||
- session: port 5432
|
||||
- transaction: port 6543
|
||||
|
||||
→ App = transaction pool 사용 (default).
|
||||
```
|
||||
|
||||
### Pool stats (모니터링)
|
||||
```ts
|
||||
setInterval(() => {
|
||||
log.info('pool', {
|
||||
total: pool.totalCount,
|
||||
idle: pool.idleCount,
|
||||
waiting: pool.waitingCount,
|
||||
});
|
||||
}, 30_000);
|
||||
```
|
||||
|
||||
→ waiting > 0 자주 = pool 부족 / leak.
|
||||
|
||||
### PgBouncer 확인
|
||||
```sql
|
||||
-- PgBouncer admin
|
||||
\c pgbouncer
|
||||
SHOW pools;
|
||||
-- cl_active / sv_active / cl_waiting
|
||||
|
||||
SHOW stats;
|
||||
-- requests, queries, etc
|
||||
|
||||
SHOW clients;
|
||||
SHOW servers;
|
||||
```
|
||||
|
||||
### Connection leak
|
||||
```ts
|
||||
// ❌ Release 안 함
|
||||
const client = await pool.connect();
|
||||
const r = await client.query('SELECT ...');
|
||||
// 누락: client.release()
|
||||
|
||||
// ✅ try-finally
|
||||
const client = await pool.connect();
|
||||
try {
|
||||
await client.query('...');
|
||||
} finally {
|
||||
client.release();
|
||||
}
|
||||
|
||||
// 또는 pool.query (자동 release)
|
||||
await pool.query('...');
|
||||
```
|
||||
|
||||
### Application restart
|
||||
```
|
||||
모든 connection re-create.
|
||||
Pool warm-up:
|
||||
- pre-create min connections at startup
|
||||
- 첫 request 가 빠름
|
||||
```
|
||||
|
||||
```ts
|
||||
const pool = new Pool({
|
||||
min: 5, // 시작 시 미리 5개
|
||||
});
|
||||
```
|
||||
|
||||
### Multiple DBs (read / write split)
|
||||
```ts
|
||||
const writer = new Pool({ connectionString: PRIMARY_URL, max: 20 });
|
||||
const reader = new Pool({ connectionString: REPLICA_URL, max: 50 });
|
||||
|
||||
async function getOrders(userId: string) {
|
||||
return reader.query('SELECT * FROM orders WHERE user_id = $1', [userId]);
|
||||
}
|
||||
|
||||
async function createOrder(data) {
|
||||
return writer.query('INSERT INTO orders ...', [...]);
|
||||
}
|
||||
```
|
||||
|
||||
→ [[DB_Read_Replica_Patterns]].
|
||||
|
||||
### Tenant pool (multi-tenant)
|
||||
```ts
|
||||
// Approach: per-tenant DB
|
||||
const pools = new Map<string, Pool>();
|
||||
|
||||
function getPool(tenantId: string): Pool {
|
||||
if (!pools.has(tenantId)) {
|
||||
pools.set(tenantId, new Pool({ ... }));
|
||||
}
|
||||
return pools.get(tenantId)!;
|
||||
}
|
||||
|
||||
// Cleanup unused tenants
|
||||
```
|
||||
|
||||
→ N tenant × pool size = 큰 — 주의.
|
||||
|
||||
### DB connection 이 가장 비싼 자원
|
||||
```
|
||||
1 connection ≈ 10 MB Postgres process.
|
||||
1000 connection ≈ 10 GB.
|
||||
|
||||
→ Pool size 작게 + multiplex.
|
||||
```
|
||||
|
||||
### Lambda connection issue
|
||||
```
|
||||
Lambda = 매 invocation 새 container 가능.
|
||||
1000 concurrent Lambda = 1000 connection.
|
||||
|
||||
해결:
|
||||
1. RDS Proxy (AWS)
|
||||
2. Hyperdrive (CF)
|
||||
3. Neon HTTP / Serverless
|
||||
4. Pool 매 container reuse (warm Lambda)
|
||||
```
|
||||
|
||||
### Idle in transaction
|
||||
```sql
|
||||
-- App 가 BEGIN 후 외부 호출 hang
|
||||
SELECT pid, state, query, query_start
|
||||
FROM pg_stat_activity
|
||||
WHERE state = 'idle in transaction';
|
||||
|
||||
-- Auto kill
|
||||
ALTER SYSTEM SET idle_in_transaction_session_timeout = '60s';
|
||||
```
|
||||
|
||||
→ 60s 안 release 안 하면 cancel.
|
||||
|
||||
→ [[DB_Lock_Analysis]].
|
||||
|
||||
### Statement timeout
|
||||
```ts
|
||||
// Connection 별
|
||||
client.query('SET statement_timeout = 30000'); // 30s
|
||||
|
||||
// 또는 connection string
|
||||
postgres://user:pw@host:5432/db?statement_timeout=30000
|
||||
```
|
||||
|
||||
→ Hang query 방지.
|
||||
|
||||
### Retry on connection error
|
||||
```ts
|
||||
async function queryWithRetry<T>(query: string, params: any[]): Promise<T> {
|
||||
for (let i = 0; i < 3; i++) {
|
||||
try {
|
||||
return await pool.query(query, params);
|
||||
} catch (e: any) {
|
||||
if (e.code === 'ECONNRESET' || e.code === 'ETIMEDOUT') {
|
||||
await sleep(100 * (i + 1));
|
||||
continue;
|
||||
}
|
||||
throw e;
|
||||
}
|
||||
}
|
||||
throw new Error('max retries');
|
||||
}
|
||||
```
|
||||
|
||||
### Health check
|
||||
```ts
|
||||
async function dbHealthy(): Promise<boolean> {
|
||||
try {
|
||||
await Promise.race([
|
||||
pool.query('SELECT 1'),
|
||||
new Promise((_, reject) => setTimeout(() => reject(), 5000)),
|
||||
]);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### PgBouncer alternative
|
||||
```
|
||||
- pgcat (Rust, modern)
|
||||
- pgpool-II (older, complex)
|
||||
- Supavisor (Supabase)
|
||||
- Odyssey (Yandex)
|
||||
|
||||
→ PgBouncer 가 가장 인기.
|
||||
```
|
||||
|
||||
### Production setup (typical)
|
||||
```
|
||||
App (10 instance) → PgBouncer (3 instance) → Postgres (primary + replicas)
|
||||
|
||||
App pool: 5-10 / instance
|
||||
PgBouncer: max_client_conn = 1000, pool_size = 25
|
||||
Postgres: max_connections = 100
|
||||
```
|
||||
|
||||
→ 1000 client → 25 DB connection (40x multiplex).
|
||||
|
||||
### Architecture
|
||||
```
|
||||
[App] [App] [App]
|
||||
↓ ↓ ↓
|
||||
[PgBouncer]
|
||||
↓
|
||||
[Postgres primary]
|
||||
↕
|
||||
[Postgres replica]
|
||||
```
|
||||
|
||||
### Cloud manage
|
||||
```
|
||||
RDS Proxy: AWS, supports MySQL / Postgres
|
||||
Aurora Serverless v2: auto-scale
|
||||
Neon / Supabase: built-in pool
|
||||
Cloud SQL: external pool 직접
|
||||
```
|
||||
|
||||
## 🤔 의사결정 기준
|
||||
| 환경 | 추천 |
|
||||
|---|---|
|
||||
| 일반 server | App pool |
|
||||
| 큰 traffic / 많은 instance | PgBouncer |
|
||||
| Lambda | RDS Proxy / Hyperdrive |
|
||||
| Cloudflare Workers | Hyperdrive / Neon HTTP |
|
||||
| Edge | Neon HTTP / Turso |
|
||||
| Production | App + PgBouncer |
|
||||
|
||||
## ❌ 안티패턴
|
||||
- **App instance × pool > Postgres max**: connection 폭발.
|
||||
- **session pool mode + multi-tenant**: 격리 약함.
|
||||
- **Transaction pool + session feature 사용**: 깨짐.
|
||||
- **Pool 안 release**: leak.
|
||||
- **Long-running transaction**: pool 다 잡음.
|
||||
- **Idle timeout 길음 NAT 보다**: zombie.
|
||||
- **모니터링 없음**: 점진 다운.
|
||||
|
||||
## 🤖 LLM 활용 힌트
|
||||
- App pool (작게) + PgBouncer (multiplex).
|
||||
- Lambda = HTTP driver / proxy.
|
||||
- Transaction mode = default.
|
||||
- Pool stats 항상 monitor.
|
||||
|
||||
## 🔗 관련 문서
|
||||
- [[Backend_Connection_Handling]]
|
||||
- [[DB_Connection_Pool]]
|
||||
- [[DB_Lock_Analysis]]
|
||||
Reference in New Issue
Block a user