---
id: wiki-2026-0508-structured-data
title: Structured Data
category: 10_Wiki/Topics
status: verified
canonical_id: self
aliases: [Tabular-Data, Schema-Data]
duplicate_of: none
source_trust_level: A
confidence_score: 0.9
verification_status: applied
tags: [data, schema, tabular, structured, formats]
raw_sources: []
last_reinforced: 2026-05-10
github_commit: pending
tech_stack:
  language: python
  framework: pandas-polars-arrow
---

# Structured Data

## 매 한 줄
> **"매 schema 가 있는 data — row × column 의 표"**. Structured 의 strict schema (RDB, Parquet), semi-structured 의 flexible (JSON, XML), unstructured 의 free-form (text, image). 1970 Codd's relational model 에서 시작 — 2026 에 Parquet/Arrow + JSON Schema + Pydantic 의 표준.

## 매 핵심

### 매 spectrum
- **Fully structured**: RDB, Parquet, CSV (with schema). 매 row=record, column=field, type 의 fixed.
- **Semi-structured**: JSON, XML, YAML, Avro. 매 schema 의 optional, flexible.
- **Unstructured**: text, image, audio, video. 매 schema 없음.
- 매 LLM era 의 unstructured → embedding 으로 structured 화 의 가능.

### 매 representation
- **Row-oriented**: CSV, JSON Lines, RDB OLTP. 매 record-at-a-time access.
- **Column-oriented**: Parquet, ORC, Arrow. 매 analytic scan / compression 의 우월.
- **Document**: MongoDB, Elasticsearch. 매 nested, schema-on-read.
- **Graph**: Neo4j, JanusGraph. 매 edge-first.

### 매 schema
- **Schema-on-write**: RDB, Parquet — 매 write 시 validation. 매 strict, fast read.
- **Schema-on-read**: JSON, MongoDB — 매 read 시 interpret. 매 flexible, slower read.
- **Schema evolution**: Avro, Protobuf — 매 backward/forward compat.

### 매 응용
1. Data warehouse — Parquet on S3 + Snowflake/BigQuery.
2. ML training — TFRecord, Parquet, HuggingFace datasets.
3. API — JSON + JSON Schema / OpenAPI.
4. Streaming — Avro / Protobuf in Kafka.
5. LLM tool calling — Pydantic schema → JSON.

## 💻 패턴

### 1. Parquet — write/read with schema
```python
import pandas as pd
df.to_parquet("data.parquet", engine="pyarrow", compression="zstd")
df2 = pd.read_parquet("data.parquet", columns=["id", "value"])  # 매 column projection
```

### 2. Polars (lazy, 2026 fast)
```python
import polars as pl
lf = pl.scan_parquet("data/*.parquet")
result = (
    lf.filter(pl.col("value") > 100)
      .group_by("category")
      .agg(pl.col("value").mean())
      .collect()
)
```

### 3. Arrow (zero-copy interchange)
```python
import pyarrow as pa
import pyarrow.parquet as pq
table = pa.Table.from_pandas(df)
pq.write_table(table, "data.parquet")
# 매 pandas, polars, duckdb 사이 zero-copy
```

### 4. Pydantic (Python schema validation, 2026)
```python
from pydantic import BaseModel, Field
class User(BaseModel):
    id: int
    name: str = Field(min_length=1)
    email: str
    age: int | None = None

user = User.model_validate({"id": 1, "name": "Alice", "email": "a@b.com"})
schema = User.model_json_schema()  # 매 LLM tool calling 의 사용 가능
```

### 5. JSON Schema validation
```python
import jsonschema
schema = {"type": "object", "properties": {"id": {"type": "integer"}}, "required": ["id"]}
jsonschema.validate(instance={"id": 42}, schema=schema)
```

### 6. SQL (DuckDB on local Parquet, 2026)
```python
import duckdb
con = duckdb.connect()
result = con.sql("""
    SELECT category, AVG(value) FROM 'data/*.parquet'
    WHERE date >= '2026-01-01' GROUP BY category
""").df()
```

### 7. Schema evolution (Avro)
```python
# 매 v1 → v2: optional field 의 add (with default) → backward compat
schema_v2 = {
    "type": "record",
    "fields": [
        {"name": "id", "type": "int"},
        {"name": "name", "type": "string"},
        {"name": "tier", "type": "string", "default": "free"}  # NEW
    ]
}
```

### 8. LLM structured output (Pydantic + Anthropic)
```python
from anthropic import Anthropic
from pydantic import BaseModel

class Extract(BaseModel):
    name: str
    age: int

# 매 tool calling with schema
schema = Extract.model_json_schema()
client = Anthropic()
resp = client.messages.create(
    model="claude-opus-4-7",
    max_tokens=1024,
    tools=[{"name": "extract", "input_schema": schema}],
    messages=[{"role": "user", "content": "Alice is 30 years old."}],
)
```

### 9. Iceberg / Delta (2026 lakehouse)
```python
# Iceberg — Parquet + manifest + snapshot 의 ACID
from pyiceberg.catalog import load_catalog
cat = load_catalog("local")
table = cat.load_table("ns.users")
df = table.scan().to_pandas()
```

## 매 결정 기준
| 상황 | Approach |
|---|---|
| Analytic / scan-heavy | Parquet (column) |
| Transactional / row-by-row | RDB / RocksDB |
| Streaming, schema evolve | Avro / Protobuf |
| Human-edit, config | YAML / TOML |
| API contract | JSON + Pydantic / JSON Schema / OpenAPI |
| Data lake | Parquet + Iceberg / Delta |
| ML training | Parquet / HuggingFace datasets / TFRecord |
| LLM tool input | JSON Schema (from Pydantic) |

**기본값**: Parquet for storage + Pydantic for validation + DuckDB/Polars for query.

## 🔗 Graph
- 부모: [[Entropy in Information Theory|Information Theory]] · [[Knowledge Graph]]
- 변형: [[Knowledge-Structure]] · [[Knowledge Graph|Knowledge-Graph-Foundations]]
- 응용: [[Information Retrieval (IR)]] · [[Statistics]]
- Adjacent: [[Hash-Functions-and-Maps]] · [[B-Tree]]

## 🤖 LLM 활용
**언제**: schema design, JSON-Schema generation, tool calling input/output, data validation guidance, format conversion explanation.
**언제 X**: 매 large-volume parse — pandas/polars/duckdb 의 사용.

## ❌ 안티패턴
- **CSV for production data**: type loss, encoding bugs, no compression — 매 Parquet 의 default.
- **JSON for analytic at scale**: 매 row-by-row parse 의 slow, 100x larger 의 Parquet vs.
- **No schema validation at boundary**: 매 silent corruption — Pydantic / JSON Schema 의 enforce.
- **Schema-on-read for hot path**: 매 latency penalty — 매 schema-on-write + indices.
- **Wide tables (1000+ cols)**: 매 most ops 의 unused — 매 column projection / split table.

## 🧪 검증 / 중복
- Verified (Apache Parquet/Arrow specs, Pydantic v2 docs, Iceberg spec).
- 신뢰도 A.

## 🕓 Changelog
| 날짜 | 변경 |
|---|---|
| 2026-05-08 | Phase 1 |
| 2026-05-10 | Manual cleanup — formats spectrum, Parquet/Pydantic/DuckDB patterns. |