Module wal

Write-Ahead Log (WAL) for crash-safe ingestion.

Implements an append-only, CRC32-validated log with segment rotation and recovery protocol per the Storage Layout spec §3.

Architecture

wal/
├── segment-000000000000.wal   (64MB max, append-only)
├── segment-000000000001.wal
└── wal.meta                   (committed state)

Entry lifecycle

1. Caller appends entry → WAL writes header + payload + CRC32
2. After downstream store + index confirm, caller calls commit()
3. On crash before commit, recovery replays uncommitted entries

Recovery protocol

1. Read wal.meta for committed sequence number
2. Scan all segments for entries after committed sequence
3. Validate CRC32 — truncate at first invalid entry
4. Return uncommitted valid entries for replay

Structs

Wal

Write-Ahead Log.

WalConfig

WAL configuration.

WalEntry

A WAL entry ready for serialization.

WalMeta 🔒

Persisted WAL metadata.

Enums

EntryType

WAL entry types.

Constants

DEFAULT_MAX_SEGMENT_SIZE 🔒

Default maximum segment size: 64 MB.

ENTRY_HEADER_SIZE 🔒

Entry header size in bytes (before payload).

SEGMENT_HEADER_SIZE 🔒

Segment header size in bytes.

WAL_MAGIC 🔒

Magic bytes identifying a CVX WAL segment.

WAL_VERSION 🔒

Current WAL format version.

Functions

crc32_hash 🔒

Simple CRC32 hash (IEEE polynomial).

decode_delete_payload

Decode a delete payload.

decode_insert_payload

Decode an insert payload back into components.

encode_delete_payload

Encode a delete payload: entity_id + timestamp.

encode_insert_payload

Encode an insert payload: entity_id + timestamp + dimension + vector.