kanta/docs/database.md

Kanta Database Format and Design Principles

This document describes the on-disk format and design principles of Kanta. It is intentionally focused on the current standalone package behavior.

Core Principles

1. Append-only durability

• State changes are persisted as appended JSON lines.
• Existing lines are never edited in place.
• In JSON mode the history is viewable in any text editor, and a manual rollback to a prior state is possible by removing final lines (one per transaction).

2. Differential persistence

• Kanta stores diffs (patches), not full state, for normal writes.
• This keeps write volume small and preserves a clear change history.

3. Deterministic replay

• Current state is reconstructed by replaying log records in order.
• Snapshot records accelerate replay while preserving deterministic results.

4. Transactional in-memory writes

• Application code mutates in-memory data inside kanta.transaction(...).
• On success, Kanta computes and queues a diff record.
• On failure, in-memory data is rolled back.

5. Explicit schema evolution

• Schema migration functions are versioned (migrate_vN).
• Migrations run at open time and advance the stored version.

On-Disk Record Types

Kanta uses a newline-delimited stream where each line is either a change record or a snapshot record.

Change record

One JSON object per line:

{"ts":"2026-06-10T02:55:00Z","a":"update","v":5,"u":"user-id","m":"2026-06-10T02:55:00Z","diff":{"users":{"alice":{"age":31}}}}

Fields:

• ts: UTC timestamp of the record.
• a: action name.
• v: schema version after this change.
• u: optional actor identifier.
• m: optional domain modification timestamp.
• diff: jsondiff patch payload.

Snapshot record

Snapshot lines are prefixed with SNAPSHOT , followed by JSON:

SNAPSHOT {"ts":"2026-06-10T00:00:00Z","v":5,"state":{"users":{}},"m":"2026-06-10T00:00:00Z","r":12,"k":"0.9.2"}

Fields:

• ts: snapshot creation time.
• v: schema version represented by the snapshot.
• state: full state dictionary.
• m: optional domain modification timestamp.
• r: revision of the snapshotted state — the total number of change records that occurred before this snapshot.
• k: meta format version (integer), the module-level constant kanta.structs.META_FORMAT at write time. It is bumped manually when the meta format changes. 0 (a missing field in old files) marks a legacy snapshot whose r cannot be trusted; readers then count change records from the start of the file instead. k1 introduced revision numbers.

Revision Numbers

A revision number identifies a state position between diffs, not a diff:

• The empty initial state is revision 0.
• Each change record — including the bootstrap record — advances the position by one, so after the first record the state is at revision 1.
• Change records carry no revision of their own; the revision of a state is counted from the preceding snapshot (snapshot.r + n records after it), or from the start of the file.
• Revisions are absolute and survive rotation: the leading snapshot of a rotated file carries the revision of its cutoff state, so r values keep referring to the same state positions across history rewrites.

kanta.rev exposes the current revision at runtime: the number of change records applied since revision 0, advanced in memory as transactions queue records. Reading it before open() raises DataIntegrityError. The CLI accepts rN range bounds to select records by revision (e.g. -r r10:r20): a bound rN addresses state N together with the record that produced it — the change record resulting in state N, or, when that change is no longer in the file, the snapshot materializing state N. -r r5 and -r r5:r6 therefore show the same record and output state 5, while -r r5..r6 also includes the next change and outputs state 6. Ranges truncate Python-style instead of erroring: on a rotated file whose leading snapshot holds revision 3, -r r1:r4 starts at the beginning of the file and shows that snapshot. Change record headers show the revision of the state the record produces as rN.

CLI Range Filtering

-r is repeatable and its specs fall into two classes:

• Unit specs — any spec with a letter-prefixed bound (l, s, v, r, including mixed forms like 2:s3) — index into the database file and resolve to a line window. Multiple unit specs intersect restrictively: only the common line range is processed.
• Plain-number specs — both bounds (or the single index) are plain numbers — index into the filtered output: the records left after all unit specs and all --grep patterns have been applied. Negative values count from the end of the filtered output.
• Time specs — a relative N + unit (s, min, h, d, w, e.g. 15min or 2w) selects the last N units ending now; a UTC ISO point (2026-09-28, 2026-09-28T10:30, or compact 20260928T1030) selects the whole natural period its precision denotes (that day, hour, minute, or second); an ISO range start/end is inclusive of the end's whole period, and either end may be left open. All times are UTC. Time specs match records individually by their timestamp rather than resolving to a line window, so non-monotonic timestamps simply match wherever they occur, and multiple time specs intersect like the others.

Like repeated --grep patterns (which must all match), repeated ranges are restrictive: -r r1:r5 -r 0:2 prints only the first two records of those in the revision window, regardless of flag order. Plain ranges truncate Python-style and print nothing when the intersection is empty; a single plain index must exist and fails with exit code 2 otherwise.

--context N (default 0) adds GNU grep-style context as the final step: each printed item is expanded by N surrounding records drawn from the whole file — expanding whatever would be shown past any range restriction, so -r s3 --context 2 shows that snapshot plus the two records before and after it. Overlapping expansions merge so nearby matches do not duplicate context records.

The JSON dump (-o) and migrations operate on the state after the last printed record, whatever filtering produced the printout — so a plain -r 2:5 dump ends at the 5th filtered record, and --grep or --context shift the dump to their last printed record as well. When nothing is printed (an empty filter result or -q), the dump falls back to the state at the end of the unit-spec window (or the end of the file).

CLI Search

--grep PATTERN (repeatable) prints only records matching the pattern, structurally and case-insensitively; matched regions get a yellow background, and a matched change record is printed in full, not just its matching line. Patterns match the record's structure, never rendered text:

• A bare pattern matches the action or user as a substring, a dotted path by element (each element in full unless it uses shell wildcards: users matches users anywhere but not foousers; us* does; elements match as a contiguous sequence, so users.*.email works), or a value (strings by substring, other scalars in full: true matches a boolean, tru does not).
• The path=value form requires both sides to match within the same change line; =value and path= match values or paths only.
• Repeated patterns are ANDed within the same record, but different patterns may match different lines of it.
• With -k, logfmt-prettified values, users, and path elements match alongside the raw ones, and the displayed (prettified) text is what gets highlighted.

Snapshots are matched against their flattened state with the same rules. A matched snapshot shows the matched records of its state as compact path.to.record = {value} lines under the indicator — normal path coloring, logfmt-prettified elements, matched regions highlighted — while unmatched parts of the state stay hidden. In the filter pipeline, grep applies after unit -r windows and before plain-number slicing and --context.

Meta Format Migrations

The snapshot k field carries an integer meta format version (kanta.structs.META_FORMAT, currently 1), bumped manually whenever the record-level format around the payloads changes. k1 introduced revision numbers.

When a writable open() finds a legacy (k0) last snapshot, the file is rewritten in full under the lock, before replay results are used further (meta migration k1, kanta/metamigr.py):

1. A leading snapshot, when the file begins with one, is preserved with its data and timestamps, re-stamped k1 with r = 0 (the only count knowable from the file).
2. All change records follow in order, re-framed; redundant intermediate snapshots are dropped.
3. A closing k1 snapshot captures the final state with r equal to the total number of change records.

After the rewrite every snapshot in the file is k1 with a trustworthy r, so readers never deal with mixed r/k meanings in one file. Rotation composes with this: it runs first and counts revisions from record positions regardless of k, so rotating a legacy file already produces k1 snapshots and no separate migration is needed. A read-only open computes the same revision the migration would have produced but leaves the file untouched.

Replay Model

1. Find the last snapshot in the file, if present.
2. Initialize replay state from snapshot state (or {} if none) and the base revision from the snapshot's r (or a count of the preceding change records when the snapshot is a legacy one without k).
3. Replay subsequent change records in order using patch application, counting them toward the revision.
4. The final replay state becomes in-memory kanta.data, and the final count becomes kanta.rev.

This model provides fast startup for large logs while retaining append-only history.

Serialization Semantics

• In-memory data is defined by an application msgspec.Struct type.
• Kanta round-trips through plain builtins for persistence and diffing.
• Dict keys are serialized as strings (str_keys=True) for stable JSON form.
• Normalization changes introduced by struct decode/encode are logged together with migrations as migrate:vN, or as migrate:msgspec when no migration ran but normalization still produces a diff.

Transaction Semantics

• kanta.transaction(action=...) captures a pre-transaction snapshot dict.
• By default a transaction updates the modification time m to the current UTC time.
• mtime=True|False|datetime controls the modification time m:
  • True (default) sets m to the current UTC time.
  • False omits m, leaving the previous modification time in effect.
  • A datetime sets m to that explicit value.
• System operations such as migrate:msgspec use mtime=False so they are not considered modifications and do not advance m.
• On success:
  • compute diff between previous builtins and current builtins,
  • queue a ChangeRecord if non-empty,
  • update kanta.mtime when the change carries an m value.
• On exception:
  • restore in-memory data from snapshot,
  • re-raise the exception.

Nested transactions are rejected.

Modification Time

kanta.mtime exposes the last modification time carried forward from change records. It is updated by normal transactions and preserved across snapshots and reloads, while system operations such as migrations leave it unchanged.

Flush and Lifecycle

• Writes are queued in memory.
• kanta.flush() appends queued records to disk.
• A background async task can flush periodically.
• kanta.close() performs final flush and releases file resources.
• async with Kanta(...) guarantees open/close lifecycle management.

Open Modes

• await kanta.open() (default) creates the database file if missing.
• await kanta.open(create=False) fails when the file is missing or empty.
• await kanta.open(readonly=True) opens an existing database read-only.
  • The file is opened without acquiring a lock and without a background flush task.
  • Existing records are replayed and migrations are still applied in memory.
  • Transactions and explicit flushes are rejected.
  • The file is never created if missing.

Callbacks

All callbacks are registered via decorators and receive arguments by their annotation types. Parameters without a supported annotation are only allowed when they have a default value.

Bootstrap Callbacks

• When open() creates a new database, it always writes a single bootstrap ChangeRecord.
• The simplest bootstrap is the initial data object passed to Kanta(...); bootstrap callbacks are optional and only needed when you want to modify or enrich that object at creation time.
• Register callbacks via:
  • @kanta.bootstrap
  • @kanta.bootstrap(action=..., user=..., mtime=...)
• Bootstrap callbacks may be sync or async. The live root data object is injected by annotating a parameter with the struct type passed to Kanta, and the Kanta instance itself can be injected by annotating a parameter with Kanta.
• Multiple bootstrap callbacks are supported:
  • callbacks execute in registration order,
  • exactly one bootstrap ChangeRecord is queued,
  • bootstrap metadata (action, user, mtime) is taken from the last callback registration.
• If no bootstrap callbacks are registered, the bootstrap record still uses action="bootstrap" and contains the initial data object.
• If any bootstrap callback raises, Kanta closes and removes the database file, then re-raises the exception.

Fatal Error Handlers

• Fatal background persistence errors can be handled with @kanta.fatal_error.
• Handlers may be sync or async. The DatabaseError is injected by annotating a parameter with DatabaseError; Kanta may also be injected.
• Multiple handlers are supported and invoked in registration order. A failing handler is logged and does not prevent subsequent handlers from running.

Clock

• @kanta.clock registers a callback () -> datetime that replaces the default UTC clock. Its value is used for all record timestamps (ts, and m when mtime is True) and for snapshot timestamps.
• The clock is only read when a timestamp is actually produced; no-op transactions and skipped snapshot checks do not read it.
• Register before open() so that bootstrap and migration records use the custom clock as well. This is mainly useful for tests and reproducible demos.

Transaction Log Formatting

• Logfmt callbacks prettify identifiers in the change log and are registered with @kanta.logfmt.
• A logfmt callback is called for every value Kanta renders: diff values, path components, and the transaction user. It receives the value as its first parameter and optionally a path: str parameter with the dot-notation path to the value. The special path "$user" is used when rendering the transaction actor.
• The callback returns str | None: a string replaces the default rendering, while None means "fall through to the next formatter".
• State dicts are injected by parameter name or annotation tag, which share the same vocabulary: prev receives the previous state dict and state the current one. Matching by name ignores the annotation entirely. The DictPrev/DictState aliases (Annotated[dict, "prev"] / Annotated[dict, "state"]) work under any parameter name, and a tag takes precedence over the name. DictPre and DictPost are kept as aliases of DictPrev and DictState. The Kanta instance can also be injected.
• Alternatively, a logfmt callback can be a class inheriting from LogFmt; the framework instantiates it with the state dicts and calls its resolve(value, path) -> str | None method.
• Multiple logfmt callbacks are stacked in registration order; the first callback to return a non-None result wins. If none handle a value, Kanta falls back to its default formatting.

The decorator accepts an optional path so the callback only runs for values at that exact path:

@kanta.logfmt(path="$user")
def resolve_user(value: str, state: dict) -> str | None:
    return state.get("users", {}).get(value, {}).get("name")

@kanta.logfmt(path="users.uuid-1")
def resolve_user_key(value: str) -> str | None:
    return names_by_id.get(value)

Transaction Log Headers

• By default a transaction is logged with an action by user header followed by the diff lines. Added paths are colored green, deleted paths red. ANSI color codes are stripped after formatting when the standard error stream does not support color: NO_COLOR disables colors, FORCE_COLOR forces them, otherwise a tty check and a journald (JOURNAL_STREAM) check decide. The CLI (python -m kanta) strips its output the same way.
• kanta.transaction(..., extra=...) accepts a display-only value that is shown after the action in the header. Anything other than None is printed str-converted (colored by Kanta), unless a custom logemit handler does something else with it; it is never persisted in the ChangeRecord.
• kanta.transaction(..., logdiff=False) skips building and printing the diff body and logs only the header, which is useful for large or noisy changesets. Diff output can also be disabled globally with configure_logging(diff=False); diff lines are emitted on the kanta.transaction.diff child logger so applications can route or silence them separately from the headers.
• The event loggers kanta.bootstrap, kanta.migration and kanta.transaction are configured at import time (via configure_logging(), callable again to change the toggles): a plain stderr handler with no prefix and propagate = False, since Kanta renders this output itself. They are set to INFO so Kanta output is visible without any application logging setup (debug=True lifts the DEBUG-emitting bootstrap/migration loggers to DEBUG); only loggers Kanta owns are touched. Operational diagnostics (integrity errors, flush failures, rotation notes) use the plain kanta logger instead, which is left unset — propagating to the root logger and following the application's normal logging configuration.

Log Emitters

Every change-related message Kanta emits (transaction/bootstrap/migration changes, file created/opened lines, migration summaries, aborted transactions) is described by a kanta.logging.LogEvent and dispatched through kanta.logging.emit_event. Kanta's own output goes through the same mechanism: when no logemit callback handles an event, kanta.logging.default_emit renders it with the built-in formatting.

• A LogEvent carries the event kind ("change", "created", "opened", "migrated", "aborted"), the preferred logger and level, the kanta instance, and all relevant state: action, user, extra, error (for aborted transactions), diff, previous/current state dicts, the built logfmt chain, and version info for migration events. Application-specific context (e.g. a connection id) can be stored in kanta.ctx — a user-writable namespace — and read back in callbacks as event.kanta.ctx, which also covers creation/bootstrap events.
• The built-in formatting is assembled from standard blocks that custom emitters can reuse as-is or replace piecemeal:
  • event.header — a lazy property producing the default one-line header for any kind: <action>[ <extra>][ by <user>] for changes, <action>[ <extra>][ by <user>] transaction aborted: <error> for aborts, and the 🛢️ <file> created|opened|migrated ... summaries. It is settable: assign event.header = ... and return truthy to restyle the header while keeping the default diff routing.
  • event.diff_lines — a lazy property producing the pretty diff body for change events (built only if accessed).
  • default_emit itself is just header plus the diff_lines routing.
• @kanta.logemit registers a callback receiving the event. The callback decides what is logged and where: it may log one or more messages on event.logger, log somewhere else, or nothing at all. A falsy return value marks the event handled and stops the chain; a truthy return value passes the event — possibly modified — to the next registered callback. When all callbacks pass, default_emit renders the event; a callback may also call default_emit(event) itself to delegate events it does not customize. Operational diagnostics (integrity errors, background flush failures) do not go through this mechanism.
• Logging never breaks functionality: a crashing logemit callback is reported with logger.exception and the event falls back to the built-in formatting; if the built-in formatting itself fails, the error is reported and swallowed. The same applies to logfmt callbacks (a failing one is treated as a fall-through) and logmigr callbacks.

@kanta.logemit
def emit(ev: LogEvent):
    if ev.kind != "change":
        return default_emit(ev)  # delegate, no chaining needed
    # Restyle the header; default_emit keeps routing the diff body.
    ev.header = str(Line().user(ev.user or "-", width=20)(" ").action(ev.action))
    return True

Terminal Formatting Helpers

• kanta.tty provides the building blocks used by Kanta's own rendering:
  • colors: the mutable color palette. Colors are bare SGR parameter strings (e.g. "1;34", "38;5;226") without escape framing. Attributes are read at render time, so assignments (colors.action = "36") and additions (colors.session = "38;5;226") take effect immediately.
  • Line: builds a terminal string part by part. Calling it appends content (str-converted); .<colorname> arms a palette color for the next call only, and the reset is folded into a single escape sequence with whatever color comes next. width=/align= pad by display width; str(line) finishes the line and restores default colors.
  • strip_ansi, displaywidth (wide chars and emoji count correctly) and pad for working with pre-colored strings.
  • use_color(stream): the color-support test used by Kanta's own output — honors NO_COLOR/FORCE_COLOR, then stream.isatty(), then the journald JOURNAL_STREAM device/inode match.

Migrations

• Migration source is configured on Kanta(...) via migrations=.
• Accepted values:
  • imported module object,
  • import path string.
• Migrations mutate replayed dict state in-place and return the new version.

Safety Invariants

• Any detected out-of-transaction mutation is treated as a fatal consistency violation.
• Flush failures mark the instance as failed and trigger shutdown behavior.
• Object identity of kanta.data is preserved across rollback when possible, minimizing stale-reference hazards for callers.