Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8fd86d20e1 | ||
|
|
e296859383 | ||
|
|
2c7487de73 | ||
|
|
788d285e38 | ||
|
|
c2821dedb5 | ||
|
|
ecf85445ea | ||
|
|
7b132bda28 | ||
|
|
5c32e495e9 | ||
|
|
b8d2222905 | ||
|
|
de2476c59c | ||
|
|
ec8695ba0c | ||
|
|
a4efbca55b | ||
|
|
44c1ed191e | ||
|
|
a41f34d332 | ||
|
|
7a7716ec7b | ||
|
|
c608e749a7 | ||
|
|
8e780d7ee4 | ||
|
|
2e6f48bac5 |
No files matched your search
@@ -15,6 +15,8 @@ This design is often preferable when you want low-latency local persistence with
|
|||||||
|
|
||||||
Queries and updates on native data are far faster than over an SQL server connection, and we can can provide fully synchronous operation. The limitation is that you can only use the same database within a single process at a time, but this is well suited for async programming.
|
Queries and updates on native data are far faster than over an SQL server connection, and we can can provide fully synchronous operation. The limitation is that you can only use the same database within a single process at a time, but this is well suited for async programming.
|
||||||
|
|
||||||
|
See also: [SQL, NoSQL, and why I started Kanta DB](https://vasanko.com/coders/kanta)
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
```python
|
```python
|
||||||
|
|||||||
+57
-5
@@ -51,7 +51,7 @@ Fields:
|
|||||||
Snapshot lines are prefixed with `SNAPSHOT `, followed by JSON:
|
Snapshot lines are prefixed with `SNAPSHOT `, followed by JSON:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
SNAPSHOT {"ts":"2026-06-10T00:00:00Z","v":5,"state":{"users":{}},"m":"2026-06-10T00:00:00Z"}
|
SNAPSHOT {"ts":"2026-06-10T00:00:00Z","v":5,"state":{"users":{}},"m":"2026-06-10T00:00:00Z","r":12,"k":"0.9.2"}
|
||||||
```
|
```
|
||||||
|
|
||||||
Fields:
|
Fields:
|
||||||
@@ -59,13 +59,63 @@ Fields:
|
|||||||
- `v`: schema version represented by the snapshot.
|
- `v`: schema version represented by the snapshot.
|
||||||
- `state`: full state dictionary.
|
- `state`: full state dictionary.
|
||||||
- `m`: optional domain modification timestamp.
|
- `m`: optional domain modification timestamp.
|
||||||
|
- `r`: revision of the snapshotted state — the total number of change records that occurred before this snapshot.
|
||||||
|
- `k`: meta format version (integer), the module-level constant `kanta.structs.META_FORMAT` at write time. It is bumped manually when the meta format changes. `0` (a missing field in old files) marks a legacy snapshot whose `r` cannot be trusted; readers then count change records from the start of the file instead. k1 introduced revision numbers.
|
||||||
|
|
||||||
|
## Revision Numbers
|
||||||
|
|
||||||
|
A revision number identifies a *state position between diffs*, not a diff:
|
||||||
|
|
||||||
|
- The empty initial state is revision 0.
|
||||||
|
- Each change record — including the bootstrap record — advances the position by one, so after the first record the state is at revision 1.
|
||||||
|
- Change records carry no revision of their own; the revision of a state is counted from the preceding snapshot (`snapshot.r + n` records after it), or from the start of the file.
|
||||||
|
- Revisions are absolute and survive rotation: the leading snapshot of a rotated file carries the revision of its cutoff state, so `r` values keep referring to the same state positions across history rewrites.
|
||||||
|
|
||||||
|
`kanta.rev` exposes the current revision at runtime: the number of change records applied since revision 0, advanced in memory as transactions queue records. Reading it before `open()` raises `DataIntegrityError`. The CLI accepts `rN` range bounds to select records by revision (e.g. `-r r10:r20`): a bound `rN` addresses state N together with the record that produced it — the change record resulting in state N, or, when that change is no longer in the file, the snapshot materializing state N. `-r r5` and `-r r5:r6` therefore show the same record and output state 5, while `-r r5..r6` also includes the next change and outputs state 6. Ranges truncate Python-style instead of erroring: on a rotated file whose leading snapshot holds revision 3, `-r r1:r4` starts at the beginning of the file and shows that snapshot. Change record headers show the revision of the state the record produces as `rN`.
|
||||||
|
|
||||||
|
## CLI Range Filtering
|
||||||
|
|
||||||
|
`-r` is repeatable and its specs fall into two classes:
|
||||||
|
|
||||||
|
- **Unit specs** — any spec with a letter-prefixed bound (`l`, `s`, `v`, `r`, including mixed forms like `2:s3`) — index into the database file and resolve to a line window. Multiple unit specs intersect restrictively: only the common line range is processed.
|
||||||
|
- **Plain-number specs** — both bounds (or the single index) are plain numbers — index into the *filtered output*: the records left after all unit specs and all `--grep` patterns have been applied. Negative values count from the end of the filtered output.
|
||||||
|
- **Time specs** — a relative `N` + unit (`s`, `min`, `h`, `d`, `w`, e.g. `15min` or `2w`) selects the last N units ending now; a UTC ISO point (`2026-09-28`, `2026-09-28T10:30`, or compact `20260928T1030`) selects the whole natural period its precision denotes (that day, hour, minute, or second); an ISO range `start/end` is inclusive of the end's whole period, and either end may be left open. All times are UTC. Time specs match records individually by their timestamp rather than resolving to a line window, so non-monotonic timestamps simply match wherever they occur, and multiple time specs intersect like the others.
|
||||||
|
|
||||||
|
Like repeated `--grep` patterns (which must all match), repeated ranges are restrictive: `-r r1:r5 -r 0:2` prints only the first two records of those in the revision window, regardless of flag order. Plain ranges truncate Python-style and print nothing when the intersection is empty; a single plain index must exist and fails with exit code 2 otherwise.
|
||||||
|
|
||||||
|
`--context N` (default 0) adds GNU grep-style context as the final step: each printed item is expanded by N surrounding records drawn from the whole file — expanding whatever would be shown past any range restriction, so `-r s3 --context 2` shows that snapshot plus the two records before and after it. Overlapping expansions merge so nearby matches do not duplicate context records.
|
||||||
|
|
||||||
|
The JSON dump (`-o`) and migrations operate on the state after the **last printed record**, whatever filtering produced the printout — so a plain `-r 2:5` dump ends at the 5th filtered record, and `--grep` or `--context` shift the dump to their last printed record as well. When nothing is printed (an empty filter result or `-q`), the dump falls back to the state at the end of the unit-spec window (or the end of the file).
|
||||||
|
|
||||||
|
## CLI Search
|
||||||
|
|
||||||
|
`--grep PATTERN` (repeatable) prints only records matching the pattern, structurally and case-insensitively; matched regions get a yellow background, and a matched change record is printed in full, not just its matching line. Patterns match the record's structure, never rendered text:
|
||||||
|
|
||||||
|
- A bare pattern matches the action or user as a substring, a dotted path by element (each element in full unless it uses shell wildcards: `users` matches `users` anywhere but not `foousers`; `us*` does; elements match as a contiguous sequence, so `users.*.email` works), or a value (strings by substring, other scalars in full: `true` matches a boolean, `tru` does not).
|
||||||
|
- The `path=value` form requires both sides to match within the same change line; `=value` and `path=` match values or paths only.
|
||||||
|
- Repeated patterns are ANDed within the same record, but different patterns may match different lines of it.
|
||||||
|
- With `-k`, logfmt-prettified values, users, and path elements match alongside the raw ones, and the displayed (prettified) text is what gets highlighted.
|
||||||
|
|
||||||
|
Snapshots are matched against their flattened state with the same rules. A matched snapshot shows the matched records of its state as compact `path.to.record = {value}` lines under the indicator — normal path coloring, logfmt-prettified elements, matched regions highlighted — while unmatched parts of the state stay hidden. In the filter pipeline, grep applies after unit `-r` windows and before plain-number slicing and `--context`.
|
||||||
|
|
||||||
|
## Meta Format Migrations
|
||||||
|
|
||||||
|
The snapshot `k` field carries an integer meta format version (`kanta.structs.META_FORMAT`, currently 1), bumped manually whenever the record-level format around the payloads changes. k1 introduced revision numbers.
|
||||||
|
|
||||||
|
When a writable `open()` finds a legacy (k0) last snapshot, the file is rewritten in full under the lock, before replay results are used further (meta migration k1, `kanta/metamigr.py`):
|
||||||
|
|
||||||
|
1. A leading snapshot, when the file begins with one, is preserved with its data and timestamps, re-stamped k1 with `r = 0` (the only count knowable from the file).
|
||||||
|
2. All change records follow in order, re-framed; redundant intermediate snapshots are dropped.
|
||||||
|
3. A closing k1 snapshot captures the final state with `r` equal to the total number of change records.
|
||||||
|
|
||||||
|
After the rewrite every snapshot in the file is k1 with a trustworthy `r`, so readers never deal with mixed r/k meanings in one file. Rotation composes with this: it runs first and counts revisions from record positions regardless of `k`, so rotating a legacy file already produces k1 snapshots and no separate migration is needed. A read-only open computes the same revision the migration would have produced but leaves the file untouched.
|
||||||
|
|
||||||
## Replay Model
|
## Replay Model
|
||||||
|
|
||||||
1. Find the last snapshot in the file, if present.
|
1. Find the last snapshot in the file, if present.
|
||||||
2. Initialize replay state from snapshot state (or `{}` if none).
|
2. Initialize replay state from snapshot state (or `{}` if none) and the base revision from the snapshot's `r` (or a count of the preceding change records when the snapshot is a legacy one without `k`).
|
||||||
3. Replay subsequent change records in order using patch application.
|
3. Replay subsequent change records in order using patch application, counting them toward the revision.
|
||||||
4. The final replay state becomes in-memory `kanta.data`.
|
4. The final replay state becomes in-memory `kanta.data`, and the final count becomes `kanta.rev`.
|
||||||
|
|
||||||
This model provides fast startup for large logs while retaining append-only history.
|
This model provides fast startup for large logs while retaining append-only history.
|
||||||
|
|
||||||
@@ -171,9 +221,10 @@ def resolve_user_key(value: str) -> str | None:
|
|||||||
|
|
||||||
#### Transaction Log Headers
|
#### Transaction Log Headers
|
||||||
|
|
||||||
- By default a transaction is logged with an `action by user` header followed by the diff lines. Added paths are colored green, deleted paths red.
|
- By default a transaction is logged with an `action by user` header followed by the diff lines. Added paths are colored green, deleted paths red. ANSI color codes are stripped after formatting when the standard error stream does not support color: `NO_COLOR` disables colors, `FORCE_COLOR` forces them, otherwise a tty check and a journald (`JOURNAL_STREAM`) check decide. The CLI (`python -m kanta`) strips its output the same way.
|
||||||
- `kanta.transaction(..., extra=...)` accepts a display-only value that is shown after the action in the header. Anything other than `None` is printed str-converted (colored by Kanta), unless a custom logemit handler does something else with it; it is never persisted in the `ChangeRecord`.
|
- `kanta.transaction(..., extra=...)` accepts a display-only value that is shown after the action in the header. Anything other than `None` is printed str-converted (colored by Kanta), unless a custom logemit handler does something else with it; it is never persisted in the `ChangeRecord`.
|
||||||
- `kanta.transaction(..., logdiff=False)` skips building and printing the diff body and logs only the header, which is useful for large or noisy changesets. Diff output can also be disabled globally with `configure_logging(diff=False)`; diff lines are emitted on the `kanta.transaction.diff` child logger so applications can route or silence them separately from the headers.
|
- `kanta.transaction(..., logdiff=False)` skips building and printing the diff body and logs only the header, which is useful for large or noisy changesets. Diff output can also be disabled globally with `configure_logging(diff=False)`; diff lines are emitted on the `kanta.transaction.diff` child logger so applications can route or silence them separately from the headers.
|
||||||
|
- The event loggers `kanta.bootstrap`, `kanta.migration` and `kanta.transaction` are configured at import time (via `configure_logging()`, callable again to change the toggles): a plain stderr handler with no prefix and `propagate = False`, since Kanta renders this output itself. They are set to INFO so Kanta output is visible without any application logging setup (`debug=True` lifts the DEBUG-emitting bootstrap/migration loggers to DEBUG); only loggers Kanta owns are touched. Operational diagnostics (integrity errors, flush failures, rotation notes) use the plain `kanta` logger instead, which is left unset — propagating to the root logger and following the application's normal logging configuration.
|
||||||
|
|
||||||
#### Log Emitters
|
#### Log Emitters
|
||||||
|
|
||||||
@@ -203,6 +254,7 @@ def emit(ev: LogEvent):
|
|||||||
- `colors`: the mutable color palette. Colors are bare SGR parameter strings (e.g. `"1;34"`, `"38;5;226"`) without escape framing. Attributes are read at render time, so assignments (`colors.action = "36"`) and additions (`colors.session = "38;5;226"`) take effect immediately.
|
- `colors`: the mutable color palette. Colors are bare SGR parameter strings (e.g. `"1;34"`, `"38;5;226"`) without escape framing. Attributes are read at render time, so assignments (`colors.action = "36"`) and additions (`colors.session = "38;5;226"`) take effect immediately.
|
||||||
- `Line`: builds a terminal string part by part. Calling it appends content (`str`-converted); `.<colorname>` arms a palette color for the next call only, and the reset is folded into a single escape sequence with whatever color comes next. `width=`/`align=` pad by display width; `str(line)` finishes the line and restores default colors.
|
- `Line`: builds a terminal string part by part. Calling it appends content (`str`-converted); `.<colorname>` arms a palette color for the next call only, and the reset is folded into a single escape sequence with whatever color comes next. `width=`/`align=` pad by display width; `str(line)` finishes the line and restores default colors.
|
||||||
- `strip_ansi`, `displaywidth` (wide chars and emoji count correctly) and `pad` for working with pre-colored strings.
|
- `strip_ansi`, `displaywidth` (wide chars and emoji count correctly) and `pad` for working with pre-colored strings.
|
||||||
|
- `use_color(stream)`: the color-support test used by Kanta's own output — honors `NO_COLOR`/`FORCE_COLOR`, then `stream.isatty()`, then the journald `JOURNAL_STREAM` device/inode match.
|
||||||
|
|
||||||
## Migrations
|
## Migrations
|
||||||
|
|
||||||
|
|||||||
+3
-3
@@ -43,13 +43,13 @@ Let `cutoff = now - retention`. All planning happens on the in-memory bytes of t
|
|||||||
4. **Copy the original aside.** The main file is copied with `shutil.copy2` directly to its final `{stem}@{ts}.kantadb` name. On filesystems with copy-on-write this performs a cheap reflink copy. A failure here aborts rotation with the original file intact.
|
4. **Copy the original aside.** The main file is copied with `shutil.copy2` directly to its final `{stem}@{ts}.kantadb` name. On filesystems with copy-on-write this performs a cheap reflink copy. A failure here aborts rotation with the original file intact.
|
||||||
|
|
||||||
5. **Rewrite the main file in place.** On the locked file descriptor the content is replaced (seek 0, truncate, write, fsync) with, in order:
|
5. **Rewrite the main file in place.** On the locked file descriptor the content is replaced (seek 0, truncate, write, fsync) with, in order:
|
||||||
1. A **snapshot of the state at the cutoff**, stamped with the schema version and modification time in effect at the cutoff. Its `ts` is the `ts` of the last pre-cutoff record — the same timestamp used in the rotated filename. This snapshot is the new replay base and carries the version forward so migrations are not re-run; it is always written.
|
1. A **snapshot of the state at the cutoff**, stamped with the schema version and modification time in effect at the cutoff. Its `ts` is the `ts` of the last pre-cutoff record — the same timestamp used in the rotated filename. Its `r` is the absolute revision of the cutoff state (the number of change records before the cutoff position in the original file), so revision numbers keep referring to the same state positions after the rotation. This snapshot is the new replay base and carries the version forward so migrations are not re-run; it is always written.
|
||||||
2. The retained change records (`ts >= cutoff`), recreated record by record — no internal snapshots are carried over.
|
2. The retained change records (`ts >= cutoff`), recreated record by record — no internal snapshots are carried over.
|
||||||
3. A **final snapshot** of the state after the last retained record, written only when enough changes were retained to warrant one (the same policy as regular snapshot writes). If no records survived the cutoff, the new file consists of the single leading snapshot and nothing else — the steady state for databases whose history has fully aged out.
|
3. A **final snapshot** of the state after the last retained record, written only when enough changes were retained to warrant one (the same policy as regular snapshot writes); its `r` is the total number of change records in the original file. If no records survived the cutoff, the new file consists of the single leading snapshot and nothing else — the steady state for databases whose history has fully aged out.
|
||||||
|
|
||||||
6. **Trim the rotated copy.** The rotated file is truncated to `cutoff_end` bytes, so it contains only the dropped history and does not duplicate the records retained in the main file. The cut is at a frame boundary, so the rotated file remains a valid, replayable database on its own. This happens only after step 5's fsync, so until then the rotated file still holds the complete original content as a crash-recovery anchor.
|
6. **Trim the rotated copy.** The rotated file is truncated to `cutoff_end` bytes, so it contains only the dropped history and does not duplicate the records retained in the main file. The cut is at a frame boundary, so the rotated file remains a valid, replayable database on its own. This happens only after step 5's fsync, so until then the rotated file still holds the complete original content as a crash-recovery anchor.
|
||||||
|
|
||||||
7. **Continue normal open.** Replay and migrations proceed on the same locked file. Because the leading snapshot carries the current version, migrations run exactly as they would have against the old content.
|
7. **Continue normal open.** Replay and migrations proceed on the same locked file. Because the leading snapshot carries the current version, migrations run exactly as they would have against the old content. The rotation's snapshots are stamped with the current meta format version (`k`), so rotating a legacy (pre-revision) file also performs the k1 meta migration — revision numbers are counted from record positions before the rotation (see [Database format](database.md#meta-format-migrations)).
|
||||||
|
|
||||||
Failure rule: any error before step 5 leaves the main file byte-identical (only an extra copy exists). A crash during step 5 may leave the main file torn, but the rotated copy still holds the complete original content — recovery is copying it back. After step 6 the split is complete and both files are consistent.
|
Failure rule: any error before step 5 leaves the main file byte-identical (only an extra copy exists). A crash during step 5 may leave the main file torn, but the rotated copy still holds the complete original content — recovery is copying it back. After step 6 the split is complete and both files are consistent.
|
||||||
|
|
||||||
|
|||||||
@@ -1,4 +1,5 @@
|
|||||||
from .callbacks import DictPrev, DictState, LogFmt
|
from .callbacks import DictPrev, DictState, LogFmt
|
||||||
|
from .diff import diff, patch
|
||||||
from .exceptions import DatabaseError
|
from .exceptions import DatabaseError
|
||||||
from .kanta import Kanta
|
from .kanta import Kanta
|
||||||
from .logging import LogEvent, configure_logging
|
from .logging import LogEvent, configure_logging
|
||||||
@@ -8,6 +9,8 @@ __all__ = [
|
|||||||
"Kanta",
|
"Kanta",
|
||||||
"DatabaseError",
|
"DatabaseError",
|
||||||
"configure_logging",
|
"configure_logging",
|
||||||
|
"diff",
|
||||||
|
"patch",
|
||||||
# Callback argument types
|
# Callback argument types
|
||||||
"DictPrev",
|
"DictPrev",
|
||||||
"DictState",
|
"DictState",
|
||||||
|
|||||||
+296
-55
@@ -6,34 +6,45 @@ import argparse
|
|||||||
import asyncio
|
import asyncio
|
||||||
import contextlib
|
import contextlib
|
||||||
import importlib
|
import importlib
|
||||||
|
import importlib.metadata
|
||||||
import importlib.util
|
import importlib.util
|
||||||
import logging
|
|
||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
|
from datetime import UTC, datetime
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
import msgspec
|
import msgspec
|
||||||
|
|
||||||
from kanta import Kanta
|
from kanta import Kanta
|
||||||
from kanta.callbacks import InjectionContext
|
from kanta.callbacks import InjectionContext, callback_error_reporter
|
||||||
from kanta.exceptions import DatabaseError, DataIntegrityError, ReplayError
|
from kanta.exceptions import DatabaseError, DataIntegrityError, ReplayError
|
||||||
from kanta.logging import LogEvent, emit_event, migration_logger
|
from kanta.grep import GrepPattern, evaluate, evaluate_snapshot, snapshot_record_lines
|
||||||
|
from kanta.logging import (
|
||||||
|
LogEvent,
|
||||||
|
emit_event,
|
||||||
|
format_action_header,
|
||||||
|
format_diff,
|
||||||
|
migration_logger,
|
||||||
|
)
|
||||||
from kanta.replaylog import (
|
from kanta.replaylog import (
|
||||||
|
Event,
|
||||||
RangeNotFoundError,
|
RangeNotFoundError,
|
||||||
Selection,
|
Selection,
|
||||||
SnapshotEvent,
|
SnapshotEvent,
|
||||||
_snapshot_lines,
|
_snapshot_lines,
|
||||||
end_of_file,
|
end_of_file,
|
||||||
|
parse_time_spec,
|
||||||
record_change_event,
|
record_change_event,
|
||||||
record_label,
|
record_label,
|
||||||
replay_events,
|
replay_events,
|
||||||
scan_events,
|
scan_events,
|
||||||
select,
|
select_all,
|
||||||
|
slice_items,
|
||||||
)
|
)
|
||||||
from kanta.serialization import Serializer, dict_to_struct, struct_to_dict
|
from kanta.serialization import Serializer, dict_to_struct, struct_to_dict
|
||||||
from kanta.structs import ChangeRecord, Snapshot
|
from kanta.structs import ChangeRecord, Snapshot
|
||||||
from kanta.tty import Line
|
from kanta.tty import Line, strip_ansi, use_color
|
||||||
|
|
||||||
EXIT_SUCCESS = 0
|
EXIT_SUCCESS = 0
|
||||||
EXIT_GENERIC = 1
|
EXIT_GENERIC = 1
|
||||||
@@ -42,7 +53,18 @@ EXIT_PARSE_ERROR = 10
|
|||||||
EXIT_MIGRATION_ERROR = 20
|
EXIT_MIGRATION_ERROR = 20
|
||||||
EXIT_VALIDATION_ERROR = 21
|
EXIT_VALIDATION_ERROR = 21
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
|
||||||
|
def _print(*args: Any) -> None:
|
||||||
|
"""Print to stderr, stripping ANSI codes when the stream has no color support.
|
||||||
|
|
||||||
|
Color detection runs per call so redirected or reassigned ``sys.stderr``
|
||||||
|
(and environment changes) are honored; ANSI codes are stripped after
|
||||||
|
formatting, not by formatting differently.
|
||||||
|
"""
|
||||||
|
text = " ".join(str(arg) for arg in args)
|
||||||
|
if not use_color():
|
||||||
|
text = strip_ansi(text)
|
||||||
|
print(text, file=sys.stderr)
|
||||||
|
|
||||||
|
|
||||||
class _CliError(Exception):
|
class _CliError(Exception):
|
||||||
@@ -148,10 +170,26 @@ def _format_ts(dt) -> str:
|
|||||||
return dt.replace(tzinfo=None, microsecond=0).isoformat(sep=" ")
|
return dt.replace(tzinfo=None, microsecond=0).isoformat(sep=" ")
|
||||||
|
|
||||||
|
|
||||||
|
def _package_version() -> str:
|
||||||
|
"""Return the installed package version, or ``"unknown"`` from a source tree."""
|
||||||
|
try:
|
||||||
|
return importlib.metadata.version("kanta")
|
||||||
|
except importlib.metadata.PackageNotFoundError:
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
prog="kanta",
|
prog="kanta",
|
||||||
description="Read a kantadb file and print each change record to the console.",
|
description=(
|
||||||
|
f"kanta {_package_version()} - read a kantadb file and print each"
|
||||||
|
" change record to the console."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--version",
|
||||||
|
action="version",
|
||||||
|
version=f"%(prog)s {_package_version()}",
|
||||||
)
|
)
|
||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"file",
|
"file",
|
||||||
@@ -201,15 +239,74 @@ def _parse_args(argv: list[str] | None = None) -> argparse.Namespace:
|
|||||||
parser.add_argument(
|
parser.add_argument(
|
||||||
"-r",
|
"-r",
|
||||||
"--range",
|
"--range",
|
||||||
|
action="append",
|
||||||
help=(
|
help=(
|
||||||
"Python-style range to process. Units: plain number = change index,"
|
"Python-style range to process; repeatable. Unit bounds index"
|
||||||
" lN = line number, sN = snapshot, vN = version. Negative snapshot"
|
" into the database file: lN = line number, sN = snapshot,"
|
||||||
" values count from the end (s-1 is the last snapshot). Use ':' for"
|
" vN = version, rN = revision (rN is the record that produced"
|
||||||
|
" state N, with its state). Plain numbers instead index into"
|
||||||
|
" the filtered output: they select among the records left after"
|
||||||
|
" all unit ranges and --grep patterns. The -o dump and"
|
||||||
|
" migrations operate on the state after the last printed"
|
||||||
|
" record. Repeated ranges are restrictive: only the common"
|
||||||
|
" subset is printed. Negative"
|
||||||
|
" snapshot values count from the end (s-1 is the last"
|
||||||
|
" snapshot); negative plain numbers count from the end of the"
|
||||||
|
" filtered output. Time specs (UTC) filter records by"
|
||||||
|
" timestamp: relative '15min', '1h', '7d' (N plus one of s, min,"
|
||||||
|
" h, d, w) cover the last N units ending now; an ISO point"
|
||||||
|
" ('2026-09-28', '2026-09-28T10:30', '20260928T1030') covers its"
|
||||||
|
" natural period by precision; an ISO range 'a/b' is inclusive"
|
||||||
|
" and either side may be open ('2026-09-24/'). Use ':' for"
|
||||||
" half-open ranges and '..' for inclusive end ranges. Examples:"
|
" half-open ranges and '..' for inclusive end ranges. Examples:"
|
||||||
" '2:5', '2..5', 'l10:l20', 's1:s3', 'v0:v2', 's-1:', ':-1', '-1'."
|
" '2:5', '2..5', 'l10:l20', 's1:s3', 'v0:v2', 'r10:r20', 's-1:',"
|
||||||
|
" ':-1', '-1', '-r r1:r5 -r 0'."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-g",
|
||||||
|
"--grep",
|
||||||
|
action="append",
|
||||||
|
metavar="PATTERN",
|
||||||
|
help=(
|
||||||
|
"Print only change records matching PATTERN, structurally and"
|
||||||
|
" case-insensitively; matched regions get a yellow background,"
|
||||||
|
" and on a match the whole record is printed, not just the"
|
||||||
|
" matching line. Snapshots are matched against their state and"
|
||||||
|
" show the matched records of it as compact lines. Repeatable:"
|
||||||
|
" every pattern must match somewhere"
|
||||||
|
" in the same record, but different patterns may match different"
|
||||||
|
" lines of it. A bare pattern matches the action or user as a"
|
||||||
|
" substring, a dotted path by element (each element in full,"
|
||||||
|
" unless it uses wildcards: 'users' matches 'users' anywhere but"
|
||||||
|
" not 'foousers', 'us*' does), or a value (strings by substring,"
|
||||||
|
" other values in full: 'true' matches a boolean, 'tru' does"
|
||||||
|
" not). The 'path=value' form requires the path and the value"
|
||||||
|
" to match within the same change line; use '=value' or 'path='"
|
||||||
|
" to match values or paths only. With -k, logfmt-prettified"
|
||||||
|
" values and users match alongside the raw ones. Examples:"
|
||||||
|
" --grep alice, --grep 'users.*.email', --grep"
|
||||||
|
" 'users.alice.admin=true', --grep create_user --grep"
|
||||||
|
" '@example.com'."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--context",
|
||||||
|
type=int,
|
||||||
|
default=0,
|
||||||
|
metavar="N",
|
||||||
|
help=(
|
||||||
|
"Print N surrounding records around each printed item, GNU grep"
|
||||||
|
" style. Applied last, after --grep filtering and plain-number"
|
||||||
|
" -r slicing: context records are drawn from the whole file,"
|
||||||
|
" expanding whatever would be shown past any range restriction,"
|
||||||
|
" and overlapping expansions merge. Without any other filter"
|
||||||
|
" this has no visible effect."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
if args.context < 0:
|
||||||
|
parser.error("--context must not be negative")
|
||||||
if args.kanta and (args.data or args.migrations):
|
if args.kanta and (args.data or args.migrations):
|
||||||
parser.error("-k/--kanta cannot be used together with -d or -m")
|
parser.error("-k/--kanta cannot be used together with -d or -m")
|
||||||
return args
|
return args
|
||||||
@@ -221,27 +318,40 @@ def _print_change_log(
|
|||||||
previous: dict[str, Any],
|
previous: dict[str, Any],
|
||||||
current: dict[str, Any],
|
current: dict[str, Any],
|
||||||
kanta: Kanta[Any],
|
kanta: Kanta[Any],
|
||||||
|
revision: int,
|
||||||
|
highlight: Any = None,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Log a single change record to stderr.
|
"""Log a single change record to stderr.
|
||||||
|
|
||||||
The record is dispatched as a :class:`LogEvent` through the Kanta
|
The record is dispatched as a :class:`LogEvent` through the Kanta
|
||||||
object's logemit handlers; the CLI's own rendering (with the ``l<N>``
|
object's logemit handlers; the CLI's own rendering (with the ``l<N>``
|
||||||
label and timestamp) is the fallback when no handler claims the event.
|
label and timestamp) is the fallback when no handler claims the event.
|
||||||
|
The fallback header shows the revision of the state the record produces
|
||||||
|
as ``r<N>``. ``highlight`` is an optional
|
||||||
|
:class:`kanta.grep.GrepHighlighter` with the record's matched regions,
|
||||||
|
applied to the fallback rendering.
|
||||||
"""
|
"""
|
||||||
event = record_change_event(record, previous, current, kanta)
|
event = record_change_event(record, previous, current, kanta)
|
||||||
|
|
||||||
def render(ev) -> None:
|
def render(ev) -> None:
|
||||||
ts = _format_ts(record.ts)
|
ts = _format_ts(record.ts)
|
||||||
lines = ev.diff_lines
|
if highlight is not None:
|
||||||
if not lines:
|
header = format_action_header(
|
||||||
print(f"{label} {ts} {ev.header}", file=sys.stderr)
|
ev.action or "", ev.user, ev.extra, highlight=highlight
|
||||||
elif len(lines) == 1:
|
)
|
||||||
print(f"{label} {ts} {ev.header}{lines[0]}", file=sys.stderr)
|
lines = format_diff(ev.diff, ev.previous, ev.logfmt, highlight=highlight)
|
||||||
else:
|
else:
|
||||||
print(f"{label} {ts} {ev.header}", file=sys.stderr)
|
header, lines = ev.header, ev.diff_lines
|
||||||
|
header += str(Line().target(f" r{revision}"))
|
||||||
|
if not lines:
|
||||||
|
_print(f"{label} {ts} {header}")
|
||||||
|
elif len(lines) == 1:
|
||||||
|
_print(f"{label} {ts} {header}{lines[0]}")
|
||||||
|
else:
|
||||||
|
_print(f"{label} {ts} {header}")
|
||||||
for line in lines:
|
for line in lines:
|
||||||
print(line, file=sys.stderr)
|
_print(line)
|
||||||
print(file=sys.stderr)
|
_print()
|
||||||
|
|
||||||
emit_event(
|
emit_event(
|
||||||
event,
|
event,
|
||||||
@@ -309,8 +419,8 @@ def _print_snapshot_indicator(
|
|||||||
) -> None:
|
) -> None:
|
||||||
"""Print a snapshot indicator line to stderr.
|
"""Print a snapshot indicator line to stderr.
|
||||||
|
|
||||||
``snapshot s<N>`` is rendered in bright white; the version, optional mtime
|
``snapshot s<N>`` is rendered in bright white; the version, optional
|
||||||
and data size are printed in normal and dark colors respectively.
|
mtime and data size are printed in normal and dark colors respectively.
|
||||||
"""
|
"""
|
||||||
ts = _format_ts(snap.ts)
|
ts = _format_ts(snap.ts)
|
||||||
line = Line().snapshot("snapshot").snapshot(f" s{index}")
|
line = Line().snapshot("snapshot").snapshot(f" s{index}")
|
||||||
@@ -319,7 +429,32 @@ def _print_snapshot_indicator(
|
|||||||
line.target(f" {_format_ts(snap.m)}")
|
line.target(f" {_format_ts(snap.m)}")
|
||||||
size = len(serializer.encode(snap.state))
|
size = len(serializer.encode(snap.state))
|
||||||
line.path_prefix(f" {_format_size(size)}")
|
line.path_prefix(f" {_format_size(size)}")
|
||||||
print(f"{label} {ts} {line}", file=sys.stderr)
|
_print(f"{label} {ts} {line}")
|
||||||
|
|
||||||
|
|
||||||
|
def _in_time_windows(ts: datetime, windows: list) -> bool:
|
||||||
|
"""Return whether a record timestamp falls within every time window."""
|
||||||
|
if ts.tzinfo is None:
|
||||||
|
ts = ts.replace(tzinfo=UTC)
|
||||||
|
return all(
|
||||||
|
(start is None or ts >= start) and (end is None or ts < end)
|
||||||
|
for start, end in windows
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _print_snapshot_matches(
|
||||||
|
state: dict[str, Any], kanta: Kanta[Any], highlight: Any
|
||||||
|
) -> None:
|
||||||
|
"""Print the grep-matched records of a snapshot's state, change-log style.
|
||||||
|
|
||||||
|
Each matched record is one ``path.to.record = {compact value}`` line
|
||||||
|
with normal path coloring and the matched regions highlighted.
|
||||||
|
"""
|
||||||
|
logfmt = kanta._impl.callback_registry.build_logfmt(
|
||||||
|
InjectionContext(kanta=kanta, previous_state=state, current_state=state)
|
||||||
|
)
|
||||||
|
for line in snapshot_record_lines(state, highlight, logfmt):
|
||||||
|
_print(line)
|
||||||
|
|
||||||
|
|
||||||
async def _log_migration(
|
async def _log_migration(
|
||||||
@@ -337,13 +472,11 @@ async def _log_migration(
|
|||||||
"""
|
"""
|
||||||
registry = kanta._impl.callback_registry
|
registry = kanta._impl.callback_registry
|
||||||
if registry.has("logmigr"):
|
if registry.has("logmigr"):
|
||||||
try:
|
await registry.invoke(
|
||||||
await registry.invoke(
|
"logmigr",
|
||||||
"logmigr",
|
InjectionContext(kanta=kanta, report=result),
|
||||||
InjectionContext(kanta=kanta, report=result),
|
on_error=callback_error_reporter("logmigr"),
|
||||||
)
|
)
|
||||||
except Exception:
|
|
||||||
_logger.exception("logmigr callback failed")
|
|
||||||
return
|
return
|
||||||
if quiet:
|
if quiet:
|
||||||
return
|
return
|
||||||
@@ -359,7 +492,7 @@ async def _log_migration(
|
|||||||
migrations=descriptions,
|
migrations=descriptions,
|
||||||
),
|
),
|
||||||
registry.logemit_handlers,
|
registry.logemit_handlers,
|
||||||
fallback=lambda ev: print(ev.header, file=sys.stderr),
|
fallback=lambda ev: _print(ev.header),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
@@ -434,40 +567,139 @@ async def _run(args: argparse.Namespace) -> int:
|
|||||||
line: idx for idx, line in enumerate(_snapshot_lines(events))
|
line: idx for idx, line in enumerate(_snapshot_lines(events))
|
||||||
}
|
}
|
||||||
|
|
||||||
# Resolve -r into a line range or a single snapshot selection.
|
# Resolve -r specs into a line range (unit specs) or a single
|
||||||
|
# snapshot selection, plus plain specs applied to the output items
|
||||||
|
# and time specs matched per record.
|
||||||
try:
|
try:
|
||||||
selection = (
|
selection, plain_specs, time_specs = (
|
||||||
select(args.range, events, change_count)
|
select_all(args.range, events, change_count)
|
||||||
if args.range is not None
|
if args.range
|
||||||
else Selection(0, end_of_file(events))
|
else (Selection(0, end_of_file(events)), [], [])
|
||||||
)
|
)
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
time_windows = [parse_time_spec(spec, now) for spec in time_specs]
|
||||||
except RangeNotFoundError as exc:
|
except RangeNotFoundError as exc:
|
||||||
raise _CliError(str(exc), EXIT_RANGE_ERROR) from exc
|
raise _CliError(str(exc), EXIT_RANGE_ERROR) from exc
|
||||||
except ValueError as exc:
|
except ValueError as exc:
|
||||||
raise _CliError(f"Invalid --range value: {exc}") from exc
|
raise _CliError(f"Invalid --range value: {exc}") from exc
|
||||||
|
|
||||||
if selection.snapshot is not None:
|
grep_patterns = [GrepPattern.parse(p) for p in args.grep or ()]
|
||||||
|
|
||||||
|
if selection.snapshot is not None and not args.context and not time_windows:
|
||||||
snap_event = selection.snapshot
|
snap_event = selection.snapshot
|
||||||
state = snap_event.snap.state
|
state = snap_event.snap.state
|
||||||
version = snap_event.snap.v
|
version = snap_event.snap.v
|
||||||
if not args.quiet:
|
if not args.quiet:
|
||||||
_print_snapshot_indicator(
|
highlight = None
|
||||||
record_label(snap_event.line_number, snap_event.record_index),
|
if grep_patterns:
|
||||||
snap_event.snap,
|
logfmt = kanta._impl.callback_registry.build_logfmt(
|
||||||
snapshot_line_to_index[snap_event.line_number],
|
InjectionContext(
|
||||||
kanta._impl.serializer,
|
kanta=kanta, previous_state=state, current_state=state
|
||||||
)
|
)
|
||||||
print(file=sys.stderr)
|
)
|
||||||
|
highlight = evaluate_snapshot(state, grep_patterns, logfmt=logfmt)
|
||||||
|
if not grep_patterns or highlight is not None:
|
||||||
|
_print_snapshot_indicator(
|
||||||
|
record_label(snap_event.line_number, snap_event.record_index),
|
||||||
|
snap_event.snap,
|
||||||
|
snapshot_line_to_index[snap_event.line_number],
|
||||||
|
kanta._impl.serializer,
|
||||||
|
)
|
||||||
|
if highlight is not None:
|
||||||
|
_print_snapshot_matches(state, kanta, highlight)
|
||||||
|
_print()
|
||||||
else:
|
else:
|
||||||
# Replay up to the range end, printing logs within the range.
|
# Replay, collecting the item pool and tracking which items
|
||||||
|
# survive the range window and --grep. With --context the pool
|
||||||
|
# spans the whole file, so context can expand whatever would be
|
||||||
|
# shown past any range restriction; otherwise the pool is
|
||||||
|
# limited to the window.
|
||||||
state = {}
|
state = {}
|
||||||
version = 0
|
version = 0
|
||||||
printed = False
|
window_state: dict[str, Any] = {}
|
||||||
for event, previous, current in replay_events(events, selection.end_line):
|
window_version = 0
|
||||||
|
pool_end = end_of_file(events) if args.context else selection.end_line
|
||||||
|
pool: list[tuple[Event, dict[str, Any] | None, dict[str, Any], Any]] = []
|
||||||
|
matched: list[int] = []
|
||||||
|
for event, previous, current in replay_events(events, pool_end):
|
||||||
|
if event.line_number < selection.end_line:
|
||||||
|
window_state = current
|
||||||
|
window_version = event.version
|
||||||
state = current
|
state = current
|
||||||
version = event.version
|
version = event.version
|
||||||
if event.line_number < selection.start_line or args.quiet:
|
if (
|
||||||
|
args.quiet
|
||||||
|
or event.line_number < selection.start_line
|
||||||
|
or event.line_number >= selection.end_line
|
||||||
|
):
|
||||||
|
if not args.quiet and args.context:
|
||||||
|
pool.append((event, previous, current, None))
|
||||||
continue
|
continue
|
||||||
|
highlight = None
|
||||||
|
if isinstance(event, SnapshotEvent):
|
||||||
|
is_match = _in_time_windows(event.snap.ts, time_windows)
|
||||||
|
if is_match and grep_patterns:
|
||||||
|
logfmt = kanta._impl.callback_registry.build_logfmt(
|
||||||
|
InjectionContext(
|
||||||
|
kanta=kanta,
|
||||||
|
previous_state=current,
|
||||||
|
current_state=current,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
highlight = evaluate_snapshot(
|
||||||
|
current, grep_patterns, logfmt=logfmt
|
||||||
|
)
|
||||||
|
is_match = highlight is not None
|
||||||
|
else:
|
||||||
|
assert previous is not None
|
||||||
|
if grep_patterns:
|
||||||
|
# Build the same logfmt the rendering uses, so both
|
||||||
|
# raw and prettified values are matched.
|
||||||
|
logfmt = kanta._impl.callback_registry.build_logfmt(
|
||||||
|
InjectionContext(
|
||||||
|
kanta=kanta,
|
||||||
|
previous_state=previous,
|
||||||
|
current_state=current,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
highlight = evaluate(
|
||||||
|
event.record, previous, grep_patterns, logfmt=logfmt
|
||||||
|
)
|
||||||
|
is_match = (
|
||||||
|
highlight is not None or not grep_patterns
|
||||||
|
) and _in_time_windows(event.record.ts, time_windows)
|
||||||
|
pool.append((event, previous, current, highlight if is_match else None))
|
||||||
|
if is_match:
|
||||||
|
matched.append(len(pool) - 1)
|
||||||
|
# Plain -r specs slice the filtered output items.
|
||||||
|
if plain_specs:
|
||||||
|
try:
|
||||||
|
item_start, item_end = slice_items(plain_specs, len(matched))
|
||||||
|
except RangeNotFoundError as exc:
|
||||||
|
raise _CliError(str(exc), EXIT_RANGE_ERROR) from exc
|
||||||
|
except ValueError as exc:
|
||||||
|
raise _CliError(f"Invalid --range value: {exc}") from exc
|
||||||
|
matched = matched[item_start:item_end]
|
||||||
|
# --context expands around the surviving items within the pool,
|
||||||
|
# free of the grep and plain-spec filters; overlaps merge.
|
||||||
|
if args.context and matched:
|
||||||
|
expanded: set[int] = set()
|
||||||
|
for idx in matched:
|
||||||
|
lo = max(0, idx - args.context)
|
||||||
|
hi = min(len(pool), idx + args.context + 1)
|
||||||
|
expanded.update(range(lo, hi))
|
||||||
|
matched = sorted(expanded)
|
||||||
|
items = [pool[idx] for idx in matched]
|
||||||
|
# The dump and migrations operate on the state after the last
|
||||||
|
# printed item; an empty display keeps the window-end state.
|
||||||
|
if items:
|
||||||
|
last_event, _, last_current, _ = items[-1]
|
||||||
|
state = last_current
|
||||||
|
version = last_event.version
|
||||||
|
else:
|
||||||
|
state = window_state
|
||||||
|
version = window_version
|
||||||
|
for event, previous, current, highlight in items:
|
||||||
label = record_label(event.line_number, event.record_index)
|
label = record_label(event.line_number, event.record_index)
|
||||||
if isinstance(event, SnapshotEvent):
|
if isinstance(event, SnapshotEvent):
|
||||||
_print_snapshot_indicator(
|
_print_snapshot_indicator(
|
||||||
@@ -476,14 +708,23 @@ async def _run(args: argparse.Namespace) -> int:
|
|||||||
snapshot_line_to_index[event.line_number],
|
snapshot_line_to_index[event.line_number],
|
||||||
kanta._impl.serializer,
|
kanta._impl.serializer,
|
||||||
)
|
)
|
||||||
|
if highlight is not None:
|
||||||
|
_print_snapshot_matches(current, kanta, highlight)
|
||||||
else:
|
else:
|
||||||
assert previous is not None
|
assert previous is not None
|
||||||
_print_change_log(label, event.record, previous, current, kanta)
|
_print_change_log(
|
||||||
printed = True
|
label,
|
||||||
if printed:
|
event.record,
|
||||||
print(file=sys.stderr)
|
previous,
|
||||||
|
current,
|
||||||
|
kanta,
|
||||||
|
event.revision,
|
||||||
|
highlight,
|
||||||
|
)
|
||||||
|
if items:
|
||||||
|
_print()
|
||||||
|
|
||||||
# Apply optional migrations to the range-end state.
|
# Apply optional migrations to the selected end state.
|
||||||
if kanta._impl.migrations is not None:
|
if kanta._impl.migrations is not None:
|
||||||
try:
|
try:
|
||||||
previous_version = version
|
previous_version = version
|
||||||
@@ -518,7 +759,7 @@ async def _run(args: argparse.Namespace) -> int:
|
|||||||
# The file was already fully decoded and validated above with
|
# The file was already fully decoded and validated above with
|
||||||
# the object's own serializer, and its migrations were applied
|
# the object's own serializer, and its migrations were applied
|
||||||
# to the state; no need to re-open through a new instance.
|
# to the state; no need to re-open through a new instance.
|
||||||
print(f"{data}", file=sys.stderr)
|
_print(f"{data}")
|
||||||
output_state = struct_to_dict(data, serializer=kanta._impl.serializer)
|
output_state = struct_to_dict(data, serializer=kanta._impl.serializer)
|
||||||
else:
|
else:
|
||||||
kanta_typed = Kanta(
|
kanta_typed = Kanta(
|
||||||
@@ -526,7 +767,7 @@ async def _run(args: argparse.Namespace) -> int:
|
|||||||
)
|
)
|
||||||
try:
|
try:
|
||||||
await kanta_typed.open(create=False, readonly=True, log=False)
|
await kanta_typed.open(create=False, readonly=True, log=False)
|
||||||
print(f"{data}", file=sys.stderr)
|
_print(f"{data}")
|
||||||
except (msgspec.ValidationError, msgspec.DecodeError) as exc:
|
except (msgspec.ValidationError, msgspec.DecodeError) as exc:
|
||||||
raise _CliError(
|
raise _CliError(
|
||||||
f"Validation error: {exc}", EXIT_VALIDATION_ERROR
|
f"Validation error: {exc}", EXIT_VALIDATION_ERROR
|
||||||
@@ -577,7 +818,7 @@ def main(argv: list[str] | None = None) -> int:
|
|||||||
try:
|
try:
|
||||||
return asyncio.run(_run(args))
|
return asyncio.run(_run(args))
|
||||||
except _CliError as exc:
|
except _CliError as exc:
|
||||||
print(exc, file=sys.stderr)
|
_print(exc)
|
||||||
return exc.code
|
return exc.code
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+43
-6
@@ -54,7 +54,40 @@ def _state_tag(ann: Any) -> str | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
|
|
||||||
|
def describe_callback(callback: Callable[..., Any]) -> str:
|
||||||
|
"""Return ``name (docstring first line)`` identifying *callback*.
|
||||||
|
|
||||||
|
Used in failure messages so a bare log line names the function that
|
||||||
|
failed, e.g. ``myformatter (Concise log formatter)``. Callables without
|
||||||
|
a ``__name__`` (partials, callable instances, ...) are described by
|
||||||
|
their type name only: less information, but never wrong information.
|
||||||
|
"""
|
||||||
|
name = getattr(callback, "__name__", None)
|
||||||
|
if not isinstance(name, str):
|
||||||
|
return type(callback).__name__
|
||||||
|
doc = inspect.getdoc(callback)
|
||||||
|
if doc:
|
||||||
|
return f"{name} ({doc.splitlines()[0]})"
|
||||||
|
return name
|
||||||
|
|
||||||
|
|
||||||
|
def callback_error_reporter(
|
||||||
|
kind: str,
|
||||||
|
) -> Callable[[Exception, Callable[..., Any]], None]:
|
||||||
|
"""Return an ``on_error`` reporter for :meth:`CallbackRegistry.invoke`.
|
||||||
|
|
||||||
|
The returned callable logs ``Kanta.<kind> <name (docstring)> failed``
|
||||||
|
with the traceback for each failing callback; invoke continues with
|
||||||
|
the rest.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _report(callback_error: Exception, callback: Callable[..., Any]) -> None:
|
||||||
|
_logger.exception("Kanta.%s %s failed", kind, describe_callback(callback))
|
||||||
|
|
||||||
|
return _report
|
||||||
|
|
||||||
|
|
||||||
class LogFmt:
|
class LogFmt:
|
||||||
@@ -245,12 +278,14 @@ class CallbackRegistry:
|
|||||||
|
|
||||||
def build_logfmt(self, ctx: InjectionContext) -> Callable[[Any, str], str | None]:
|
def build_logfmt(self, ctx: InjectionContext) -> Callable[[Any, str], str | None]:
|
||||||
"""Build a chained formatter from registered logfmt callbacks."""
|
"""Build a chained formatter from registered logfmt callbacks."""
|
||||||
formatters: list[tuple[Callable[[Any, str], str | None], str | None]] = []
|
formatters: list[
|
||||||
|
tuple[Callable[[Any, str], str | None], str | None, Callable[..., Any]]
|
||||||
|
] = []
|
||||||
for spec in self._logfmt_callbacks:
|
for spec in self._logfmt_callbacks:
|
||||||
if isinstance(spec, _LogFmtClassSpec):
|
if isinstance(spec, _LogFmtClassSpec):
|
||||||
kwargs = self._build_kwargs(spec.inject_params, ctx)
|
kwargs = self._build_kwargs(spec.inject_params, ctx)
|
||||||
instance: Callable[[Any, str], str | None] = spec.cls(**kwargs)
|
instance: Callable[[Any, str], str | None] = spec.cls(**kwargs)
|
||||||
formatters.append((instance, spec.path))
|
formatters.append((instance, spec.path, spec.cls))
|
||||||
else:
|
else:
|
||||||
kwargs = self._build_kwargs(spec.inject_params, ctx)
|
kwargs = self._build_kwargs(spec.inject_params, ctx)
|
||||||
|
|
||||||
@@ -270,10 +305,10 @@ class CallbackRegistry:
|
|||||||
|
|
||||||
return formatter
|
return formatter
|
||||||
|
|
||||||
formatters.append((make_formatter(), spec.path))
|
formatters.append((make_formatter(), spec.path, spec.callback))
|
||||||
|
|
||||||
def format_value(value: Any, path: str) -> str | None:
|
def format_value(value: Any, path: str) -> str | None:
|
||||||
for fn, pattern in formatters:
|
for fn, pattern, callback in formatters:
|
||||||
if pattern is not None and path != pattern:
|
if pattern is not None and path != pattern:
|
||||||
continue
|
continue
|
||||||
try:
|
try:
|
||||||
@@ -281,7 +316,9 @@ class CallbackRegistry:
|
|||||||
except Exception:
|
except Exception:
|
||||||
# Formatting must never break functionality; a failing
|
# Formatting must never break functionality; a failing
|
||||||
# callback is reported and treated as a fall-through.
|
# callback is reported and treated as a fall-through.
|
||||||
_logger.exception("logfmt callback %r failed", fn)
|
_logger.exception(
|
||||||
|
"Kanta.logfmt %s failed", describe_callback(callback)
|
||||||
|
)
|
||||||
continue
|
continue
|
||||||
if resolved is not None:
|
if resolved is not None:
|
||||||
return resolved
|
return resolved
|
||||||
|
|||||||
+4
-4
@@ -49,16 +49,16 @@ def _diff(previous, current):
|
|||||||
return current
|
return current
|
||||||
|
|
||||||
|
|
||||||
def compute_diff(previous: dict, current: dict) -> dict | None:
|
def diff(previous: dict, current: dict) -> dict | None:
|
||||||
"""Compute a marshaled diff between two state dicts.
|
"""Compute a marshaled diff between two state dicts.
|
||||||
|
|
||||||
Returns None if there is no difference.
|
Returns None if there is no difference.
|
||||||
"""
|
"""
|
||||||
diff = _diff(previous, current)
|
result = _diff(previous, current)
|
||||||
return diff if diff is not _UNCHANGED else None
|
return result if result is not _UNCHANGED else None
|
||||||
|
|
||||||
|
|
||||||
def patch_state(state: dict, diff: dict) -> dict:
|
def patch(state: dict, diff: dict) -> dict:
|
||||||
"""Apply a marshaled diff to a state dict."""
|
"""Apply a marshaled diff to a state dict."""
|
||||||
return apply_diff(state, diff)
|
return apply_diff(state, diff)
|
||||||
|
|
||||||
|
|||||||
+1
-1
@@ -16,7 +16,7 @@ from pathlib import Path
|
|||||||
|
|
||||||
from kanta.exceptions import FileLockError
|
from kanta.exceptions import FileLockError
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
|
|
||||||
def _fatal(msg: str, *, db_path: Path | None = None) -> None:
|
def _fatal(msg: str, *, db_path: Path | None = None) -> None:
|
||||||
|
|||||||
+572
@@ -0,0 +1,572 @@
|
|||||||
|
"""Structural grep matching and match highlighting for the kanta CLI.
|
||||||
|
|
||||||
|
Backs the ``--grep`` option of the ``python -m kanta`` CLI. Patterns
|
||||||
|
match against the structure of a transaction — its action, user, and the
|
||||||
|
dotted paths and values of its diff — or against a snapshot's flattened
|
||||||
|
state, never against rendered output text. Matching is
|
||||||
|
case-insensitive. A matched change record is printed in full; a matched
|
||||||
|
snapshot shows only the matched lines of its state, rendered diff-style.
|
||||||
|
|
||||||
|
Path terms match element-wise: each dotted element of the term must match
|
||||||
|
a whole path element (``users`` matches ``users`` anywhere in the path
|
||||||
|
but not ``foousers``), unless the element uses shell wildcards
|
||||||
|
(``us*``). The term's elements match as a contiguous sequence, so
|
||||||
|
``users.*.email`` matches the path ``users.alice.email``. Only the
|
||||||
|
matched elements are highlighted.
|
||||||
|
|
||||||
|
Value terms match strings by substring and all other values (booleans,
|
||||||
|
numbers, null) only in full: ``true`` matches a boolean but ``tru`` does
|
||||||
|
not. A term with wildcards matches the whole value text. Matched
|
||||||
|
substrings — or the whole scalar — are highlighted. When a logfmt
|
||||||
|
formatter (``-k``) prettifies a value or the user, both the raw and the
|
||||||
|
prettified form are matched.
|
||||||
|
|
||||||
|
A matched record carries a :class:`GrepHighlighter`, which the
|
||||||
|
:kanta.logging formatters consult to wrap exactly the matched regions
|
||||||
|
with a yellow background.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import dataclasses
|
||||||
|
import fnmatch
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
from collections.abc import Iterator
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from kanta.logging import (
|
||||||
|
_USER_PATH,
|
||||||
|
_collect_changes,
|
||||||
|
_format_path,
|
||||||
|
_format_value,
|
||||||
|
_get_nested,
|
||||||
|
)
|
||||||
|
from kanta.serialization.base import unmarshal
|
||||||
|
from kanta.structs import ChangeRecord
|
||||||
|
from kanta.tty import (
|
||||||
|
ANSI_RE,
|
||||||
|
Line,
|
||||||
|
_parse_sgr,
|
||||||
|
_sgr_transition,
|
||||||
|
strip_ansi,
|
||||||
|
)
|
||||||
|
|
||||||
|
_GLOB_CHARS = frozenset("*?[")
|
||||||
|
|
||||||
|
# Marked regions: xterm256 black text (16, not theme-dependent "black") on
|
||||||
|
# xterm256 yellow (220 ~ #ffd700).
|
||||||
|
_MARK_FG = "38;5;16"
|
||||||
|
_MARK_BG = "48;5;220"
|
||||||
|
_SGR_RE = re.compile(r"\x1b\[([0-9;]*)m")
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_sgr(active: dict[str, str], escape: str) -> None:
|
||||||
|
"""Fold one escape sequence into the active SGR state; non-SGR ignored."""
|
||||||
|
match = _SGR_RE.fullmatch(escape)
|
||||||
|
if not match:
|
||||||
|
return
|
||||||
|
spec = match.group(1)
|
||||||
|
if not spec:
|
||||||
|
active.clear()
|
||||||
|
return
|
||||||
|
if "0" in spec.split(";"):
|
||||||
|
active.clear()
|
||||||
|
active.update(_parse_sgr(spec))
|
||||||
|
|
||||||
|
|
||||||
|
def mark_spans(styled: str, spans: list[tuple[int, int]]) -> str:
|
||||||
|
"""Wrap the given visible-text spans of a styled string with the mark colors.
|
||||||
|
|
||||||
|
``spans`` are ``(start, end)`` offsets into the visible text of
|
||||||
|
*styled*; ANSI sequences are not counted. Overlapping and adjacent
|
||||||
|
spans are merged first, so overlapping matches from different patterns
|
||||||
|
produce one continuous highlight. Marked regions get black text on a
|
||||||
|
yellow background; the active SGR state is tracked through *styled* so
|
||||||
|
the original styling — foreground included — is restored exactly where
|
||||||
|
each mark ends.
|
||||||
|
"""
|
||||||
|
spans = _merge_spans(spans)
|
||||||
|
if not spans:
|
||||||
|
return styled
|
||||||
|
out: list[str] = []
|
||||||
|
plain_pos = 0
|
||||||
|
prev_end = 0
|
||||||
|
active: dict[str, str] = {}
|
||||||
|
for match in ANSI_RE.finditer(styled):
|
||||||
|
out.append(
|
||||||
|
_wrap_run(styled[prev_end : match.start()], plain_pos, spans, active)
|
||||||
|
)
|
||||||
|
plain_pos += match.start() - prev_end
|
||||||
|
out.append(match.group(0))
|
||||||
|
_apply_sgr(active, match.group(0))
|
||||||
|
prev_end = match.end()
|
||||||
|
out.append(_wrap_run(styled[prev_end:], plain_pos, spans, active))
|
||||||
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _merge_spans(spans: list[tuple[int, int]]) -> list[tuple[int, int]]:
|
||||||
|
"""Return *spans* sorted, with overlapping and adjacent spans merged."""
|
||||||
|
merged: list[list[int]] = []
|
||||||
|
for start, end in sorted(spans):
|
||||||
|
if start >= end:
|
||||||
|
continue
|
||||||
|
if merged and start <= merged[-1][1]:
|
||||||
|
merged[-1][1] = max(end, merged[-1][1])
|
||||||
|
else:
|
||||||
|
merged.append([start, end])
|
||||||
|
return [(start, end) for start, end in merged]
|
||||||
|
|
||||||
|
|
||||||
|
def _wrap_run(
|
||||||
|
run: str, plain_start: int, spans: list[tuple[int, int]], active: dict[str, str]
|
||||||
|
) -> str:
|
||||||
|
"""Wrap the intersections of *spans* with one escape-free text run.
|
||||||
|
|
||||||
|
*active* is the SGR state in effect at this run; each mark sets the
|
||||||
|
black-on-yellow mark state and each unmark restores *active* exactly.
|
||||||
|
"""
|
||||||
|
if not run:
|
||||||
|
return run
|
||||||
|
mark_state = {**active, "fg": _MARK_FG, "bg": _MARK_BG}
|
||||||
|
out: list[str] = []
|
||||||
|
pos = 0
|
||||||
|
for start, end in spans:
|
||||||
|
s = max(start - plain_start, 0)
|
||||||
|
e = min(end - plain_start, len(run))
|
||||||
|
if s >= e or e <= pos:
|
||||||
|
continue
|
||||||
|
out.append(run[pos:s])
|
||||||
|
out.append(_sgr_transition(active, mark_state))
|
||||||
|
out.append(run[s:e])
|
||||||
|
out.append(_sgr_transition(mark_state, active))
|
||||||
|
pos = e
|
||||||
|
out.append(run[pos:])
|
||||||
|
return "".join(out)
|
||||||
|
|
||||||
|
|
||||||
|
def _find_spans(text: str, needle: str) -> list[tuple[int, int]]:
|
||||||
|
"""Return visible-text spans of every case-insensitive occurrence of *needle*."""
|
||||||
|
if not needle:
|
||||||
|
return []
|
||||||
|
haystack = strip_ansi(text).lower()
|
||||||
|
needle = needle.lower()
|
||||||
|
spans = []
|
||||||
|
pos = 0
|
||||||
|
while (found := haystack.find(needle, pos)) >= 0:
|
||||||
|
end = found + len(needle)
|
||||||
|
spans.append((found, end))
|
||||||
|
pos = end
|
||||||
|
return spans
|
||||||
|
|
||||||
|
|
||||||
|
def snapshot_record_lines(
|
||||||
|
state: dict, highlighter: GrepHighlighter, logfmt: Any = None
|
||||||
|
) -> list[str]:
|
||||||
|
"""Render the records of a snapshot state that matched grep patterns.
|
||||||
|
|
||||||
|
Each matched entry is shown as one ``path.to.record = {compact value}``
|
||||||
|
line, where the record is the top-level key plus one subkey — mirroring
|
||||||
|
change-record rendering, with normal path coloring, logfmt-prettified
|
||||||
|
elements, and the matched regions highlighted.
|
||||||
|
"""
|
||||||
|
records: dict[tuple[str, ...], None] = {}
|
||||||
|
for path in highlighter.matched_paths:
|
||||||
|
record = tuple(path[:2]) if len(path) > 1 else tuple(path)
|
||||||
|
records.setdefault(record)
|
||||||
|
lines = []
|
||||||
|
for record in records:
|
||||||
|
path = list(record)
|
||||||
|
dotted = ".".join(path)
|
||||||
|
value = _get_nested(state, path)
|
||||||
|
path_str = _format_path(path, logfmt, highlight=highlighter)
|
||||||
|
value_str = _format_value(value, dotted, logfmt=logfmt, highlight=highlighter)
|
||||||
|
lines.append(str(Line()(" ", path_str, " ").sep("=")(" ", value_str)))
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass(frozen=True)
|
||||||
|
class GrepPattern:
|
||||||
|
"""One parsed ``--grep`` pattern.
|
||||||
|
|
||||||
|
The bare form (``term`` set) matches the action, the user, or any
|
||||||
|
dotted path or value in the diff. The ``path=value`` form
|
||||||
|
(``path_term`` and ``value_term`` set) requires both sides to match
|
||||||
|
within the same change line; either side may be left empty to match
|
||||||
|
values only (``=value``) or paths only (``path=``).
|
||||||
|
"""
|
||||||
|
|
||||||
|
raw: str
|
||||||
|
term: str | None = None
|
||||||
|
path_term: str | None = None
|
||||||
|
value_term: str | None = None
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def parse(cls, raw: str) -> GrepPattern:
|
||||||
|
"""Parse a pattern, splitting the ``path=value`` form on the first ``=``."""
|
||||||
|
if "=" in raw:
|
||||||
|
path_term, value_term = raw.split("=", 1)
|
||||||
|
return cls(raw, path_term=path_term, value_term=value_term)
|
||||||
|
return cls(raw, term=raw)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass(frozen=True)
|
||||||
|
class _ValueMark:
|
||||||
|
"""A value-side match.
|
||||||
|
|
||||||
|
``needle`` is the text to locate in the displayed value ("" = a match
|
||||||
|
that marks nothing, e.g. from an empty term). ``whole`` marks the
|
||||||
|
entire displayed value: used when the raw value matched but a logfmt
|
||||||
|
formatter displays something else, so no needle can be located.
|
||||||
|
"""
|
||||||
|
|
||||||
|
needle: str = ""
|
||||||
|
whole: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
@dataclasses.dataclass
|
||||||
|
class _Entry:
|
||||||
|
"""One matchable ``(path, value)`` line of a record's flattened diff.
|
||||||
|
|
||||||
|
``anchor`` is set for deleted content: the deleted path whose line is
|
||||||
|
displayed for this entry (the entry itself may sit below it).
|
||||||
|
"""
|
||||||
|
|
||||||
|
path: list[str]
|
||||||
|
raw: Any
|
||||||
|
text: str
|
||||||
|
anchor: list[str] | None = None
|
||||||
|
|
||||||
|
|
||||||
|
def _element_matches(pattern: str, element: str) -> bool:
|
||||||
|
"""Match one path element: in full, or as a glob when it uses wildcards."""
|
||||||
|
pattern = pattern.casefold()
|
||||||
|
element = element.casefold()
|
||||||
|
if any(char in pattern for char in _GLOB_CHARS):
|
||||||
|
return fnmatch.fnmatchcase(element, pattern)
|
||||||
|
return element == pattern
|
||||||
|
|
||||||
|
|
||||||
|
def _match_path(term: str, path: list[str]) -> frozenset[int] | None:
|
||||||
|
"""Match a dotted term against *path* as a contiguous element sequence.
|
||||||
|
|
||||||
|
Returns the indices of the matched elements, or ``None``. An empty
|
||||||
|
term matches anything and marks no elements.
|
||||||
|
"""
|
||||||
|
if not term:
|
||||||
|
return frozenset()
|
||||||
|
patterns = term.split(".")
|
||||||
|
for start in range(len(path) - len(patterns) + 1):
|
||||||
|
if all(
|
||||||
|
_element_matches(pattern, path[start + i])
|
||||||
|
for i, pattern in enumerate(patterns)
|
||||||
|
):
|
||||||
|
return frozenset(range(start, start + len(patterns)))
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def _match_value(term: str, raw: Any, text: str) -> _ValueMark | None:
|
||||||
|
"""Match a term against a value.
|
||||||
|
|
||||||
|
String values match by substring; containers do not match (their
|
||||||
|
leaves are matched individually); all other values match only in
|
||||||
|
full. A term with wildcards matches the whole value text.
|
||||||
|
"""
|
||||||
|
if not term:
|
||||||
|
return _ValueMark()
|
||||||
|
needle = term.casefold()
|
||||||
|
haystack = text.casefold()
|
||||||
|
if any(char in needle for char in _GLOB_CHARS):
|
||||||
|
return _ValueMark(text) if fnmatch.fnmatchcase(haystack, needle) else None
|
||||||
|
if isinstance(raw, str):
|
||||||
|
return _ValueMark(term) if needle in haystack else None
|
||||||
|
if isinstance(raw, (dict, list)):
|
||||||
|
return None
|
||||||
|
return _ValueMark(text) if needle == haystack else None
|
||||||
|
|
||||||
|
|
||||||
|
def _dual_mark(term: str, raw: Any, text: str, pretty: str | None) -> _ValueMark | None:
|
||||||
|
"""Match a term against both the raw and the prettified form of a value.
|
||||||
|
|
||||||
|
*pretty* is the logfmt-resolved display text, which is what the log
|
||||||
|
shows when set. A match on the displayed form locates its needle
|
||||||
|
there; a match on the raw form alone marks the whole displayed value.
|
||||||
|
"""
|
||||||
|
raw_mark = _match_value(term, raw, text)
|
||||||
|
if pretty is None:
|
||||||
|
return raw_mark
|
||||||
|
pretty_mark = _match_value(term, pretty, pretty)
|
||||||
|
if pretty_mark is not None:
|
||||||
|
return pretty_mark
|
||||||
|
if raw_mark is not None and (raw_mark.needle or raw_mark.whole):
|
||||||
|
return _ValueMark(whole=True)
|
||||||
|
return raw_mark
|
||||||
|
|
||||||
|
|
||||||
|
def _value_text(value: Any) -> str:
|
||||||
|
"""Render a value as matchable text, following the display conventions."""
|
||||||
|
if value is None:
|
||||||
|
return "null"
|
||||||
|
if isinstance(value, bool):
|
||||||
|
return "true" if value else "false"
|
||||||
|
if isinstance(value, (int, float)):
|
||||||
|
return str(value)
|
||||||
|
if isinstance(value, str):
|
||||||
|
return value
|
||||||
|
if isinstance(value, (dict, list)):
|
||||||
|
try:
|
||||||
|
return json.dumps(value, default=str)
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
pass
|
||||||
|
return str(value)
|
||||||
|
|
||||||
|
|
||||||
|
def _leaf_entries(
|
||||||
|
path: list[str], value: Any, anchor: list[str] | None
|
||||||
|
) -> Iterator[_Entry]:
|
||||||
|
"""Yield an entry for every value under *value*.
|
||||||
|
|
||||||
|
Added, replaced or deleted containers are a single change line in the
|
||||||
|
log but hold many values; descending into them lets patterns match
|
||||||
|
their content. List elements are addressed by index (``users.0``).
|
||||||
|
"""
|
||||||
|
items: Iterator[tuple[str, Any]]
|
||||||
|
if isinstance(value, dict):
|
||||||
|
items = ((str(key), item) for key, item in value.items())
|
||||||
|
elif isinstance(value, list):
|
||||||
|
items = ((str(index), item) for index, item in enumerate(value))
|
||||||
|
else:
|
||||||
|
return
|
||||||
|
for key, item in items:
|
||||||
|
item_path = [*path, key]
|
||||||
|
yield _Entry(item_path, item, _value_text(item), anchor)
|
||||||
|
yield from _leaf_entries(item_path, item, anchor)
|
||||||
|
|
||||||
|
|
||||||
|
def _record_entries(record: ChangeRecord, previous: dict | None) -> list[_Entry]:
|
||||||
|
"""Flatten a record's diff into matchable entries.
|
||||||
|
|
||||||
|
Uses the same traversal as the change-log rendering, so paths match
|
||||||
|
what the log shows; deleted paths carry their previous value.
|
||||||
|
"""
|
||||||
|
changes: list[tuple[str, list[str], Any]] = []
|
||||||
|
_collect_changes(unmarshal(record.diff), [], changes, previous)
|
||||||
|
entries: list[_Entry] = []
|
||||||
|
for change_type, path, value in changes:
|
||||||
|
anchor = None
|
||||||
|
if change_type == "delete":
|
||||||
|
value = _get_nested(previous, path)
|
||||||
|
anchor = path
|
||||||
|
entries.append(_Entry(path, value, _value_text(value), anchor))
|
||||||
|
entries.extend(_leaf_entries(path, value, anchor))
|
||||||
|
return entries
|
||||||
|
|
||||||
|
|
||||||
|
class GrepHighlighter:
|
||||||
|
"""The matched regions of one record, wrapping rendered text on demand.
|
||||||
|
|
||||||
|
Implements the highlighter hook of the :mod:`kanta.logging`
|
||||||
|
formatters: ``path`` for path elements, ``value`` for values,
|
||||||
|
``meta`` for header fields and ``delete`` for deletion markers.
|
||||||
|
All wrapping goes through :func:`mark_spans`, so overlapping matches
|
||||||
|
merge into one highlight.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self._lit_paths: set[str] = set()
|
||||||
|
self._lit_deletes: set[str] = set()
|
||||||
|
self._value_marks: dict[str, list[_ValueMark]] = {}
|
||||||
|
self._meta_marks: dict[str, list[_ValueMark]] = {}
|
||||||
|
self.matched_paths: list[list[str]] = []
|
||||||
|
|
||||||
|
def _light_elements(self, path: list[str], indices) -> None:
|
||||||
|
for i in indices:
|
||||||
|
self._lit_paths.add(".".join(path[: i + 1]))
|
||||||
|
|
||||||
|
def add(
|
||||||
|
self,
|
||||||
|
entry: _Entry,
|
||||||
|
elements: frozenset[int] | None,
|
||||||
|
vmark: _ValueMark | None,
|
||||||
|
) -> None:
|
||||||
|
"""Record one entry's match: lit element indices and/or a value mark."""
|
||||||
|
self.matched_paths.append(entry.path)
|
||||||
|
marked = vmark is not None and (vmark.needle or vmark.whole)
|
||||||
|
if entry.anchor is None:
|
||||||
|
if elements:
|
||||||
|
self._light_elements(entry.path, elements)
|
||||||
|
if marked:
|
||||||
|
key = ".".join(entry.path)
|
||||||
|
self._value_marks.setdefault(key, []).append(vmark)
|
||||||
|
return
|
||||||
|
# Deleted content: only the anchor path line is displayed. Light
|
||||||
|
# the genuinely matched elements within it; a match on the removed
|
||||||
|
# value or below the anchor marks the deletion marker (✗) instead.
|
||||||
|
if elements:
|
||||||
|
shown = {i for i in elements if i < len(entry.anchor)}
|
||||||
|
if shown:
|
||||||
|
self._light_elements(entry.path, shown)
|
||||||
|
if len(shown) != len(elements):
|
||||||
|
self._lit_deletes.add(".".join(entry.anchor))
|
||||||
|
if marked:
|
||||||
|
self._lit_deletes.add(".".join(entry.anchor))
|
||||||
|
|
||||||
|
def add_meta(self, field: str, mark: _ValueMark) -> None:
|
||||||
|
"""Record a header match on ``field`` (``"action"`` or ``"user"``)."""
|
||||||
|
if mark.needle or mark.whole:
|
||||||
|
self._meta_marks.setdefault(field, []).append(mark)
|
||||||
|
|
||||||
|
def path(self, text: str, path: str) -> str:
|
||||||
|
"""Wrap a rendered path element when its element matched."""
|
||||||
|
if text and path in self._lit_paths:
|
||||||
|
return mark_spans(text, [(0, len(strip_ansi(text)))])
|
||||||
|
return text
|
||||||
|
|
||||||
|
def delete(self, text: str, path: str) -> str:
|
||||||
|
"""Wrap the deletion marker when the removed content matched."""
|
||||||
|
if text and path in self._lit_deletes:
|
||||||
|
return mark_spans(text, [(0, len(strip_ansi(text)))])
|
||||||
|
return text
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _apply_marks(text: str, marks: list[_ValueMark]) -> str:
|
||||||
|
if not text or not marks:
|
||||||
|
return text
|
||||||
|
spans: list[tuple[int, int]] = []
|
||||||
|
for mark in marks:
|
||||||
|
if mark.whole:
|
||||||
|
spans.append((0, len(strip_ansi(text))))
|
||||||
|
else:
|
||||||
|
spans.extend(_find_spans(text, mark.needle))
|
||||||
|
return mark_spans(text, spans)
|
||||||
|
|
||||||
|
def value(self, text: str, path: str) -> str:
|
||||||
|
"""Wrap the matched regions of a rendered value."""
|
||||||
|
return self._apply_marks(text, self._value_marks.get(path, []))
|
||||||
|
|
||||||
|
def meta(self, text: str, field: str) -> str:
|
||||||
|
"""Wrap the matched regions of a rendered header field."""
|
||||||
|
return self._apply_marks(text, self._meta_marks.get(field, []))
|
||||||
|
|
||||||
|
|
||||||
|
def _match_path_dual(term: str, path: list[str], logfmt: Any) -> frozenset[int] | None:
|
||||||
|
"""Match a path term against raw and logfmt-prettified elements.
|
||||||
|
|
||||||
|
A logfmt formatter may display an element differently (e.g. resolving a
|
||||||
|
user id to a name); the term matches either form, and the matched
|
||||||
|
element indices are lit so the displayed text gets marked.
|
||||||
|
"""
|
||||||
|
elements = _match_path(term, path)
|
||||||
|
if logfmt is not None:
|
||||||
|
pretty = []
|
||||||
|
for i, element in enumerate(path):
|
||||||
|
resolved = logfmt(element, ".".join(path[: i + 1]))
|
||||||
|
pretty.append(resolved if resolved is not None else element)
|
||||||
|
pretty_elements = _match_path(term, pretty)
|
||||||
|
if pretty_elements is not None:
|
||||||
|
elements = (
|
||||||
|
pretty_elements if elements is None else elements | pretty_elements
|
||||||
|
)
|
||||||
|
return elements
|
||||||
|
|
||||||
|
|
||||||
|
def _match_entry(
|
||||||
|
pattern: GrepPattern, entry: _Entry, logfmt: Any
|
||||||
|
) -> tuple[frozenset[int] | None, _ValueMark | None] | None:
|
||||||
|
"""Match one pattern against one entry, returning its match marks.
|
||||||
|
|
||||||
|
Returns ``None`` when the entry does not match. Otherwise returns the
|
||||||
|
matched path-element indices and/or the value mark, ready for
|
||||||
|
:meth:`GrepHighlighter.add`.
|
||||||
|
"""
|
||||||
|
pretty = None
|
||||||
|
if logfmt is not None:
|
||||||
|
pretty = logfmt(entry.raw, ".".join(entry.path))
|
||||||
|
if pattern.term is not None:
|
||||||
|
elements = _match_path_dual(pattern.term, entry.path, logfmt)
|
||||||
|
vmark = _dual_mark(pattern.term, entry.raw, entry.text, pretty)
|
||||||
|
if elements is None and vmark is None:
|
||||||
|
return None
|
||||||
|
else:
|
||||||
|
elements = _match_path_dual(pattern.path_term or "", entry.path, logfmt)
|
||||||
|
vmark = _dual_mark(pattern.value_term or "", entry.raw, entry.text, pretty)
|
||||||
|
if elements is None or vmark is None:
|
||||||
|
return None
|
||||||
|
return elements, vmark
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate_snapshot(
|
||||||
|
state: dict, patterns: list[GrepPattern], logfmt: Any = None
|
||||||
|
) -> GrepHighlighter | None:
|
||||||
|
"""Match *patterns* against a snapshot's full state, returning its matched regions.
|
||||||
|
|
||||||
|
The state is flattened into the same ``(path, value)`` entries change
|
||||||
|
records are matched against, so path and value matching semantics are
|
||||||
|
identical; a snapshot simply has no action or user to match. Returns
|
||||||
|
``None`` when any pattern matches nowhere in the state.
|
||||||
|
"""
|
||||||
|
entries = list(_leaf_entries([], unmarshal(state), None))
|
||||||
|
highlighter = GrepHighlighter()
|
||||||
|
for pattern in patterns:
|
||||||
|
matched = False
|
||||||
|
for entry in entries:
|
||||||
|
match = _match_entry(pattern, entry, logfmt)
|
||||||
|
if match is None:
|
||||||
|
continue
|
||||||
|
matched = True
|
||||||
|
highlighter.add(entry, *match)
|
||||||
|
if not matched:
|
||||||
|
return None
|
||||||
|
return highlighter
|
||||||
|
|
||||||
|
|
||||||
|
def matches_snapshot(
|
||||||
|
state: dict, patterns: list[GrepPattern], logfmt: Any = None
|
||||||
|
) -> bool:
|
||||||
|
"""Return whether every pattern matched somewhere in a snapshot's state."""
|
||||||
|
return evaluate_snapshot(state, patterns, logfmt) is not None
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(
|
||||||
|
record: ChangeRecord,
|
||||||
|
previous: dict | None,
|
||||||
|
patterns: list[GrepPattern],
|
||||||
|
logfmt: Any = None,
|
||||||
|
) -> GrepHighlighter | None:
|
||||||
|
"""Match *patterns* against a record, returning its matched regions.
|
||||||
|
|
||||||
|
Returns ``None`` when any pattern matches nowhere in the transaction.
|
||||||
|
Otherwise every pattern contributed its matches — action, user, or
|
||||||
|
change lines — to the returned highlighter; different patterns may
|
||||||
|
match different lines of the same record.
|
||||||
|
|
||||||
|
``logfmt`` is the optional composed logfmt callable; when given, both
|
||||||
|
the raw and the prettified form of each value (and of the user) are
|
||||||
|
matched.
|
||||||
|
"""
|
||||||
|
entries = _record_entries(record, previous)
|
||||||
|
highlighter = GrepHighlighter()
|
||||||
|
for pattern in patterns:
|
||||||
|
matched = False
|
||||||
|
if pattern.term is not None:
|
||||||
|
mark = _match_value(pattern.term, record.a, record.a)
|
||||||
|
if mark is not None:
|
||||||
|
highlighter.add_meta("action", mark)
|
||||||
|
matched = True
|
||||||
|
if record.u:
|
||||||
|
pretty_user = (
|
||||||
|
logfmt(record.u, _USER_PATH) if logfmt is not None else None
|
||||||
|
)
|
||||||
|
mark = _dual_mark(pattern.term, record.u, record.u, pretty_user)
|
||||||
|
if mark is not None:
|
||||||
|
highlighter.add_meta("user", mark)
|
||||||
|
matched = True
|
||||||
|
for entry in entries:
|
||||||
|
match = _match_entry(pattern, entry, logfmt)
|
||||||
|
if match is None:
|
||||||
|
continue
|
||||||
|
matched = True
|
||||||
|
highlighter.add(entry, *match)
|
||||||
|
if not matched:
|
||||||
|
return None
|
||||||
|
return highlighter
|
||||||
@@ -7,6 +7,7 @@ from pathlib import Path
|
|||||||
from types import ModuleType, SimpleNamespace
|
from types import ModuleType, SimpleNamespace
|
||||||
from typing import Any, Generic, TypeVar
|
from typing import Any, Generic, TypeVar
|
||||||
|
|
||||||
|
from kanta.exceptions import DataIntegrityError
|
||||||
from kanta.kantaimpl import KantaImpl
|
from kanta.kantaimpl import KantaImpl
|
||||||
from kanta.serialization import JsonSerializer, Serializer
|
from kanta.serialization import JsonSerializer, Serializer
|
||||||
from kanta.transaction import transaction as _transaction
|
from kanta.transaction import transaction as _transaction
|
||||||
@@ -146,6 +147,30 @@ class Kanta(Generic[T]):
|
|||||||
"""
|
"""
|
||||||
return self._impl.ctx
|
return self._impl.ctx
|
||||||
|
|
||||||
|
@property
|
||||||
|
def rev(self) -> int:
|
||||||
|
"""Current revision: the state position as a count of change records.
|
||||||
|
|
||||||
|
Revision numbers identify state positions *between* diffs, not the
|
||||||
|
diffs themselves: the empty initial state is revision 0, and each
|
||||||
|
change record (including the bootstrap record) advances the position
|
||||||
|
by one. Snapshots persist the revision of the state they contain in
|
||||||
|
their ``r`` field; change records carry no revision of their own.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
The number of change records applied since revision 0.
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
kanta.exceptions.DataIntegrityError: If the instance is not open.
|
||||||
|
"""
|
||||||
|
if not self._impl.opened:
|
||||||
|
raise DataIntegrityError(
|
||||||
|
"Kanta instance must be opened before reading rev",
|
||||||
|
db_path=self._impl.filename,
|
||||||
|
action="rev",
|
||||||
|
)
|
||||||
|
return self._impl.rev
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def mtime(self) -> datetime | None:
|
def mtime(self) -> datetime | None:
|
||||||
"""Last modification time carried forward from change records.
|
"""Last modification time carried forward from change records.
|
||||||
|
|||||||
+33
-11
@@ -10,7 +10,7 @@ from datetime import UTC, datetime, timedelta
|
|||||||
from types import SimpleNamespace
|
from types import SimpleNamespace
|
||||||
from typing import Any, Generic, TypeVar
|
from typing import Any, Generic, TypeVar
|
||||||
|
|
||||||
from kanta.callbacks import CallbackRegistry, InjectionContext
|
from kanta.callbacks import CallbackRegistry, InjectionContext, callback_error_reporter
|
||||||
from kanta.exceptions import DatabaseError, DataIntegrityError, ReplayError
|
from kanta.exceptions import DatabaseError, DataIntegrityError, ReplayError
|
||||||
from kanta.logging import (
|
from kanta.logging import (
|
||||||
_USER_PATH,
|
_USER_PATH,
|
||||||
@@ -19,22 +19,18 @@ from kanta.logging import (
|
|||||||
emit_event,
|
emit_event,
|
||||||
migration_logger,
|
migration_logger,
|
||||||
)
|
)
|
||||||
|
from kanta.metamigr import migrate_k1
|
||||||
from kanta.migrations import MigrationReport, Migrations
|
from kanta.migrations import MigrationReport, Migrations
|
||||||
from kanta.persistence import PersistenceMixin
|
from kanta.persistence import PersistenceMixin
|
||||||
from kanta.rotation import execute_rotation, plan_rotation
|
from kanta.rotation import execute_rotation, plan_rotation
|
||||||
from kanta.serialization import restore_data_in_place, struct_to_dict
|
from kanta.serialization import restore_data_in_place, struct_to_dict
|
||||||
from kanta.serialization.base import replay
|
from kanta.serialization.base import replay
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
T = TypeVar("T")
|
T = TypeVar("T")
|
||||||
|
|
||||||
|
|
||||||
def _log_callback_error(callback_error, callback):
|
|
||||||
"""Report a failing logging callback and continue with the next one."""
|
|
||||||
_logger.exception("Log callback %r failed: %s", callback, callback_error)
|
|
||||||
|
|
||||||
|
|
||||||
class KantaImpl(PersistenceMixin, Generic[T]):
|
class KantaImpl(PersistenceMixin, Generic[T]):
|
||||||
"""Internal state and logic for Kanta."""
|
"""Internal state and logic for Kanta."""
|
||||||
|
|
||||||
@@ -119,7 +115,7 @@ class KantaImpl(PersistenceMixin, Generic[T]):
|
|||||||
kanta=self._kanta,
|
kanta=self._kanta,
|
||||||
report=report,
|
report=report,
|
||||||
),
|
),
|
||||||
on_error=_log_callback_error,
|
on_error=callback_error_reporter("logmigr"),
|
||||||
)
|
)
|
||||||
return
|
return
|
||||||
|
|
||||||
@@ -222,6 +218,26 @@ class KantaImpl(PersistenceMixin, Generic[T]):
|
|||||||
cause_type=type(e).__name__,
|
cause_type=type(e).__name__,
|
||||||
) from e
|
) from e
|
||||||
|
|
||||||
|
# Meta migration k0 → k1: a file whose last snapshot is legacy
|
||||||
|
# is rewritten in full so every snapshot gains a trustworthy r
|
||||||
|
# (see kanta/metamigr.py). Rotation, when it ran, already
|
||||||
|
# produced k1 snapshots with positionally counted revisions, so
|
||||||
|
# this only fires when no rotation happened. Read-only opens
|
||||||
|
# skip the rewrite; replay counted the same revision either way.
|
||||||
|
meta_migrated_at: datetime | None = None
|
||||||
|
if rr.legacy_snapshot and not self.readonly:
|
||||||
|
meta_migrated_at = self.now()
|
||||||
|
new_content = migrate_k1(
|
||||||
|
content,
|
||||||
|
framer=self.framer,
|
||||||
|
serializer=self.serializer,
|
||||||
|
state=rr.state,
|
||||||
|
version=rr.version,
|
||||||
|
m=rr.m,
|
||||||
|
now=meta_migrated_at,
|
||||||
|
)
|
||||||
|
await asyncio.to_thread(self.file.replace_content, new_content)
|
||||||
|
|
||||||
migration_report = None
|
migration_report = None
|
||||||
state_before_migrations = None
|
state_before_migrations = None
|
||||||
previous_version = rr.version
|
previous_version = rr.version
|
||||||
@@ -235,9 +251,13 @@ class KantaImpl(PersistenceMixin, Generic[T]):
|
|||||||
migrations_ran = rr.version != previous_version
|
migrations_ran = rr.version != previous_version
|
||||||
|
|
||||||
self.snapshot.ts = (
|
self.snapshot.ts = (
|
||||||
datetime.fromtimestamp(rr.last_snapshot_mtime, UTC)
|
meta_migrated_at
|
||||||
if rr.last_snapshot_mtime is not None
|
if meta_migrated_at is not None
|
||||||
else None
|
else (
|
||||||
|
datetime.fromtimestamp(rr.last_snapshot_mtime, UTC)
|
||||||
|
if rr.last_snapshot_mtime is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
)
|
)
|
||||||
|
|
||||||
self.statedict = copy.deepcopy(
|
self.statedict = copy.deepcopy(
|
||||||
@@ -263,6 +283,7 @@ class KantaImpl(PersistenceMixin, Generic[T]):
|
|||||||
raise
|
raise
|
||||||
self.version = rr.version
|
self.version = rr.version
|
||||||
self.mtime = rr.m
|
self.mtime = rr.m
|
||||||
|
self.rev = rr.rev
|
||||||
if log is not False and not migrations_ran:
|
if log is not False and not migrations_ran:
|
||||||
logger = log if isinstance(log, logging.Logger) else bootstrap_logger
|
logger = log if isinstance(log, logging.Logger) else bootstrap_logger
|
||||||
emit_event(
|
emit_event(
|
||||||
@@ -319,6 +340,7 @@ class KantaImpl(PersistenceMixin, Generic[T]):
|
|||||||
self.statedict,
|
self.statedict,
|
||||||
m=self.mtime,
|
m=self.mtime,
|
||||||
now=self.now,
|
now=self.now,
|
||||||
|
r=self.rev,
|
||||||
)
|
)
|
||||||
elif self.readonly:
|
elif self.readonly:
|
||||||
self.opened = False
|
self.opened = False
|
||||||
|
|||||||
+159
-64
@@ -5,6 +5,8 @@ through :func:`emit_event`, which runs any registered ``logemit`` callbacks
|
|||||||
and falls back to :func:`default_emit` for the built-in formatting. Diff
|
and falls back to :func:`default_emit` for the built-in formatting. Diff
|
||||||
output is formatted in a human-readable path notation style with color
|
output is formatted in a human-readable path notation style with color
|
||||||
coding; see :mod:`kanta.tty` for the color palette and line builder.
|
coding; see :mod:`kanta.tty` for the color palette and line builder.
|
||||||
|
ANSI codes are stripped at emit time when the standard error stream does
|
||||||
|
not support color (``NO_COLOR``/``FORCE_COLOR``, tty and journald checks).
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import logging
|
import logging
|
||||||
@@ -15,14 +17,25 @@ from typing import Any
|
|||||||
|
|
||||||
import msgspec
|
import msgspec
|
||||||
|
|
||||||
|
from kanta.callbacks import describe_callback
|
||||||
from kanta.serialization.base import _apply, unmarshal
|
from kanta.serialization.base import _apply, unmarshal
|
||||||
from kanta.tty import Line, displaywidth
|
from kanta.tty import Line, displaywidth, strip_ansi, use_color
|
||||||
|
|
||||||
transaction_logger = logging.getLogger("kanta.transaction")
|
transaction_logger = logging.getLogger("kanta.transaction")
|
||||||
bootstrap_logger = logging.getLogger("kanta.bootstrap")
|
bootstrap_logger = logging.getLogger("kanta.bootstrap")
|
||||||
migration_logger = logging.getLogger("kanta.migration")
|
migration_logger = logging.getLogger("kanta.migration")
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
# Event loggers carry Kanta-rendered content (colored headers, diffs) and are
|
||||||
|
# configured at import time; diagnostics from Kanta's internals use the plain
|
||||||
|
# "kanta" logger so they follow the application's root logging configuration.
|
||||||
|
EVENT_LOGGERS = ("kanta.bootstrap", "kanta.migration", "kanta.transaction")
|
||||||
|
|
||||||
|
# Loggers that emit DEBUG-level events (file-opened summary, migration diffs).
|
||||||
|
_DEBUG_LOGGERS = ("kanta.bootstrap", "kanta.migration")
|
||||||
|
|
||||||
|
_PLAIN_HANDLER_NAME = "kanta.plain"
|
||||||
|
|
||||||
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
# Pattern to match control characters and bidirectional overrides
|
# Pattern to match control characters and bidirectional overrides
|
||||||
_UNSAFE_CHARS = re.compile(
|
_UNSAFE_CHARS = re.compile(
|
||||||
@@ -146,13 +159,21 @@ def emit_event(
|
|||||||
try:
|
try:
|
||||||
proceed = handler(ev)
|
proceed = handler(ev)
|
||||||
except Exception:
|
except Exception:
|
||||||
_logger.exception("logemit callback failed, using default formatting")
|
_logger.exception(
|
||||||
|
"Kanta.logemit %s failed, using default formatting",
|
||||||
|
describe_callback(handler),
|
||||||
|
)
|
||||||
break
|
break
|
||||||
if not proceed:
|
if not proceed:
|
||||||
return
|
return
|
||||||
render(ev)
|
render(ev)
|
||||||
except Exception:
|
except Exception:
|
||||||
_logger.exception("failed to emit %s log event", ev.kind)
|
_logger.exception("Kanta failed to emit %s log event", ev.kind)
|
||||||
|
|
||||||
|
|
||||||
|
def _maybe_strip(text: str) -> str:
|
||||||
|
"""Strip ANSI codes from *text* when stderr has no color support."""
|
||||||
|
return text if use_color() else strip_ansi(text)
|
||||||
|
|
||||||
|
|
||||||
def default_emit(ev: LogEvent) -> None:
|
def default_emit(ev: LogEvent) -> None:
|
||||||
@@ -163,25 +184,28 @@ def default_emit(ev: LogEvent) -> None:
|
|||||||
logger so it can be silenced or routed separately from the headers.
|
logger so it can be silenced or routed separately from the headers.
|
||||||
This is what runs when no logemit callback handles the event; custom
|
This is what runs when no logemit callback handles the event; custom
|
||||||
callbacks may call it to delegate events they do not care about.
|
callbacks may call it to delegate events they do not care about.
|
||||||
|
|
||||||
|
ANSI color codes are stripped after formatting when the standard error
|
||||||
|
stream does not support color (see :func:`kanta.tty.use_color`).
|
||||||
"""
|
"""
|
||||||
if ev.kind != "change":
|
if ev.kind != "change":
|
||||||
ev.logger.log(ev.level, ev.header)
|
ev.logger.log(ev.level, _maybe_strip(ev.header))
|
||||||
return
|
return
|
||||||
|
|
||||||
diff_logger = logging.getLogger(f"{ev.logger.name}.diff")
|
diff_logger = logging.getLogger(f"{ev.logger.name}.diff")
|
||||||
lines = ev.diff_lines if ev.show_diff and diff_logger.isEnabledFor(ev.level) else []
|
lines = ev.diff_lines if ev.show_diff and diff_logger.isEnabledFor(ev.level) else []
|
||||||
|
|
||||||
if not lines:
|
if not lines:
|
||||||
ev.logger.log(ev.level, ev.header)
|
ev.logger.log(ev.level, _maybe_strip(ev.header))
|
||||||
return
|
return
|
||||||
|
|
||||||
if len(lines) == 1:
|
if len(lines) == 1:
|
||||||
diff_logger.log(ev.level, f"{ev.header}{lines[0]}")
|
diff_logger.log(ev.level, _maybe_strip(f"{ev.header}{lines[0]}"))
|
||||||
return
|
return
|
||||||
|
|
||||||
ev.logger.log(ev.level, ev.header)
|
ev.logger.log(ev.level, _maybe_strip(ev.header))
|
||||||
for line in lines:
|
for line in lines:
|
||||||
diff_logger.log(ev.level, line)
|
diff_logger.log(ev.level, _maybe_strip(line))
|
||||||
|
|
||||||
|
|
||||||
def _join_path(path: str, key: str) -> str:
|
def _join_path(path: str, key: str) -> str:
|
||||||
@@ -202,56 +226,80 @@ def _format_value(
|
|||||||
*,
|
*,
|
||||||
max_len: int = 60,
|
max_len: int = 60,
|
||||||
logfmt: Callable[[Any, str], str | None] | None = None,
|
logfmt: Callable[[Any, str], str | None] | None = None,
|
||||||
|
highlight: Any = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Format a value for display, truncating if needed."""
|
"""Format a value for display, truncating if needed.
|
||||||
|
|
||||||
|
``highlight`` is an optional hook with ``path(text, path)`` and
|
||||||
|
``value(text, path)`` methods (see :class:`kanta.grep.GrepHighlighter`);
|
||||||
|
it wraps matched keys and scalar values, and recurses into containers.
|
||||||
|
"""
|
||||||
if logfmt is not None:
|
if logfmt is not None:
|
||||||
resolved = logfmt(value, path)
|
resolved = logfmt(value, path)
|
||||||
if resolved is not None:
|
if resolved is not None:
|
||||||
|
if highlight is not None:
|
||||||
|
resolved = highlight.value(resolved, path)
|
||||||
return resolved
|
return resolved
|
||||||
|
|
||||||
|
def keyed(key: Any, key_path: str) -> str:
|
||||||
|
display = _format_value(key, key_path, max_len=30, logfmt=logfmt)
|
||||||
|
if highlight is not None:
|
||||||
|
display = highlight.path(display, key_path)
|
||||||
|
return display
|
||||||
|
|
||||||
if value is None:
|
if value is None:
|
||||||
return "null"
|
text = "null"
|
||||||
if isinstance(value, bool):
|
elif isinstance(value, bool):
|
||||||
return "true" if value else "false"
|
text = "true" if value else "false"
|
||||||
if isinstance(value, (int, float)):
|
elif isinstance(value, (int, float)):
|
||||||
return str(value)
|
text = str(value)
|
||||||
if isinstance(value, str):
|
elif isinstance(value, str):
|
||||||
value = _UNSAFE_CHARS.sub("", value)
|
text = _UNSAFE_CHARS.sub("", value)
|
||||||
if len(value) > max_len:
|
if len(text) > max_len:
|
||||||
return value[: max_len - 1] + _dim_ellipsis()
|
text = text[: max_len - 1] + _dim_ellipsis()
|
||||||
return value
|
elif isinstance(value, dict):
|
||||||
if isinstance(value, dict):
|
|
||||||
if not value:
|
if not value:
|
||||||
return "{}"
|
return "{}"
|
||||||
all_true = all(v is True for v in value.values())
|
all_true = all(v is True for v in value.values())
|
||||||
parts = []
|
parts = []
|
||||||
for k, v in value.items():
|
for k, v in value.items():
|
||||||
key_path = _join_path(path, str(k))
|
key_path = _join_path(path, str(k))
|
||||||
key_display = _format_value(k, key_path, max_len=30, logfmt=logfmt)
|
key_display = keyed(k, key_path)
|
||||||
if all_true:
|
if all_true:
|
||||||
parts.append(key_display)
|
parts.append(key_display)
|
||||||
else:
|
else:
|
||||||
val_display = _format_value(v, key_path, max_len=30, logfmt=logfmt)
|
val_display = _format_value(
|
||||||
|
v, key_path, max_len=30, logfmt=logfmt, highlight=highlight
|
||||||
|
)
|
||||||
parts.append(f"{key_display}: {val_display}")
|
parts.append(f"{key_display}: {val_display}")
|
||||||
return "{" + ", ".join(parts) + "}"
|
return "{" + ", ".join(parts) + "}"
|
||||||
if isinstance(value, list):
|
elif isinstance(value, list):
|
||||||
if not value:
|
if not value:
|
||||||
return "[]"
|
return "[]"
|
||||||
parts = []
|
parts = []
|
||||||
for i, v in enumerate(value):
|
for i, v in enumerate(value):
|
||||||
item_path = _join_path(path, str(i))
|
item_path = _join_path(path, str(i))
|
||||||
parts.append(_format_value(v, item_path, max_len=30, logfmt=logfmt))
|
parts.append(
|
||||||
|
_format_value(
|
||||||
|
v, item_path, max_len=30, logfmt=logfmt, highlight=highlight
|
||||||
|
)
|
||||||
|
)
|
||||||
return "[" + ", ".join(parts) + "]"
|
return "[" + ", ".join(parts) + "]"
|
||||||
text = str(value)
|
else:
|
||||||
if len(text) > max_len:
|
text = str(value)
|
||||||
text = text[: max_len - 1] + _dim_ellipsis()
|
if len(text) > max_len:
|
||||||
|
text = text[: max_len - 1] + _dim_ellipsis()
|
||||||
|
if highlight is not None:
|
||||||
|
text = highlight.value(text, path)
|
||||||
return text
|
return text
|
||||||
|
|
||||||
|
|
||||||
def _format_path_components(
|
def _format_path_components(
|
||||||
path: list[str], logfmt: Callable[[Any, str], str | None] | None
|
path: list[str],
|
||||||
|
logfmt: Callable[[Any, str], str | None] | None,
|
||||||
|
highlight: Any = None,
|
||||||
) -> list[str]:
|
) -> list[str]:
|
||||||
"""Return path components after applying formatters."""
|
"""Return path components after applying formatters and match highlights."""
|
||||||
if not path:
|
if not path:
|
||||||
return []
|
return []
|
||||||
result = []
|
result = []
|
||||||
@@ -262,6 +310,8 @@ def _format_path_components(
|
|||||||
resolved = logfmt(component, prefix_path)
|
resolved = logfmt(component, prefix_path)
|
||||||
if resolved is not None:
|
if resolved is not None:
|
||||||
display = resolved
|
display = resolved
|
||||||
|
if highlight is not None:
|
||||||
|
display = highlight.path(display, prefix_path)
|
||||||
result.append(display)
|
result.append(display)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
@@ -270,12 +320,13 @@ def _format_path(
|
|||||||
path: list[str],
|
path: list[str],
|
||||||
logfmt: Callable[[Any, str], str | None] | None,
|
logfmt: Callable[[Any, str], str | None] | None,
|
||||||
final_color: str = "path_final",
|
final_color: str = "path_final",
|
||||||
|
highlight: Any = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Format a path as dot notation with prefix in dark grey, final colored.
|
"""Format a path as dot notation with prefix in dark grey, final colored.
|
||||||
|
|
||||||
*final_color* names a color in the :data:`kanta.tty.colors` palette.
|
*final_color* names a color in the :data:`kanta.tty.colors` palette.
|
||||||
"""
|
"""
|
||||||
components = _format_path_components(path, logfmt)
|
components = _format_path_components(path, logfmt, highlight)
|
||||||
if not components:
|
if not components:
|
||||||
return ""
|
return ""
|
||||||
line = Line()
|
line = Line()
|
||||||
@@ -370,25 +421,32 @@ def _format_change_lines(
|
|||||||
path: list[str],
|
path: list[str],
|
||||||
value: Any,
|
value: Any,
|
||||||
logfmt: Callable[[Any, str], str | None] | None = None,
|
logfmt: Callable[[Any, str], str | None] | None = None,
|
||||||
|
highlight: Any = None,
|
||||||
) -> list[str]:
|
) -> list[str]:
|
||||||
"""Format a single change as one or more lines."""
|
"""Format a single change as one or more lines."""
|
||||||
if change_type == "delete":
|
if change_type == "delete":
|
||||||
components = _format_path_components(path, logfmt)
|
components = _format_path_components(path, logfmt, highlight)
|
||||||
line = Line()(" ")
|
line = Line()(" ")
|
||||||
if len(components) > 1:
|
if len(components) > 1:
|
||||||
line.path_prefix(".".join(components[:-1]) + ".")
|
line.path_prefix(".".join(components[:-1]) + ".")
|
||||||
line.delete(components[-1], " ✗")
|
marker = "✗"
|
||||||
|
if highlight is not None:
|
||||||
|
marker = highlight.delete(marker, ".".join(path))
|
||||||
|
line.delete(components[-1], " ", marker)
|
||||||
return [str(line)]
|
return [str(line)]
|
||||||
|
|
||||||
if change_type == "add":
|
if change_type == "add":
|
||||||
path_str = _format_path(path, logfmt, final_color="add")
|
path_str = _format_path(path, logfmt, final_color="add", highlight=highlight)
|
||||||
if isinstance(value, dict) and value:
|
if isinstance(value, dict) and value:
|
||||||
lines = [str(Line()(" ", path_str, " ").sep("="))]
|
lines = [str(Line()(" ", path_str, " ").sep("="))]
|
||||||
base_path = ".".join(path)
|
base_path = ".".join(path)
|
||||||
keys = []
|
keys = []
|
||||||
for k in value:
|
for k in value:
|
||||||
key_path = _join_path(base_path, str(k))
|
key_path = _join_path(base_path, str(k))
|
||||||
keys.append((k, _format_value(k, key_path, max_len=30, logfmt=logfmt)))
|
key_display = _format_value(k, key_path, max_len=30, logfmt=logfmt)
|
||||||
|
if highlight is not None:
|
||||||
|
key_display = highlight.path(key_display, key_path)
|
||||||
|
keys.append((k, key_display))
|
||||||
field_width = max(displaywidth(kd) for _, kd in keys)
|
field_width = max(displaywidth(kd) for _, kd in keys)
|
||||||
field_width = max(field_width, 12)
|
field_width = max(field_width, 12)
|
||||||
# Each item line is " {key:{field_width}}: {value}"; budget the
|
# Each item line is " {key:{field_width}}: {value}"; budget the
|
||||||
@@ -397,7 +455,9 @@ def _format_change_lines(
|
|||||||
formatted_items = []
|
formatted_items = []
|
||||||
for (k, key_display), v in zip(keys, value.values()):
|
for (k, key_display), v in zip(keys, value.values()):
|
||||||
key_path = _join_path(base_path, str(k))
|
key_path = _join_path(base_path, str(k))
|
||||||
v_str = _format_value(v, key_path, max_len=value_width, logfmt=logfmt)
|
v_str = _format_value(
|
||||||
|
v, key_path, max_len=value_width, logfmt=logfmt, highlight=highlight
|
||||||
|
)
|
||||||
formatted_items.append((key_display, v_str))
|
formatted_items.append((key_display, v_str))
|
||||||
return lines + [
|
return lines + [
|
||||||
str(
|
str(
|
||||||
@@ -407,11 +467,13 @@ def _format_change_lines(
|
|||||||
)
|
)
|
||||||
for k, v in formatted_items
|
for k, v in formatted_items
|
||||||
]
|
]
|
||||||
value_str = _format_value(value, ".".join(path), logfmt=logfmt)
|
value_str = _format_value(
|
||||||
|
value, ".".join(path), logfmt=logfmt, highlight=highlight
|
||||||
|
)
|
||||||
return [str(Line()(" ", path_str, " ").sep("=")(" ", value_str))]
|
return [str(Line()(" ", path_str, " ").sep("=")(" ", value_str))]
|
||||||
|
|
||||||
value_str = _format_value(value, ".".join(path), logfmt=logfmt)
|
value_str = _format_value(value, ".".join(path), logfmt=logfmt, highlight=highlight)
|
||||||
path_str = _format_path(path, logfmt=logfmt)
|
path_str = _format_path(path, logfmt=logfmt, highlight=highlight)
|
||||||
return [str(Line()(" ", path_str, " ").sep("=")(" ", value_str))]
|
return [str(Line()(" ", path_str, " ").sep("=")(" ", value_str))]
|
||||||
|
|
||||||
|
|
||||||
@@ -419,6 +481,7 @@ def format_diff(
|
|||||||
diff: dict,
|
diff: dict,
|
||||||
previous: dict | None = None,
|
previous: dict | None = None,
|
||||||
logfmt: Callable[[Any, str], str | None] | None = None,
|
logfmt: Callable[[Any, str], str | None] | None = None,
|
||||||
|
highlight: Any = None,
|
||||||
) -> list[str]:
|
) -> list[str]:
|
||||||
"""Format a JSON diff as human-readable lines.
|
"""Format a JSON diff as human-readable lines.
|
||||||
|
|
||||||
@@ -429,6 +492,8 @@ def format_diff(
|
|||||||
``path`` is a dot-notation string; ``"$user"`` is used for the
|
``path`` is a dot-notation string; ``"$user"`` is used for the
|
||||||
transaction actor. If the callable returns ``None``, default
|
transaction actor. If the callable returns ``None``, default
|
||||||
formatting is used.
|
formatting is used.
|
||||||
|
highlight: Optional match highlighter hook (see
|
||||||
|
:class:`kanta.grep.GrepHighlighter`) wrapping matched regions.
|
||||||
|
|
||||||
Returns a list of formatted lines (without newlines).
|
Returns a list of formatted lines (without newlines).
|
||||||
"""
|
"""
|
||||||
@@ -438,7 +503,7 @@ def format_diff(
|
|||||||
return []
|
return []
|
||||||
lines = []
|
lines = []
|
||||||
for change_type, path, value in changes:
|
for change_type, path, value in changes:
|
||||||
lines.extend(_format_change_lines(change_type, path, value, logfmt))
|
lines.extend(_format_change_lines(change_type, path, value, logfmt, highlight))
|
||||||
return lines
|
return lines
|
||||||
|
|
||||||
|
|
||||||
@@ -446,12 +511,22 @@ def format_action_header(
|
|||||||
action: str,
|
action: str,
|
||||||
user: str | None = None,
|
user: str | None = None,
|
||||||
extra: Any = None,
|
extra: Any = None,
|
||||||
|
highlight: Any = None,
|
||||||
) -> str:
|
) -> str:
|
||||||
"""Format the default action header line."""
|
"""Format the default action header line.
|
||||||
|
|
||||||
|
``highlight`` is an optional hook with a ``meta(text, field)`` method
|
||||||
|
(see :class:`kanta.grep.GrepHighlighter`) wrapping matched regions of
|
||||||
|
the action and user fields.
|
||||||
|
"""
|
||||||
|
if highlight is not None:
|
||||||
|
action = highlight.meta(action, "action")
|
||||||
line = Line().action(action)
|
line = Line().action(action)
|
||||||
if extra is not None and (extra := f"{extra}"):
|
if extra is not None and (extra := f"{extra}"):
|
||||||
line(" ").target(extra)
|
line(" ").target(extra)
|
||||||
if user is not None and (user := f"{user}"):
|
if user is not None and (user := f"{user}"):
|
||||||
|
if highlight is not None:
|
||||||
|
user = highlight.meta(user, "user")
|
||||||
line(" by ").user(user)
|
line(" by ").user(user)
|
||||||
return str(line)
|
return str(line)
|
||||||
|
|
||||||
@@ -505,6 +580,15 @@ def log_change(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_plain_handler(logger: logging.Logger) -> None:
|
||||||
|
"""Attach Kanta's no-prefix stderr handler to *logger* if it has none."""
|
||||||
|
if not logger.handlers:
|
||||||
|
handler = logging.StreamHandler(sys.stderr)
|
||||||
|
handler.setFormatter(logging.Formatter("%(message)s"))
|
||||||
|
handler.name = _PLAIN_HANDLER_NAME
|
||||||
|
logger.addHandler(handler)
|
||||||
|
|
||||||
|
|
||||||
def configure_logging(
|
def configure_logging(
|
||||||
*,
|
*,
|
||||||
skiproot: bool = True,
|
skiproot: bool = True,
|
||||||
@@ -516,13 +600,22 @@ def configure_logging(
|
|||||||
) -> None:
|
) -> None:
|
||||||
"""Configure Kanta's default logging output.
|
"""Configure Kanta's default logging output.
|
||||||
|
|
||||||
|
Called once at import time with default arguments; call again to change
|
||||||
|
the toggles. The event loggers ``kanta.bootstrap``, ``kanta.migration``
|
||||||
|
and ``kanta.transaction`` carry Kanta-rendered output (colored headers,
|
||||||
|
diffs) and print it bare through a plain stderr handler with
|
||||||
|
``propagate = False``. Diagnostic messages use the plain ``kanta``
|
||||||
|
logger, which is left untouched so the application's root logging
|
||||||
|
configuration controls its verbosity.
|
||||||
|
|
||||||
|
The event loggers are set to ``INFO`` so Kanta output is visible without
|
||||||
|
any application logging setup; only loggers Kanta owns are touched.
|
||||||
|
|
||||||
Args:
|
Args:
|
||||||
skiproot: If ``True`` (default), attach a no-prefix stderr handler to
|
skiproot: If ``True`` (default), event loggers print through Kanta's
|
||||||
the ``kanta`` logger and set ``kanta.propagate = False`` so Kanta
|
own plain handler without propagating to the root logger. If
|
||||||
output is rendered directly without propagating to the root logger.
|
``False``, Kanta's handler is removed and propagation enabled so
|
||||||
If ``False``, the child logger enable flags are still applied, but
|
the application's root logger renders event output instead.
|
||||||
no handler is added and ``kanta`` propagation is left untouched so
|
|
||||||
the application's root logger handles Kanta output.
|
|
||||||
bootstrap: Whether bootstrap logs are enabled.
|
bootstrap: Whether bootstrap logs are enabled.
|
||||||
migration: Whether migration logs are enabled.
|
migration: Whether migration logs are enabled.
|
||||||
transaction: Whether transaction logs are enabled.
|
transaction: Whether transaction logs are enabled.
|
||||||
@@ -530,13 +623,10 @@ def configure_logging(
|
|||||||
only transaction headers are printed and diff formatting is
|
only transaction headers are printed and diff formatting is
|
||||||
skipped. Per transaction this is controlled by the ``logdiff``
|
skipped. Per transaction this is controlled by the ``logdiff``
|
||||||
argument of :meth:`Kanta.transaction`.
|
argument of :meth:`Kanta.transaction`.
|
||||||
debug: Whether to set the ``kanta`` logger level to ``DEBUG`` instead
|
debug: Whether to set the event loggers that emit DEBUG-level output
|
||||||
of ``INFO``. This reveals debug-level output such as migration
|
(bootstrap and migration) to ``DEBUG``, revealing output such as
|
||||||
diffs, which are hidden by default.
|
the file-opened summary and migration diffs. ``False`` resets
|
||||||
|
all event loggers to ``INFO``.
|
||||||
This helper is not called automatically; applications that want Kanta's
|
|
||||||
default output can call it, but most applications will configure logging
|
|
||||||
themselves.
|
|
||||||
"""
|
"""
|
||||||
logging.getLogger("kanta.transaction.diff").disabled = not diff
|
logging.getLogger("kanta.transaction.diff").disabled = not diff
|
||||||
|
|
||||||
@@ -545,16 +635,21 @@ def configure_logging(
|
|||||||
("kanta.migration", migration),
|
("kanta.migration", migration),
|
||||||
("kanta.transaction", transaction),
|
("kanta.transaction", transaction),
|
||||||
):
|
):
|
||||||
logging.getLogger(name).propagate = enabled
|
logging.getLogger(name).disabled = not enabled
|
||||||
|
|
||||||
if not skiproot:
|
for name in EVENT_LOGGERS:
|
||||||
return
|
logger = logging.getLogger(name)
|
||||||
|
logger.setLevel(
|
||||||
|
logging.DEBUG if debug and name in _DEBUG_LOGGERS else logging.INFO
|
||||||
|
)
|
||||||
|
if skiproot:
|
||||||
|
logger.propagate = False
|
||||||
|
_ensure_plain_handler(logger)
|
||||||
|
else:
|
||||||
|
logger.propagate = True
|
||||||
|
logger.handlers[:] = [
|
||||||
|
h for h in logger.handlers if h.name != _PLAIN_HANDLER_NAME
|
||||||
|
]
|
||||||
|
|
||||||
target = logging.getLogger("kanta")
|
|
||||||
target.propagate = False
|
|
||||||
|
|
||||||
if not target.handlers:
|
configure_logging() # Import-time default setup; call again to reconfigure.
|
||||||
handler = logging.StreamHandler(sys.stderr)
|
|
||||||
handler.setFormatter(logging.Formatter("%(message)s"))
|
|
||||||
target.addHandler(handler)
|
|
||||||
target.setLevel(logging.DEBUG if debug else logging.INFO)
|
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
"""Meta format migrations: rewriting files between snapshot ``k`` versions.
|
||||||
|
|
||||||
|
The k0 → k1 migration introduces revision numbers. Like a rotation, the
|
||||||
|
whole file is rewritten under the lock so that every snapshot in it is a
|
||||||
|
k1 snapshot with a trustworthy ``r`` — mixed r/k meanings never persist in
|
||||||
|
one file. Unlike a rotation, all change records are kept; redundant
|
||||||
|
intermediate snapshots are dropped.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from kanta.structs import META_FORMAT, ChangeRecord, Snapshot
|
||||||
|
from kanta.serialization.base import Serializer
|
||||||
|
from kanta.serialization.framing import Framer
|
||||||
|
|
||||||
|
|
||||||
|
def migrate_k1(
|
||||||
|
content: bytes,
|
||||||
|
*,
|
||||||
|
framer: Framer,
|
||||||
|
serializer: Serializer,
|
||||||
|
state: dict[str, Any],
|
||||||
|
version: int,
|
||||||
|
m: datetime | None,
|
||||||
|
now: datetime,
|
||||||
|
) -> bytes:
|
||||||
|
"""Rewrite *content* wholly into k1 form.
|
||||||
|
|
||||||
|
The leading snapshot, when the file begins with one (the file start is
|
||||||
|
not the database start), is preserved with its data and timestamps,
|
||||||
|
re-stamped k1; its ``r`` is 0, the only count knowable from the file.
|
||||||
|
All change records follow, re-framed in order, and a closing snapshot
|
||||||
|
captures the replayed final state with ``r`` equal to the total number
|
||||||
|
of change records. *state*, *version* and *m* are the replayed final
|
||||||
|
values (pre-schema-migration, so migrations still run afterwards).
|
||||||
|
|
||||||
|
The caller gates on a legacy last snapshot and performs the in-place
|
||||||
|
rewrite under the file lock.
|
||||||
|
"""
|
||||||
|
entries = []
|
||||||
|
for is_snapshot, payload, _line, _pos in framer.iter_records(content, 0):
|
||||||
|
entries.append(
|
||||||
|
(
|
||||||
|
is_snapshot,
|
||||||
|
serializer.decode(
|
||||||
|
payload, type=Snapshot if is_snapshot else ChangeRecord
|
||||||
|
),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
out = bytearray()
|
||||||
|
if entries and entries[0][0]:
|
||||||
|
snap = entries[0][1]
|
||||||
|
assert isinstance(snap, Snapshot)
|
||||||
|
leading = Snapshot(
|
||||||
|
ts=snap.ts, v=snap.v, state=snap.state, m=snap.m, r=0, k=META_FORMAT
|
||||||
|
)
|
||||||
|
out += framer.frame_snapshot(serializer.encode(leading), record_offset=0)
|
||||||
|
|
||||||
|
n_changes = 0
|
||||||
|
for is_snapshot, record in entries:
|
||||||
|
if is_snapshot:
|
||||||
|
continue
|
||||||
|
out += framer.frame_change(serializer.encode(record), record_offset=len(out))
|
||||||
|
n_changes += 1
|
||||||
|
|
||||||
|
if n_changes:
|
||||||
|
closing = Snapshot(
|
||||||
|
ts=now, v=version, state=state, m=m, r=n_changes, k=META_FORMAT
|
||||||
|
)
|
||||||
|
out += framer.frame_snapshot(serializer.encode(closing), record_offset=len(out))
|
||||||
|
return bytes(out)
|
||||||
+3
-3
@@ -13,7 +13,7 @@ from dataclasses import dataclass
|
|||||||
from types import ModuleType
|
from types import ModuleType
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from kanta.diff import compute_diff
|
from kanta.diff import diff
|
||||||
from kanta.exceptions import DatabaseError
|
from kanta.exceptions import DatabaseError
|
||||||
|
|
||||||
# Cache registries by imported module object so that many Kanta instances using
|
# Cache registries by imported module object so that many Kanta instances using
|
||||||
@@ -179,7 +179,7 @@ class Migrations:
|
|||||||
self._call_migration(fn, data_dict, kanta)
|
self._call_migration(fn, data_dict, kanta)
|
||||||
current_version = version
|
current_version = version
|
||||||
changed = before != data_dict
|
changed = before != data_dict
|
||||||
diff = compute_diff(before, data_dict) if changed else None
|
delta = diff(before, data_dict) if changed else None
|
||||||
desc = (fn.__doc__ or f"v{version}").split("\n")[0].rstrip(".")
|
desc = (fn.__doc__ or f"v{version}").split("\n")[0].rstrip(".")
|
||||||
migrations.append(
|
migrations.append(
|
||||||
MigrationInfo(
|
MigrationInfo(
|
||||||
@@ -187,7 +187,7 @@ class Migrations:
|
|||||||
description=desc,
|
description=desc,
|
||||||
version=version,
|
version=version,
|
||||||
changed=changed,
|
changed=changed,
|
||||||
diff=diff,
|
diff=delta,
|
||||||
before=before,
|
before=before,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|||||||
+20
-18
@@ -12,8 +12,8 @@ from datetime import UTC, datetime
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from kanta.callbacks import CallbackRegistry, InjectionContext
|
from kanta.callbacks import CallbackRegistry, InjectionContext, callback_error_reporter
|
||||||
from kanta.diff import compute_diff
|
from kanta.diff import diff
|
||||||
from kanta.exceptions import DatabaseError, DataIntegrityError
|
from kanta.exceptions import DatabaseError, DataIntegrityError
|
||||||
from kanta.filelock import LockedFile
|
from kanta.filelock import LockedFile
|
||||||
from kanta.structs import ChangeRecord
|
from kanta.structs import ChangeRecord
|
||||||
@@ -21,7 +21,7 @@ from kanta.serialization import JsonSerializer, Serializer
|
|||||||
from kanta.serialization.framing import Framer
|
from kanta.serialization.framing import Framer
|
||||||
from kanta.snapshot import SnapshotState
|
from kanta.snapshot import SnapshotState
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
|
|
||||||
class PersistenceMixin:
|
class PersistenceMixin:
|
||||||
@@ -40,6 +40,7 @@ class PersistenceMixin:
|
|||||||
background_error: DatabaseError | None
|
background_error: DatabaseError | None
|
||||||
flush_interval: float
|
flush_interval: float
|
||||||
version: int
|
version: int
|
||||||
|
rev: int
|
||||||
opened: bool
|
opened: bool
|
||||||
readonly: bool
|
readonly: bool
|
||||||
mtime: datetime | None
|
mtime: datetime | None
|
||||||
@@ -64,6 +65,7 @@ class PersistenceMixin:
|
|||||||
self.background_error = None
|
self.background_error = None
|
||||||
self.flush_interval = flush_interval
|
self.flush_interval = flush_interval
|
||||||
self.version = 0
|
self.version = 0
|
||||||
|
self.rev = 0
|
||||||
self.mtime: datetime | None = None
|
self.mtime: datetime | None = None
|
||||||
self.clock: Callable[[], datetime] | None = None
|
self.clock: Callable[[], datetime] | None = None
|
||||||
|
|
||||||
@@ -110,26 +112,25 @@ class PersistenceMixin:
|
|||||||
break
|
break
|
||||||
except DatabaseError as e:
|
except DatabaseError as e:
|
||||||
self.background_error = e
|
self.background_error = e
|
||||||
|
|
||||||
def _log_callback_error(callback_error, callback):
|
|
||||||
_logger.exception(
|
|
||||||
"Background error callback %r failed: %s",
|
|
||||||
callback,
|
|
||||||
callback_error,
|
|
||||||
)
|
|
||||||
|
|
||||||
await self.callback_registry.invoke(
|
await self.callback_registry.invoke(
|
||||||
"fatal_error",
|
"fatal_error",
|
||||||
InjectionContext(error=e, kanta=self._kanta),
|
InjectionContext(error=e, kanta=self._kanta),
|
||||||
on_error=_log_callback_error,
|
on_error=callback_error_reporter("fatal_error"),
|
||||||
|
)
|
||||||
|
_logger.error(
|
||||||
|
"Kanta background flush failed; automatic flushing stopped: %s", e
|
||||||
)
|
)
|
||||||
_logger.error("Background flush loop stopped: %s", e)
|
|
||||||
break
|
break
|
||||||
|
|
||||||
def maybe_snapshot(self) -> None:
|
def maybe_snapshot(self) -> None:
|
||||||
"""Evaluate and possibly write a snapshot from current state."""
|
"""Evaluate and possibly write a snapshot from current state."""
|
||||||
self.snapshot.maybe_write(
|
self.snapshot.maybe_write(
|
||||||
self.file, self.version, self.statedict, m=self.mtime, now=self.now
|
self.file,
|
||||||
|
self.version,
|
||||||
|
self.statedict,
|
||||||
|
m=self.mtime,
|
||||||
|
now=self.now,
|
||||||
|
r=self.rev,
|
||||||
)
|
)
|
||||||
|
|
||||||
def queue_change(
|
def queue_change(
|
||||||
@@ -158,11 +159,11 @@ class PersistenceMixin:
|
|||||||
The queued :class:`ChangeRecord`, or ``None`` if the diff was empty
|
The queued :class:`ChangeRecord`, or ``None`` if the diff was empty
|
||||||
and *force* is ``False``.
|
and *force* is ``False``.
|
||||||
"""
|
"""
|
||||||
diff = compute_diff(self.statedict, current)
|
delta = diff(self.statedict, current)
|
||||||
if not diff:
|
if not delta:
|
||||||
if not force:
|
if not force:
|
||||||
return None
|
return None
|
||||||
diff = {}
|
delta = {}
|
||||||
|
|
||||||
# The clock is only read when a record is actually queued.
|
# The clock is only read when a record is actually queued.
|
||||||
now = self.now()
|
now = self.now()
|
||||||
@@ -182,10 +183,11 @@ class PersistenceMixin:
|
|||||||
v=self.version,
|
v=self.version,
|
||||||
u=user,
|
u=user,
|
||||||
m=m,
|
m=m,
|
||||||
diff=diff,
|
diff=delta,
|
||||||
)
|
)
|
||||||
self.pending_changes.append(record)
|
self.pending_changes.append(record)
|
||||||
self.statedict = copy.deepcopy(current)
|
self.statedict = copy.deepcopy(current)
|
||||||
|
self.rev += 1
|
||||||
if m is not None:
|
if m is not None:
|
||||||
self.mtime = m
|
self.mtime = m
|
||||||
return record
|
return record
|
||||||
|
|||||||
+236
-6
@@ -10,13 +10,15 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import copy
|
import copy
|
||||||
import dataclasses
|
import dataclasses
|
||||||
|
import re
|
||||||
from collections.abc import Iterator
|
from collections.abc import Iterator
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
from typing import TYPE_CHECKING, Any, Union
|
from typing import TYPE_CHECKING, Any, Union
|
||||||
|
|
||||||
import msgspec
|
import msgspec
|
||||||
|
|
||||||
from kanta.callbacks import InjectionContext
|
from kanta.callbacks import InjectionContext
|
||||||
from kanta.diff import patch_state
|
from kanta.diff import patch
|
||||||
from kanta.exceptions import ReplayError
|
from kanta.exceptions import ReplayError
|
||||||
from kanta.logging import _USER_PATH, LogEvent, transaction_logger
|
from kanta.logging import _USER_PATH, LogEvent, transaction_logger
|
||||||
from kanta.structs import ChangeRecord, Snapshot
|
from kanta.structs import ChangeRecord, Snapshot
|
||||||
@@ -33,6 +35,7 @@ class SnapshotEvent:
|
|||||||
byte_pos: int
|
byte_pos: int
|
||||||
record_index: int
|
record_index: int
|
||||||
snap: Snapshot
|
snap: Snapshot
|
||||||
|
revision: int = 0
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def version(self) -> int:
|
def version(self) -> int:
|
||||||
@@ -47,6 +50,7 @@ class ChangeEvent:
|
|||||||
byte_pos: int
|
byte_pos: int
|
||||||
record_index: int
|
record_index: int
|
||||||
record: ChangeRecord
|
record: ChangeRecord
|
||||||
|
revision: int = 0
|
||||||
|
|
||||||
@property
|
@property
|
||||||
def version(self) -> int:
|
def version(self) -> int:
|
||||||
@@ -96,6 +100,7 @@ def scan_events(content: bytes, kanta: Kanta[Any]) -> tuple[list[Event], int]:
|
|||||||
state: dict[str, Any] = {}
|
state: dict[str, Any] = {}
|
||||||
events: list[Event] = []
|
events: list[Event] = []
|
||||||
change_count = 0
|
change_count = 0
|
||||||
|
revision = 0
|
||||||
for is_snapshot, payload, line_number, byte_pos in impl.framer.iter_records(
|
for is_snapshot, payload, line_number, byte_pos in impl.framer.iter_records(
|
||||||
content, 0
|
content, 0
|
||||||
):
|
):
|
||||||
@@ -113,12 +118,22 @@ def scan_events(content: bytes, kanta: Kanta[Any]) -> tuple[list[Event], int]:
|
|||||||
record_type="snapshot",
|
record_type="snapshot",
|
||||||
)
|
)
|
||||||
state = snap.state
|
state = snap.state
|
||||||
events.append(SnapshotEvent(line_number, byte_pos, record_index, snap))
|
# A k-marked snapshot carries an absolute revision, which may
|
||||||
|
# start past zero (rotated files). Legacy snapshots leave the
|
||||||
|
# file-relative count in effect, which is exact for them.
|
||||||
|
if snap.k:
|
||||||
|
revision = snap.r
|
||||||
|
events.append(
|
||||||
|
SnapshotEvent(line_number, byte_pos, record_index, snap, revision)
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
record = impl.serializer.decode(payload, type=ChangeRecord)
|
record = impl.serializer.decode(payload, type=ChangeRecord)
|
||||||
state = patch_state(state, record.diff)
|
state = patch(state, record.diff)
|
||||||
events.append(ChangeEvent(line_number, byte_pos, record_index, record))
|
|
||||||
change_count += 1
|
change_count += 1
|
||||||
|
revision += 1
|
||||||
|
events.append(
|
||||||
|
ChangeEvent(line_number, byte_pos, record_index, record, revision)
|
||||||
|
)
|
||||||
except msgspec.DecodeError as exc:
|
except msgspec.DecodeError as exc:
|
||||||
raise ReplayError(
|
raise ReplayError(
|
||||||
f"Parse error at {label}: {exc}",
|
f"Parse error at {label}: {exc}",
|
||||||
@@ -146,7 +161,7 @@ def replay_events(
|
|||||||
yield event, None, state
|
yield event, None, state
|
||||||
else:
|
else:
|
||||||
previous = copy.deepcopy(state)
|
previous = copy.deepcopy(state)
|
||||||
state = patch_state(state, event.record.diff)
|
state = patch(state, event.record.diff)
|
||||||
yield event, previous, state
|
yield event, previous, state
|
||||||
|
|
||||||
|
|
||||||
@@ -224,6 +239,21 @@ def _version_lines(events: list[Event]) -> dict[int, int]:
|
|||||||
return lines
|
return lines
|
||||||
|
|
||||||
|
|
||||||
|
def _revision_lines(events: list[Event]) -> dict[int, int]:
|
||||||
|
"""Map each selectable revision to the line of the record that produced it.
|
||||||
|
|
||||||
|
A revision bound ``rN`` addresses state N together with the record that
|
||||||
|
produced it: the change record whose resulting revision is N, or — when
|
||||||
|
that change is no longer in the file — the snapshot that materializes
|
||||||
|
state N. A change takes precedence over a redundant snapshot of the
|
||||||
|
same revision (file order).
|
||||||
|
"""
|
||||||
|
lines: dict[int, int] = {}
|
||||||
|
for event in events:
|
||||||
|
lines.setdefault(event.revision, event.line_number)
|
||||||
|
return lines
|
||||||
|
|
||||||
|
|
||||||
def _event_at_line(events: list[Event], line: int) -> Event | None:
|
def _event_at_line(events: list[Event], line: int) -> Event | None:
|
||||||
"""Return the event whose file line number exactly matches ``line``."""
|
"""Return the event whose file line number exactly matches ``line``."""
|
||||||
for event in events:
|
for event in events:
|
||||||
@@ -236,7 +266,7 @@ def _parse_bound(bound_str: str) -> tuple[str, int | None]:
|
|||||||
"""Parse a range bound with optional unit prefix (l, s, v) or change index."""
|
"""Parse a range bound with optional unit prefix (l, s, v) or change index."""
|
||||||
if not bound_str:
|
if not bound_str:
|
||||||
return "change", None
|
return "change", None
|
||||||
unit_map = {"l": "line", "s": "snapshot", "v": "version"}
|
unit_map = {"l": "line", "s": "snapshot", "v": "version", "r": "revision"}
|
||||||
if bound_str[0] in unit_map:
|
if bound_str[0] in unit_map:
|
||||||
unit = unit_map[bound_str[0]]
|
unit = unit_map[bound_str[0]]
|
||||||
rest = bound_str[1:]
|
rest = bound_str[1:]
|
||||||
@@ -287,6 +317,16 @@ def _bound_to_line(
|
|||||||
raise ValueError("version numbers do not support negative indexing")
|
raise ValueError("version numbers do not support negative indexing")
|
||||||
return _version_lines(events).get(value, eof)
|
return _version_lines(events).get(value, eof)
|
||||||
|
|
||||||
|
if unit == "revision":
|
||||||
|
if value < 0:
|
||||||
|
raise ValueError("revision numbers do not support negative indexing")
|
||||||
|
lines = _revision_lines(events)
|
||||||
|
if value in lines:
|
||||||
|
return lines[value]
|
||||||
|
if not lines or value < min(lines):
|
||||||
|
return 0
|
||||||
|
return eof
|
||||||
|
|
||||||
raise ValueError(f"unknown range unit: {unit}")
|
raise ValueError(f"unknown range unit: {unit}")
|
||||||
|
|
||||||
|
|
||||||
@@ -368,6 +408,23 @@ def select(spec: str, events: list[Event], total: int) -> Selection:
|
|||||||
later = [line for line in lines.values() if line > start_line]
|
later = [line for line in lines.values() if line > start_line]
|
||||||
return Selection(start_line, min(later) if later else end_of_file(events))
|
return Selection(start_line, min(later) if later else end_of_file(events))
|
||||||
|
|
||||||
|
if unit == "revision":
|
||||||
|
_negative_check(unit, value)
|
||||||
|
lines = _revision_lines(events)
|
||||||
|
if value not in lines:
|
||||||
|
raise RangeNotFoundError(
|
||||||
|
f"Revision {spec!r} not found in file"
|
||||||
|
f" ({_plural(len(lines), 'revision')})"
|
||||||
|
)
|
||||||
|
start_line = lines[value]
|
||||||
|
event = _event_at_line(events, start_line)
|
||||||
|
if isinstance(event, SnapshotEvent):
|
||||||
|
# The producing change is no longer in the file; the snapshot is
|
||||||
|
# the record materializing this state.
|
||||||
|
return Selection(start_line, start_line + 1, event)
|
||||||
|
later = [line for line in lines.values() if line > start_line]
|
||||||
|
return Selection(start_line, min(later) if later else end_of_file(events))
|
||||||
|
|
||||||
lines = _change_lines(events)
|
lines = _change_lines(events)
|
||||||
idx = total + value if value < 0 else value
|
idx = total + value if value < 0 else value
|
||||||
if not 0 <= idx < total:
|
if not 0 <= idx < total:
|
||||||
@@ -376,3 +433,176 @@ def select(spec: str, events: list[Event], total: int) -> Selection:
|
|||||||
)
|
)
|
||||||
end_line = lines[idx + 1] if idx + 1 < total else end_of_file(events)
|
end_line = lines[idx + 1] if idx + 1 < total else end_of_file(events)
|
||||||
return Selection(lines[idx], end_line)
|
return Selection(lines[idx], end_line)
|
||||||
|
|
||||||
|
|
||||||
|
_REL_RE = re.compile(r"(\d+)(s|min|h|d|w)")
|
||||||
|
_REL_UNITS = {"s": 1, "min": 60, "h": 3600, "d": 86400, "w": 604800}
|
||||||
|
|
||||||
|
# ISO 8601, UTC: hyphenated date with optional T-time (colons optional), or a
|
||||||
|
# compact date requiring T. Precision (day/hour/minute/second) selects the
|
||||||
|
# period the point denotes.
|
||||||
|
_ISO_NORM = re.compile(
|
||||||
|
r"(\d{4})-(\d{2})-(\d{2})(?:T(\d{2})(?::?(\d{2}))?(?::?(\d{2}))?)?"
|
||||||
|
)
|
||||||
|
_ISO_COMPACT = re.compile(r"(\d{4})(\d{2})(\d{2})T(\d{2})(?:(\d{2}))?(?:(\d{2}))?")
|
||||||
|
|
||||||
|
|
||||||
|
def _parse_iso_point(text: str) -> tuple[datetime, timedelta]:
|
||||||
|
"""Parse an ISO point into its UTC start and the period it denotes."""
|
||||||
|
match = _ISO_NORM.fullmatch(text) or _ISO_COMPACT.fullmatch(text)
|
||||||
|
if match is None:
|
||||||
|
raise ValueError(
|
||||||
|
f"invalid time {text!r} (UTC ISO like 2026-09-28, 2026-09-28T10:30"
|
||||||
|
" or 20260928T1030)"
|
||||||
|
)
|
||||||
|
year, month, day, hour, minute, second = (
|
||||||
|
int(group) if group else None for group in match.groups()
|
||||||
|
)
|
||||||
|
assert year is not None and month is not None and day is not None
|
||||||
|
try:
|
||||||
|
base = datetime(
|
||||||
|
year, month, day, hour or 0, minute or 0, second or 0, tzinfo=UTC
|
||||||
|
)
|
||||||
|
except ValueError:
|
||||||
|
raise ValueError(
|
||||||
|
f"invalid time {text!r} (UTC ISO like 2026-09-28, 2026-09-28T10:30"
|
||||||
|
" or 20260928T1030)"
|
||||||
|
) from None
|
||||||
|
if second is not None:
|
||||||
|
period = timedelta(seconds=1)
|
||||||
|
elif minute is not None:
|
||||||
|
period = timedelta(minutes=1)
|
||||||
|
elif hour is not None:
|
||||||
|
period = timedelta(hours=1)
|
||||||
|
else:
|
||||||
|
period = timedelta(days=1)
|
||||||
|
return base, period
|
||||||
|
|
||||||
|
|
||||||
|
def is_time_spec(spec: str) -> bool:
|
||||||
|
"""Return True if the spec is a time spec rather than a position spec.
|
||||||
|
|
||||||
|
Time specs are relative (``15min``, ``1h``, ``7d`` — a trailing unit
|
||||||
|
letter), ISO points or ranges (a hyphenated date, a ``T`` time, or a
|
||||||
|
``/`` separator). A leading hyphen stays a negative plain index.
|
||||||
|
"""
|
||||||
|
if _REL_RE.fullmatch(spec):
|
||||||
|
return True
|
||||||
|
if "/" in spec:
|
||||||
|
return True
|
||||||
|
return bool(
|
||||||
|
_ISO_NORM.fullmatch(spec) or _ISO_COMPACT.fullmatch(spec)
|
||||||
|
) or _looks_like_iso(spec)
|
||||||
|
|
||||||
|
|
||||||
|
def _looks_like_iso(spec: str) -> bool:
|
||||||
|
"""Heuristic for malformed ISO specs, so they error as times, not indexes."""
|
||||||
|
return bool(re.match(r"\d{4}-", spec) or re.match(r"\d{8}T", spec))
|
||||||
|
|
||||||
|
|
||||||
|
def parse_time_spec(
|
||||||
|
spec: str, now: datetime
|
||||||
|
) -> tuple[datetime | None, datetime | None]:
|
||||||
|
"""Resolve a time spec into a UTC [start, end) window; None = unbounded.
|
||||||
|
|
||||||
|
A relative spec covers the last N units ending at ``now``. An ISO point
|
||||||
|
covers its natural period (day/hour/minute/second by precision). An ISO
|
||||||
|
range ``a/b`` runs from the start of a's period through the end of b's
|
||||||
|
(inclusive); either side may be left open.
|
||||||
|
"""
|
||||||
|
match = _REL_RE.fullmatch(spec)
|
||||||
|
if match:
|
||||||
|
delta = timedelta(seconds=int(match.group(1)) * _REL_UNITS[match.group(2)])
|
||||||
|
return now - delta, now
|
||||||
|
if "/" in spec:
|
||||||
|
start_str, end_str = spec.split("/", 1)
|
||||||
|
start = _parse_iso_point(start_str)[0] if start_str else None
|
||||||
|
end = None
|
||||||
|
if end_str:
|
||||||
|
end_base, end_period = _parse_iso_point(end_str)
|
||||||
|
end = end_base + end_period
|
||||||
|
return start, end
|
||||||
|
base, period = _parse_iso_point(spec)
|
||||||
|
return base, base + period
|
||||||
|
|
||||||
|
|
||||||
|
def is_plain_spec(spec: str) -> bool:
|
||||||
|
"""Return True if every bound of the spec is a plain number (no unit letter).
|
||||||
|
|
||||||
|
Plain specs index into the filtered output items rather than the file.
|
||||||
|
"""
|
||||||
|
if ".." in spec:
|
||||||
|
bounds = spec.split("..", 1)
|
||||||
|
elif ":" in spec:
|
||||||
|
bounds = spec.split(":", 1)
|
||||||
|
else:
|
||||||
|
bounds = [spec]
|
||||||
|
return all(_parse_bound(b)[0] == "change" for b in bounds)
|
||||||
|
|
||||||
|
|
||||||
|
def select_all(
|
||||||
|
specs: list[str], events: list[Event], total: int
|
||||||
|
) -> tuple[Selection, list[str], list[str]]:
|
||||||
|
"""Resolve repeated ``-r`` specs into a database window plus plain specs.
|
||||||
|
|
||||||
|
Unit specs (``l``, ``s``, ``v``, ``r``, and mixed specs) resolve to line
|
||||||
|
windows via :func:`select` and intersect restrictively. The snapshot
|
||||||
|
shortcut applies only to a single unit spec. Plain-number specs and
|
||||||
|
time specs are returned unresolved for the caller to apply against the
|
||||||
|
filtered output.
|
||||||
|
"""
|
||||||
|
time_specs: list[str] = []
|
||||||
|
plain_specs: list[str] = []
|
||||||
|
unit_specs: list[str] = []
|
||||||
|
for spec in specs:
|
||||||
|
if is_time_spec(spec):
|
||||||
|
time_specs.append(spec)
|
||||||
|
elif is_plain_spec(spec):
|
||||||
|
plain_specs.append(spec)
|
||||||
|
else:
|
||||||
|
unit_specs.append(spec)
|
||||||
|
if not unit_specs:
|
||||||
|
return Selection(0, end_of_file(events)), plain_specs, time_specs
|
||||||
|
selections = [select(spec, events, total) for spec in unit_specs]
|
||||||
|
if len(selections) == 1:
|
||||||
|
return selections[0], plain_specs, time_specs
|
||||||
|
start = max(sel.start_line for sel in selections)
|
||||||
|
end = min(sel.end_line for sel in selections)
|
||||||
|
return Selection(min(start, end), end), plain_specs, time_specs
|
||||||
|
|
||||||
|
|
||||||
|
def _clamp_index(value: int, count: int) -> int:
|
||||||
|
"""Clamp an item index Python-slice style; negatives count from the end."""
|
||||||
|
if value < 0:
|
||||||
|
value = count + value
|
||||||
|
return max(0, min(value, count))
|
||||||
|
|
||||||
|
|
||||||
|
def slice_items(specs: list[str], count: int) -> tuple[int, int]:
|
||||||
|
"""Resolve plain item specs into ``[start, end)`` indices over ``count`` items.
|
||||||
|
|
||||||
|
Multiple specs intersect restrictively; ranges truncate Python-slice
|
||||||
|
style. A single index must exist and raises :class:`RangeNotFoundError`
|
||||||
|
otherwise. Syntax errors raise :class:`ValueError`.
|
||||||
|
"""
|
||||||
|
start, end = 0, count
|
||||||
|
for spec in specs:
|
||||||
|
if ":" in spec or ".." in spec:
|
||||||
|
sep = ".." if ".." in spec else ":"
|
||||||
|
start_str, end_str = spec.split(sep, 1)
|
||||||
|
item_start = _clamp_index(int(start_str), count) if start_str else 0
|
||||||
|
item_end = _clamp_index(int(end_str), count) if end_str else count
|
||||||
|
if sep == ".." and end_str:
|
||||||
|
item_end = min(item_end + 1, count)
|
||||||
|
else:
|
||||||
|
value = int(spec)
|
||||||
|
idx = count + value if value < 0 else value
|
||||||
|
if not 0 <= idx < count:
|
||||||
|
raise RangeNotFoundError(
|
||||||
|
f"Item index {spec!r} not found in filtered output"
|
||||||
|
f" ({_plural(count, 'item')})"
|
||||||
|
)
|
||||||
|
item_start, item_end = idx, idx + 1
|
||||||
|
start = max(start, item_start)
|
||||||
|
end = min(end, item_end)
|
||||||
|
return start, max(start, end)
|
||||||
+21
-5
@@ -16,11 +16,11 @@ from pathlib import Path
|
|||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from kanta.exceptions import DatabaseError
|
from kanta.exceptions import DatabaseError
|
||||||
from kanta.structs import ChangeRecord, Snapshot
|
from kanta.structs import META_FORMAT, ChangeRecord, Snapshot
|
||||||
from kanta.serialization.base import Serializer, apply_diff
|
from kanta.serialization.base import Serializer, apply_diff
|
||||||
from kanta.serialization.framing import Framer
|
from kanta.serialization.framing import Framer
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
@@ -174,10 +174,18 @@ def plan_rotation(
|
|||||||
# Build the new content: leading cutoff snapshot, retained changes
|
# Build the new content: leading cutoff snapshot, retained changes
|
||||||
# re-framed at fresh offsets, and a final snapshot when enough changes
|
# re-framed at fresh offsets, and a final snapshot when enough changes
|
||||||
# survived to warrant one (mirrors the regular snapshot policy).
|
# survived to warrant one (mirrors the regular snapshot policy).
|
||||||
|
# Revisions are absolute positions counted from the start of the
|
||||||
|
# original file, so they survive the rotation unchanged.
|
||||||
|
r_at_cutoff = sum(1 for e in changes if e.byte_pos < cutoff_end)
|
||||||
out = bytearray()
|
out = bytearray()
|
||||||
leading = serializer.encode(
|
leading = serializer.encode(
|
||||||
Snapshot(
|
Snapshot(
|
||||||
ts=rotated_ts, v=version_at_cutoff, state=state_at_cutoff, m=m_at_cutoff
|
ts=rotated_ts,
|
||||||
|
v=version_at_cutoff,
|
||||||
|
state=state_at_cutoff,
|
||||||
|
m=m_at_cutoff,
|
||||||
|
r=r_at_cutoff,
|
||||||
|
k=META_FORMAT,
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
out += framer.frame_snapshot(leading, record_offset=0)
|
out += framer.frame_snapshot(leading, record_offset=0)
|
||||||
@@ -186,7 +194,14 @@ def plan_rotation(
|
|||||||
out += framer.frame_change(payload, record_offset=len(out))
|
out += framer.frame_change(payload, record_offset=len(out))
|
||||||
if len(retained) >= min_diffs:
|
if len(retained) >= min_diffs:
|
||||||
closing = serializer.encode(
|
closing = serializer.encode(
|
||||||
Snapshot(ts=now, v=final_version, state=state, m=final_m)
|
Snapshot(
|
||||||
|
ts=now,
|
||||||
|
v=final_version,
|
||||||
|
state=state,
|
||||||
|
m=final_m,
|
||||||
|
r=len(changes),
|
||||||
|
k=META_FORMAT,
|
||||||
|
)
|
||||||
)
|
)
|
||||||
out += framer.frame_snapshot(closing, record_offset=len(out))
|
out += framer.frame_snapshot(closing, record_offset=len(out))
|
||||||
|
|
||||||
@@ -216,7 +231,8 @@ def execute_rotation(
|
|||||||
f.truncate(plan.cutoff_end)
|
f.truncate(plan.cutoff_end)
|
||||||
if log:
|
if log:
|
||||||
_logger.info(
|
_logger.info(
|
||||||
"rotated %s: kept %d change record(s), history before %s moved to %s",
|
"Rotated database %s: kept %d change record(s), "
|
||||||
|
"moved history before %s to %s",
|
||||||
path,
|
path,
|
||||||
plan.retained_changes,
|
plan.retained_changes,
|
||||||
plan.rotated_ts.isoformat(),
|
plan.rotated_ts.isoformat(),
|
||||||
|
|||||||
@@ -25,12 +25,16 @@ class ReplayResult:
|
|||||||
has_migration: bool = False,
|
has_migration: bool = False,
|
||||||
last_snapshot_mtime: float | None = None,
|
last_snapshot_mtime: float | None = None,
|
||||||
m: datetime | None = None,
|
m: datetime | None = None,
|
||||||
|
rev: int = 0,
|
||||||
|
legacy_snapshot: bool = False,
|
||||||
):
|
):
|
||||||
self.state = state
|
self.state = state
|
||||||
self.version = version
|
self.version = version
|
||||||
self.has_migration = has_migration
|
self.has_migration = has_migration
|
||||||
self.last_snapshot_mtime = last_snapshot_mtime
|
self.last_snapshot_mtime = last_snapshot_mtime
|
||||||
self.m = m
|
self.m = m
|
||||||
|
self.rev = rev
|
||||||
|
self.legacy_snapshot = legacy_snapshot
|
||||||
|
|
||||||
|
|
||||||
class Serializer(Protocol):
|
class Serializer(Protocol):
|
||||||
@@ -61,6 +65,8 @@ def replay(
|
|||||||
last_snapshot_mtime: float | None = None
|
last_snapshot_mtime: float | None = None
|
||||||
m: datetime | None = None
|
m: datetime | None = None
|
||||||
has_migration = False
|
has_migration = False
|
||||||
|
r_base = 0
|
||||||
|
legacy_snapshot = False
|
||||||
|
|
||||||
if snap_payload is not None:
|
if snap_payload is not None:
|
||||||
try:
|
try:
|
||||||
@@ -76,7 +82,21 @@ def replay(
|
|||||||
version = snap.v
|
version = snap.v
|
||||||
last_snapshot_mtime = snap.ts.timestamp()
|
last_snapshot_mtime = snap.ts.timestamp()
|
||||||
m = snap.m
|
m = snap.m
|
||||||
|
if snap.k == 0:
|
||||||
|
# Legacy snapshot: r cannot be trusted, so count change records
|
||||||
|
# from the start of the file (framing only, no payload decode).
|
||||||
|
legacy_snapshot = True
|
||||||
|
r_base = sum(
|
||||||
|
1
|
||||||
|
for is_snapshot, _payload, _line, byte_pos in framer.iter_records(
|
||||||
|
data, 0
|
||||||
|
)
|
||||||
|
if not is_snapshot and byte_pos < snap_byte_pos
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
r_base = snap.r
|
||||||
|
|
||||||
|
changes = 0
|
||||||
for is_snapshot, payload, line_number, byte_pos in framer.iter_records(
|
for is_snapshot, payload, line_number, byte_pos in framer.iter_records(
|
||||||
data, offset
|
data, offset
|
||||||
):
|
):
|
||||||
@@ -111,12 +131,15 @@ def replay(
|
|||||||
m = change.m
|
m = change.m
|
||||||
version = change.v
|
version = change.v
|
||||||
state = _patch_state(state, change.diff)
|
state = _patch_state(state, change.diff)
|
||||||
|
changes += 1
|
||||||
return ReplayResult(
|
return ReplayResult(
|
||||||
state=state,
|
state=state,
|
||||||
version=version,
|
version=version,
|
||||||
has_migration=has_migration,
|
has_migration=has_migration,
|
||||||
last_snapshot_mtime=last_snapshot_mtime,
|
last_snapshot_mtime=last_snapshot_mtime,
|
||||||
m=m,
|
m=m,
|
||||||
|
rev=r_base + changes,
|
||||||
|
legacy_snapshot=legacy_snapshot,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
+15
-6
@@ -6,11 +6,11 @@ import logging
|
|||||||
from collections.abc import Callable
|
from collections.abc import Callable
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
from kanta.structs import Snapshot
|
from kanta.structs import META_FORMAT, Snapshot
|
||||||
from kanta.serialization import JsonSerializer, Serializer
|
from kanta.serialization import JsonSerializer, Serializer
|
||||||
from kanta.serialization.framing import Framer, LineFramer
|
from kanta.serialization.framing import Framer, LineFramer
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
MINDIFFS = 100
|
MINDIFFS = 100
|
||||||
|
|
||||||
@@ -50,6 +50,7 @@ class SnapshotState:
|
|||||||
state: dict,
|
state: dict,
|
||||||
m: datetime | None = None,
|
m: datetime | None = None,
|
||||||
now: Callable[[], datetime] | None = None,
|
now: Callable[[], datetime] | None = None,
|
||||||
|
r: int = 0,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Write snapshot when thresholds/time policy allows it."""
|
"""Write snapshot when thresholds/time policy allows it."""
|
||||||
force = self._force_pending
|
force = self._force_pending
|
||||||
@@ -66,16 +67,24 @@ class SnapshotState:
|
|||||||
if not file.is_open:
|
if not file.is_open:
|
||||||
return
|
return
|
||||||
try:
|
try:
|
||||||
self._write(file, version, state, ts, m=m)
|
self._write(file, version, state, ts, m=m, r=r)
|
||||||
self._force_pending = False
|
self._force_pending = False
|
||||||
except Exception as exc:
|
except Exception as exc:
|
||||||
_logger.error("snapshot: failed to write snapshot: %r", exc)
|
_logger.error("Kanta snapshot failed: %r", exc)
|
||||||
|
|
||||||
def _write(
|
def _write(
|
||||||
self, file, version: int, state: dict, now: datetime, m: datetime | None = None
|
self,
|
||||||
|
file,
|
||||||
|
version: int,
|
||||||
|
state: dict,
|
||||||
|
now: datetime,
|
||||||
|
m: datetime | None = None,
|
||||||
|
r: int = 0,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Write a snapshot and update internal state."""
|
"""Write a snapshot and update internal state."""
|
||||||
payload = self._serializer.encode(Snapshot(ts=now, v=version, state=state, m=m))
|
payload = self._serializer.encode(
|
||||||
|
Snapshot(ts=now, v=version, state=state, m=m, r=r, k=META_FORMAT)
|
||||||
|
)
|
||||||
record_offset = file.size() if hasattr(file, "size") else 0
|
record_offset = file.size() if hasattr(file, "size") else 0
|
||||||
file.write(self._framer.frame_snapshot(payload, record_offset=record_offset))
|
file.write(self._framer.frame_snapshot(payload, record_offset=record_offset))
|
||||||
self.changes = 0
|
self.changes = 0
|
||||||
|
|||||||
+20
-1
@@ -5,6 +5,11 @@ from typing import Any
|
|||||||
|
|
||||||
import msgspec
|
import msgspec
|
||||||
|
|
||||||
|
# On-disk meta format version, stored in the snapshot ``k`` field and bumped
|
||||||
|
# manually when the meta format changes. k0 (missing field) predates
|
||||||
|
# revision numbers; k1 introduces them (see docs/database.md).
|
||||||
|
META_FORMAT = 1
|
||||||
|
|
||||||
|
|
||||||
class ChangeRecord(msgspec.Struct, omit_defaults=True, kw_only=True):
|
class ChangeRecord(msgspec.Struct, omit_defaults=True, kw_only=True):
|
||||||
"""A single change record in the JSONL log.
|
"""A single change record in the JSONL log.
|
||||||
@@ -27,9 +32,23 @@ class ChangeRecord(msgspec.Struct, omit_defaults=True, kw_only=True):
|
|||||||
|
|
||||||
|
|
||||||
class Snapshot(msgspec.Struct, omit_defaults=True):
|
class Snapshot(msgspec.Struct, omit_defaults=True):
|
||||||
"""Full state snapshot embedded in the JSONL log."""
|
"""Full state snapshot embedded in the JSONL log.
|
||||||
|
|
||||||
|
Attributes:
|
||||||
|
ts: Snapshot creation time.
|
||||||
|
v: Schema version represented by the snapshot.
|
||||||
|
state: Full state dictionary.
|
||||||
|
m: Optional domain modification timestamp.
|
||||||
|
r: Revision of the snapshotted state: the total number of change
|
||||||
|
records that occurred before this snapshot.
|
||||||
|
k: Meta format version (``META_FORMAT`` when written). 0 marks a
|
||||||
|
legacy snapshot whose ``r`` cannot be trusted; readers count
|
||||||
|
change records from the start of the file instead.
|
||||||
|
"""
|
||||||
|
|
||||||
ts: datetime
|
ts: datetime
|
||||||
v: int
|
v: int
|
||||||
state: dict[str, Any]
|
state: dict[str, Any]
|
||||||
m: datetime | None = None
|
m: datetime | None = None
|
||||||
|
r: int = 0
|
||||||
|
k: int = 0
|
||||||
@@ -7,13 +7,13 @@ from contextlib import contextmanager
|
|||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from kanta.diff import compute_diff
|
from kanta.diff import diff
|
||||||
from kanta.exceptions import DataIntegrityError
|
from kanta.exceptions import DataIntegrityError
|
||||||
from kanta.callbacks import InjectionContext
|
from kanta.callbacks import InjectionContext
|
||||||
from kanta.logging import _USER_PATH, LogEvent, emit_event, transaction_logger
|
from kanta.logging import _USER_PATH, LogEvent, emit_event, transaction_logger
|
||||||
from kanta.serialization import restore_data_in_place, struct_to_dict
|
from kanta.serialization import restore_data_in_place, struct_to_dict
|
||||||
|
|
||||||
_logger = logging.getLogger(__name__)
|
_logger = logging.getLogger("kanta")
|
||||||
|
|
||||||
|
|
||||||
def _build_logfmt(impl, previous: dict, current: dict):
|
def _build_logfmt(impl, previous: dict, current: dict):
|
||||||
@@ -65,19 +65,19 @@ def transaction(
|
|||||||
if current_dict != impl.statedict:
|
if current_dict != impl.statedict:
|
||||||
is_bootstrap = action in {"bootstrap"}
|
is_bootstrap = action in {"bootstrap"}
|
||||||
if not (is_bootstrap and not impl.statedict):
|
if not (is_bootstrap and not impl.statedict):
|
||||||
diff = compute_diff(impl.statedict, current_dict)
|
delta = diff(impl.statedict, current_dict)
|
||||||
if diff:
|
if delta:
|
||||||
_logger.critical(
|
_logger.critical(
|
||||||
"Database state modified outside of transaction! "
|
"Database state modified outside of transaction! "
|
||||||
"This indicates a bug where changes occurred without a transaction wrapper.\n"
|
"This indicates a bug where changes occurred without a transaction wrapper.\n"
|
||||||
"Changes detected: %s",
|
"Changes detected: %s",
|
||||||
diff,
|
delta,
|
||||||
)
|
)
|
||||||
raise DataIntegrityError(
|
raise DataIntegrityError(
|
||||||
"Database state modified outside of transaction",
|
"Database state modified outside of transaction",
|
||||||
db_path=impl.db_path,
|
db_path=impl.db_path,
|
||||||
action=action,
|
action=action,
|
||||||
diff=diff,
|
diff=delta,
|
||||||
)
|
)
|
||||||
|
|
||||||
impl.in_transaction = True
|
impl.in_transaction = True
|
||||||
@@ -86,8 +86,8 @@ def transaction(
|
|||||||
try:
|
try:
|
||||||
yield impl.data
|
yield impl.data
|
||||||
new_dict = struct_to_dict(impl.data, serializer=impl.serializer)
|
new_dict = struct_to_dict(impl.data, serializer=impl.serializer)
|
||||||
diff = compute_diff(impl.statedict, new_dict)
|
delta = diff(impl.statedict, new_dict)
|
||||||
if diff:
|
if delta:
|
||||||
if impl.callback_registry.has("validate"):
|
if impl.callback_registry.has("validate"):
|
||||||
impl.callback_registry.invoke_sync(
|
impl.callback_registry.invoke_sync(
|
||||||
"validate",
|
"validate",
|
||||||
|
|||||||
@@ -10,8 +10,12 @@ color instead of emitting a separate one.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import os
|
||||||
import re
|
import re
|
||||||
|
import sys
|
||||||
import unicodedata
|
import unicodedata
|
||||||
|
from contextlib import suppress
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
ESC = "\x1b["
|
ESC = "\x1b["
|
||||||
@@ -25,6 +29,21 @@ def strip_ansi(text: str) -> str:
|
|||||||
return ANSI_RE.sub("", text)
|
return ANSI_RE.sub("", text)
|
||||||
|
|
||||||
|
|
||||||
|
def use_color(stream: io.TextIOBase = sys.stderr) -> bool:
|
||||||
|
"""Test if the stream supports color codes."""
|
||||||
|
if os.environ.get("NO_COLOR"): # Non empty means no (no-color.org)
|
||||||
|
return False
|
||||||
|
if os.environ.get("FORCE_COLOR", "") not in {"", "0"}: # force-color.org, node
|
||||||
|
return True
|
||||||
|
if hasattr(stream, "isatty") and stream.isatty():
|
||||||
|
return True
|
||||||
|
with suppress(KeyError, ValueError, OSError): # Journald does color (-ocat)
|
||||||
|
dev, ino = map(int, os.environ["JOURNAL_STREAM"].split(":", 1))
|
||||||
|
st = os.fstat(stream.fileno())
|
||||||
|
return st.st_dev == dev and st.st_ino == ino
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
def displaywidth(text: str) -> int:
|
def displaywidth(text: str) -> int:
|
||||||
"""Return the terminal column width of *text*, ignoring ANSI sequences.
|
"""Return the terminal column width of *text*, ignoring ANSI sequences.
|
||||||
|
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ bin = [
|
|||||||
[project.urls]
|
[project.urls]
|
||||||
Homepage = "https://vasanko.com/coders/kanta"
|
Homepage = "https://vasanko.com/coders/kanta"
|
||||||
Repository = "https://git.zi.fi/LeoVasanko/kanta"
|
Repository = "https://git.zi.fi/LeoVasanko/kanta"
|
||||||
|
Issues = "https://github.com/LeoVasanko/kanta"
|
||||||
|
|
||||||
[dependency-groups]
|
[dependency-groups]
|
||||||
dev = [
|
dev = [
|
||||||
|
|||||||
@@ -1,3 +1,5 @@
|
|||||||
|
import logging
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from kanta.serialization import JsonSerializer, MsgPackSerializer
|
from kanta.serialization import JsonSerializer, MsgPackSerializer
|
||||||
@@ -12,3 +14,20 @@ from kanta.serialization import JsonSerializer, MsgPackSerializer
|
|||||||
)
|
)
|
||||||
def format_config(request):
|
def format_config(request):
|
||||||
return request.param
|
return request.param
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture(autouse=True)
|
||||||
|
def _kanta_event_loggers_propagate():
|
||||||
|
"""Let kanta's event loggers propagate so caplog captures their records.
|
||||||
|
|
||||||
|
Kanta configures them with ``propagate = False`` at import time, which
|
||||||
|
would hide their records from pytest's root-logger capture handler.
|
||||||
|
"""
|
||||||
|
names = ("kanta.bootstrap", "kanta.migration", "kanta.transaction")
|
||||||
|
loggers = [logging.getLogger(name) for name in names]
|
||||||
|
previous = [logger.propagate for logger in loggers]
|
||||||
|
for logger in loggers:
|
||||||
|
logger.propagate = True
|
||||||
|
yield
|
||||||
|
for logger, propagate in zip(loggers, previous):
|
||||||
|
logger.propagate = propagate
|
||||||
+49
-1
@@ -3,6 +3,8 @@
|
|||||||
import sys
|
import sys
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
from kanta.__main__ import (
|
from kanta.__main__ import (
|
||||||
_extra_import_paths,
|
_extra_import_paths,
|
||||||
_format_ts,
|
_format_ts,
|
||||||
@@ -76,8 +78,10 @@ def test_extra_import_paths_ignores_other_python_versions(tmp_path, monkeypatch)
|
|||||||
assert str(other_site) not in sys.path
|
assert str(other_site) not in sys.path
|
||||||
|
|
||||||
|
|
||||||
def test_cli_snapshot_line_format(tmp_path, capsys):
|
def test_cli_snapshot_line_format(tmp_path, capsys, monkeypatch):
|
||||||
"""Snapshot lines are timestamped and colored with metadata."""
|
"""Snapshot lines are timestamped and colored with metadata."""
|
||||||
|
monkeypatch.setenv("FORCE_COLOR", "1")
|
||||||
|
monkeypatch.delenv("NO_COLOR", raising=False)
|
||||||
path = tmp_path / "test.kantadb"
|
path = tmp_path / "test.kantadb"
|
||||||
ts = datetime(2026, 8, 12, 10, 6, 52, 375398, tzinfo=UTC)
|
ts = datetime(2026, 8, 12, 10, 6, 52, 375398, tzinfo=UTC)
|
||||||
mtime = datetime(2026, 8, 12, 9, 0, 0, tzinfo=UTC)
|
mtime = datetime(2026, 8, 12, 9, 0, 0, tzinfo=UTC)
|
||||||
@@ -104,6 +108,50 @@ def test_cli_snapshot_line_format(tmp_path, capsys):
|
|||||||
assert "\x1b[38;5;250m v1 2026-08-12 09:00:00" in err
|
assert "\x1b[38;5;250m v1 2026-08-12 09:00:00" in err
|
||||||
assert "\x1b[38;5;242m 13 B" in err
|
assert "\x1b[38;5;242m 13 B" in err
|
||||||
|
|
||||||
|
# The change record header carries the revision it produces.
|
||||||
|
assert "\x1b[38;5;250m r1" in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_strips_ansi_without_color_support(tmp_path, capsys, monkeypatch):
|
||||||
|
"""Without a tty and with NO_COLOR set, output contains no ANSI codes."""
|
||||||
|
monkeypatch.setenv("NO_COLOR", "1")
|
||||||
|
monkeypatch.delenv("FORCE_COLOR", raising=False)
|
||||||
|
path = tmp_path / "test.kantadb"
|
||||||
|
ts = datetime(2026, 8, 12, 10, 6, 52, 375398, tzinfo=UTC)
|
||||||
|
serializer = JsonSerializer()
|
||||||
|
framer = LineFramer()
|
||||||
|
|
||||||
|
snapshot = Snapshot(ts=ts, v=1, m=None, state={"counter": 5})
|
||||||
|
change = ChangeRecord(ts=ts, a="inc", v=1, u="user1", diff={"counter": 6})
|
||||||
|
data = framer.frame_snapshot(
|
||||||
|
serializer.encode(snapshot), record_offset=0
|
||||||
|
) + framer.frame_change(serializer.encode(change), record_offset=0)
|
||||||
|
path.write_bytes(data)
|
||||||
|
|
||||||
|
code = main([str(path)])
|
||||||
|
assert code == 0
|
||||||
|
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "\x1b[" not in err
|
||||||
|
assert "snapshot s0" in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_version_on_help_and_version_flag(capsys):
|
||||||
|
"""--help and --version print the installed package version."""
|
||||||
|
import importlib.metadata
|
||||||
|
|
||||||
|
version = importlib.metadata.version("kanta")
|
||||||
|
|
||||||
|
with pytest.raises(SystemExit) as help_exit:
|
||||||
|
main(["--help"])
|
||||||
|
assert help_exit.value.code == 0
|
||||||
|
assert f"kanta {version}" in capsys.readouterr().out
|
||||||
|
|
||||||
|
with pytest.raises(SystemExit) as version_exit:
|
||||||
|
main(["--version"])
|
||||||
|
assert version_exit.value.code == 0
|
||||||
|
assert capsys.readouterr().out.strip() == f"kanta {version}"
|
||||||
|
|
||||||
|
|
||||||
def test_import_dotted_from_file_path(tmp_path):
|
def test_import_dotted_from_file_path(tmp_path):
|
||||||
"""--data can be a filesystem path with an optional colon-separated symbol."""
|
"""--data can be a filesystem path with an optional colon-separated symbol."""
|
||||||
|
|||||||
+40
-40
@@ -3,7 +3,7 @@
|
|||||||
jsondiff is a dev dependency used only here, to verify that:
|
jsondiff is a dev dependency used only here, to verify that:
|
||||||
|
|
||||||
- jsondiff.patch(..., marshal=True) can apply patches produced by
|
- jsondiff.patch(..., marshal=True) can apply patches produced by
|
||||||
compute_diff (our format is a subset of jsondiff's marshaled syntax);
|
diff (our format is a subset of jsondiff's marshaled syntax);
|
||||||
- apply_diff can apply patches produced by jsondiff.diff(..., marshal=True),
|
- apply_diff can apply patches produced by jsondiff.diff(..., marshal=True),
|
||||||
including positional $insert/$delete list edits and per-index nested diffs.
|
including positional $insert/$delete list edits and per-index nested diffs.
|
||||||
"""
|
"""
|
||||||
@@ -11,88 +11,88 @@ jsondiff is a dev dependency used only here, to verify that:
|
|||||||
import jsondiff
|
import jsondiff
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
from kanta.diff import compute_diff, patch_state
|
from kanta.diff import diff, patch
|
||||||
from kanta.logging import format_diff
|
from kanta.logging import format_diff
|
||||||
from kanta.serialization.base import apply_diff
|
from kanta.serialization.base import apply_diff
|
||||||
|
|
||||||
# --- Producer: compute_diff ------------------------------------------------
|
# --- Producer: diff ------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def test_no_diff():
|
def test_no_diff():
|
||||||
assert compute_diff({"a": 1}, {"a": 1}) is None
|
assert diff({"a": 1}, {"a": 1}) is None
|
||||||
assert compute_diff({}, {}) is None
|
assert diff({}, {}) is None
|
||||||
|
|
||||||
|
|
||||||
def test_simple_diff():
|
def test_simple_diff():
|
||||||
diff = compute_diff({"a": 1}, {"a": 2})
|
delta = diff({"a": 1}, {"a": 2})
|
||||||
assert diff is not None
|
assert delta is not None
|
||||||
assert diff == {"a": 2}
|
assert delta == {"a": 2}
|
||||||
|
|
||||||
|
|
||||||
def test_nested_diff():
|
def test_nested_diff():
|
||||||
diff = compute_diff({"x": {"y": 1}}, {"x": {"y": 2}})
|
delta = diff({"x": {"y": 1}}, {"x": {"y": 2}})
|
||||||
assert diff == {"x": {"y": 2}}
|
assert delta == {"x": {"y": 2}}
|
||||||
|
|
||||||
|
|
||||||
def test_key_added():
|
def test_key_added():
|
||||||
assert compute_diff({"a": 1}, {"a": 1, "b": 2}) == {"b": 2}
|
assert diff({"a": 1}, {"a": 1, "b": 2}) == {"b": 2}
|
||||||
|
|
||||||
|
|
||||||
def test_key_removed():
|
def test_key_removed():
|
||||||
assert compute_diff({"a": 1, "b": 2}, {"a": 1}) == {"$delete": ["b"]}
|
assert diff({"a": 1, "b": 2}, {"a": 1}) == {"$delete": ["b"]}
|
||||||
|
|
||||||
|
|
||||||
def test_last_key_removed_is_delete_not_replace():
|
def test_last_key_removed_is_delete_not_replace():
|
||||||
# jsondiff's minimal-diff search emits {"$replace": {}} here; we emit
|
# jsondiff's minimal-diff search emits {"$replace": {}} here; we emit
|
||||||
# what actually happened: the key was deleted.
|
# what actually happened: the key was deleted.
|
||||||
assert compute_diff({"a": 1}, {}) == {"$delete": ["a"]}
|
assert diff({"a": 1}, {}) == {"$delete": ["a"]}
|
||||||
assert compute_diff({"x": {"y": 1}}, {"x": {}}) == {"x": {"$delete": ["y"]}}
|
assert diff({"x": {"y": 1}}, {"x": {}}) == {"x": {"$delete": ["y"]}}
|
||||||
|
|
||||||
|
|
||||||
def test_list_changes_are_full_assignment():
|
def test_list_changes_are_full_assignment():
|
||||||
# No $insert/$delete positional edits: lists are replaced wholesale.
|
# No $insert/$delete positional edits: lists are replaced wholesale.
|
||||||
assert compute_diff({"l": [1, 2]}, {"l": [1, 2, 3]}) == {"l": [1, 2, 3]}
|
assert diff({"l": [1, 2]}, {"l": [1, 2, 3]}) == {"l": [1, 2, 3]}
|
||||||
assert compute_diff({"l": [1, 2, 3]}, {"l": [1, 3]}) == {"l": [1, 3]}
|
assert diff({"l": [1, 2, 3]}, {"l": [1, 3]}) == {"l": [1, 3]}
|
||||||
assert compute_diff({"l": [1]}, {"l": []}) == {"l": []}
|
assert diff({"l": [1]}, {"l": []}) == {"l": []}
|
||||||
|
|
||||||
|
|
||||||
def test_list_with_unchanged_prefix_is_full_assignment():
|
def test_list_with_unchanged_prefix_is_full_assignment():
|
||||||
diff = compute_diff({"l": ["a", "b", "c"]}, {"l": ["a", "x", "b", "c"]})
|
delta = diff({"l": ["a", "b", "c"]}, {"l": ["a", "x", "b", "c"]})
|
||||||
assert diff == {"l": ["a", "x", "b", "c"]}
|
assert delta == {"l": ["a", "x", "b", "c"]}
|
||||||
|
|
||||||
|
|
||||||
def test_type_changes_are_full_assignment():
|
def test_type_changes_are_full_assignment():
|
||||||
assert compute_diff({"a": {"x": 1}}, {"a": [1]}) == {"a": [1]}
|
assert diff({"a": {"x": 1}}, {"a": [1]}) == {"a": [1]}
|
||||||
# A dict replacing a non-dict is a plain assignment too: the consumer
|
# A dict replacing a non-dict is a plain assignment too: the consumer
|
||||||
# sees from the old value whether to patch (dict) or replace.
|
# sees from the old value whether to patch (dict) or replace.
|
||||||
assert compute_diff({"a": [1]}, {"a": {"x": 1}}) == {"a": {"x": 1}}
|
assert diff({"a": [1]}, {"a": {"x": 1}}) == {"a": {"x": 1}}
|
||||||
assert compute_diff({"a": 1}, {"a": None}) == {"a": None}
|
assert diff({"a": 1}, {"a": None}) == {"a": None}
|
||||||
|
|
||||||
|
|
||||||
def test_new_dict_value_assigned_wholesale():
|
def test_new_dict_value_assigned_wholesale():
|
||||||
assert compute_diff({}, {"a": {"x": 1}}) == {"a": {"x": 1}}
|
assert diff({}, {"a": {"x": 1}}) == {"a": {"x": 1}}
|
||||||
|
|
||||||
|
|
||||||
def test_dollar_keys_escaped():
|
def test_dollar_keys_escaped():
|
||||||
assert compute_diff({}, {"$weird": 1}) == {"$$weird": 1}
|
assert diff({}, {"$weird": 1}) == {"$$weird": 1}
|
||||||
assert compute_diff({"$weird": 1}, {"$weird": 2}) == {"$$weird": 2}
|
assert diff({"$weird": 1}, {"$weird": 2}) == {"$$weird": 2}
|
||||||
assert compute_diff({"$weird": 1}, {}) == {"$delete": ["$$weird"]}
|
assert diff({"$weird": 1}, {}) == {"$delete": ["$$weird"]}
|
||||||
|
|
||||||
|
|
||||||
def test_dollar_values_not_escaped():
|
def test_dollar_values_not_escaped():
|
||||||
# Only keys are escaped; values are stored verbatim, even "$delete".
|
# Only keys are escaped; values are stored verbatim, even "$delete".
|
||||||
assert compute_diff({"s": 1}, {"s": "$y"}) == {"s": "$y"}
|
assert diff({"s": 1}, {"s": "$y"}) == {"s": "$y"}
|
||||||
assert compute_diff({"s": 1}, {"s": "$delete"}) == {"s": "$delete"}
|
assert diff({"s": 1}, {"s": "$delete"}) == {"s": "$delete"}
|
||||||
assert compute_diff({}, {"o": {"s": "$y", "l": ["$z"]}}) == {
|
assert diff({}, {"o": {"s": "$y", "l": ["$z"]}}) == {
|
||||||
"o": {"s": "$y", "l": ["$z"]}
|
"o": {"s": "$y", "l": ["$z"]}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
# --- Consumer: apply_diff / patch_state -------------------------------------
|
# --- Consumer: apply_diff / patch -------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def test_patch_state_delegates():
|
def test_patch_delegates():
|
||||||
assert patch_state({"a": 1}, {"a": 2}) == {"a": 2}
|
assert patch({"a": 1}, {"a": 2}) == {"a": 2}
|
||||||
|
|
||||||
|
|
||||||
def test_apply_scalar_and_add():
|
def test_apply_scalar_and_add():
|
||||||
@@ -218,9 +218,9 @@ JSONDIFF_APPLIES_CASES = [
|
|||||||
"name,old,new", JSONDIFF_APPLIES_CASES, ids=[c[0] for c in JSONDIFF_APPLIES_CASES]
|
"name,old,new", JSONDIFF_APPLIES_CASES, ids=[c[0] for c in JSONDIFF_APPLIES_CASES]
|
||||||
)
|
)
|
||||||
def test_jsondiff_applies_our_patches(name, old, new):
|
def test_jsondiff_applies_our_patches(name, old, new):
|
||||||
diff = compute_diff(old, new)
|
delta = diff(old, new)
|
||||||
assert diff is not None
|
assert delta is not None
|
||||||
assert jsondiff.patch(old, diff, marshal=True) == new
|
assert jsondiff.patch(old, delta, marshal=True) == new
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("name,old,new", COMPAT_CASES, ids=[c[0] for c in COMPAT_CASES])
|
@pytest.mark.parametrize("name,old,new", COMPAT_CASES, ids=[c[0] for c in COMPAT_CASES])
|
||||||
@@ -231,15 +231,15 @@ def test_we_apply_jsondiff_patches(name, old, new):
|
|||||||
|
|
||||||
@pytest.mark.parametrize("name,old,new", COMPAT_CASES, ids=[c[0] for c in COMPAT_CASES])
|
@pytest.mark.parametrize("name,old,new", COMPAT_CASES, ids=[c[0] for c in COMPAT_CASES])
|
||||||
def test_our_own_round_trip(name, old, new):
|
def test_our_own_round_trip(name, old, new):
|
||||||
diff = compute_diff(old, new)
|
delta = diff(old, new)
|
||||||
assert diff is not None
|
assert delta is not None
|
||||||
assert apply_diff(old, diff) == new
|
assert apply_diff(old, delta) == new
|
||||||
|
|
||||||
|
|
||||||
def test_no_diff_means_equal_states():
|
def test_no_diff_means_equal_states():
|
||||||
for _name, old, new in COMPAT_CASES:
|
for _name, old, new in COMPAT_CASES:
|
||||||
assert compute_diff(old, new) is not None # cases really differ
|
assert diff(old, new) is not None # cases really differ
|
||||||
assert compute_diff({"a": [1, {"b": "$x"}]}, {"a": [1, {"b": "$x"}]}) is None
|
assert diff({"a": [1, {"b": "$x"}]}, {"a": [1, {"b": "$x"}]}) is None
|
||||||
|
|
||||||
|
|
||||||
# --- Logging ----------------------------------------------------------------
|
# --- Logging ----------------------------------------------------------------
|
||||||
|
|||||||
@@ -0,0 +1,805 @@
|
|||||||
|
"""Tests for structural ``--grep`` matching and match highlighting."""
|
||||||
|
|
||||||
|
from datetime import UTC, datetime
|
||||||
|
|
||||||
|
from kanta.__main__ import main
|
||||||
|
from kanta.grep import (
|
||||||
|
GrepPattern,
|
||||||
|
_Entry,
|
||||||
|
_find_spans,
|
||||||
|
_match_path,
|
||||||
|
_match_value,
|
||||||
|
evaluate,
|
||||||
|
evaluate_snapshot,
|
||||||
|
mark_spans,
|
||||||
|
matches_snapshot,
|
||||||
|
snapshot_record_lines,
|
||||||
|
)
|
||||||
|
from kanta.logging import _USER_PATH
|
||||||
|
from kanta.serialization import JsonSerializer
|
||||||
|
from kanta.serialization.framing import LineFramer
|
||||||
|
from kanta.structs import ChangeRecord, Snapshot
|
||||||
|
from kanta.tty import strip_ansi
|
||||||
|
|
||||||
|
TS = datetime(2026, 1, 1, tzinfo=UTC)
|
||||||
|
|
||||||
|
MARK = "\x1b[38;5;16;48;5;220m" # xterm256 black text on yellow background
|
||||||
|
UNMARK = "\x1b[0m" # full reset (unstyled context); styled runs restore their style
|
||||||
|
|
||||||
|
|
||||||
|
def test_match_path_whole_elements_anywhere():
|
||||||
|
assert _match_path("users", ["users"]) == frozenset({0})
|
||||||
|
assert _match_path("users", ["data", "users", "alice"]) == frozenset({1})
|
||||||
|
# Partial element matches require explicit wildcards.
|
||||||
|
assert _match_path("users", ["foousers"]) is None
|
||||||
|
assert _match_path("us*rs", ["foousers"]) is None
|
||||||
|
assert _match_path("*users", ["foousers"]) == frozenset({0})
|
||||||
|
|
||||||
|
|
||||||
|
def test_match_path_contiguous_element_sequence():
|
||||||
|
assert _match_path("alice.email", ["users", "alice", "email"]) == frozenset({1, 2})
|
||||||
|
# The sequence must be contiguous.
|
||||||
|
assert _match_path("users.email", ["users", "alice", "email"]) is None
|
||||||
|
assert _match_path("users.*.email", ["users", "alice", "email"]) == frozenset(
|
||||||
|
{0, 1, 2}
|
||||||
|
)
|
||||||
|
# Matching is case-insensitive and empty terms match without marking.
|
||||||
|
assert _match_path("USERS", ["users"]) == frozenset({0})
|
||||||
|
assert _match_path("", ["anything"]) == frozenset()
|
||||||
|
|
||||||
|
|
||||||
|
def test_match_value_substring_for_strings_full_for_scalars():
|
||||||
|
assert _match_value("lice", "alice@example.com", "alice@example.com")
|
||||||
|
assert not _match_value("bob", "alice@example.com", "alice@example.com")
|
||||||
|
# Booleans, numbers and null match only in full.
|
||||||
|
assert _match_value("true", True, "true")
|
||||||
|
assert not _match_value("tru", True, "true")
|
||||||
|
assert _match_value("42", 42, "42")
|
||||||
|
assert not _match_value("4", 42, "42")
|
||||||
|
assert _match_value("null", None, "null")
|
||||||
|
# Wildcards match the whole value text of any type.
|
||||||
|
assert _match_value("tru*", True, "true")
|
||||||
|
assert _match_value("4*", 42, "42")
|
||||||
|
# Containers do not match; their leaves are matched individually.
|
||||||
|
assert not _match_value("email", {"email": "a@b.c"}, '{"email": "a@b.c"}')
|
||||||
|
# An empty term matches anything.
|
||||||
|
assert _match_value("", True, "true")
|
||||||
|
|
||||||
|
|
||||||
|
def test_mark_spans_merges_overlaps_and_adjacents():
|
||||||
|
assert mark_spans("alice@example.com", [(0, 5), (3, 11)]) == (
|
||||||
|
f"{MARK}alice@examp{UNMARK}le.com"
|
||||||
|
)
|
||||||
|
assert mark_spans("aab", [(0, 1), (1, 3)]) == f"{MARK}aab{UNMARK}"
|
||||||
|
assert mark_spans("ab", [(1, 1), (2, 2)]) == "ab"
|
||||||
|
|
||||||
|
|
||||||
|
def test_mark_spans_offsets_into_styled_text():
|
||||||
|
styled = "\x1b[32mal\x1b[0mlice"
|
||||||
|
restore_green = "\x1b[0;32m" # the mark restores the active style exactly
|
||||||
|
# A span crossing an escape sequence wraps each escape-free run.
|
||||||
|
assert mark_spans(styled, [(0, 3)]) == (
|
||||||
|
f"\x1b[32m{MARK}al{restore_green}\x1b[0m{MARK}l{UNMARK}ice"
|
||||||
|
)
|
||||||
|
# A span covering everything wraps each escape-free run separately.
|
||||||
|
assert mark_spans(styled, [(0, 6)]) == (
|
||||||
|
f"\x1b[32m{MARK}al{restore_green}\x1b[0m{MARK}lice{UNMARK}"
|
||||||
|
)
|
||||||
|
assert mark_spans(styled, []) == styled
|
||||||
|
|
||||||
|
|
||||||
|
def test_find_spans_case_insensitive_occurrences():
|
||||||
|
assert _find_spans("Alice likes ALICE", "alice") == [(0, 5), (12, 17)]
|
||||||
|
assert _find_spans("nope", "alice") == []
|
||||||
|
assert _find_spans("anything", "") == []
|
||||||
|
|
||||||
|
|
||||||
|
def test_pattern_parse_forms():
|
||||||
|
bare = GrepPattern.parse("alice")
|
||||||
|
assert bare.term == "alice" and bare.path_term is None
|
||||||
|
|
||||||
|
pair = GrepPattern.parse("users.alice.age=30")
|
||||||
|
assert pair.term is None
|
||||||
|
assert pair.path_term == "users.alice.age"
|
||||||
|
assert pair.value_term == "30"
|
||||||
|
|
||||||
|
value_only = GrepPattern.parse("=alice@example.com")
|
||||||
|
assert value_only.path_term == "" and value_only.value_term == "alice@example.com"
|
||||||
|
|
||||||
|
path_only = GrepPattern.parse("users.alice=")
|
||||||
|
assert path_only.path_term == "users.alice" and path_only.value_term == ""
|
||||||
|
|
||||||
|
# Split on the first '=' only; the value may contain '='.
|
||||||
|
multi = GrepPattern.parse("key=a=b")
|
||||||
|
assert multi.path_term == "key" and multi.value_term == "a=b"
|
||||||
|
|
||||||
|
|
||||||
|
def _patterns(*raws: str) -> list[GrepPattern]:
|
||||||
|
return [GrepPattern.parse(raw) for raw in raws]
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_bare_term_against_path_value_action_user():
|
||||||
|
record = ChangeRecord(
|
||||||
|
ts=TS, a="create_user", u="admin", diff={"users": {"alice": {"age": 30}}}
|
||||||
|
)
|
||||||
|
assert evaluate(record, {}, _patterns("users.alice"))
|
||||||
|
assert evaluate(record, {}, _patterns("30"))
|
||||||
|
assert evaluate(record, {}, _patterns("CREATE_user"))
|
||||||
|
assert evaluate(record, {}, _patterns("admin"))
|
||||||
|
assert evaluate(record, {}, _patterns("age=30"))
|
||||||
|
assert not evaluate(record, {}, _patterns("bob"))
|
||||||
|
assert not evaluate(record, {}, _patterns("3")) # not a full number match
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_unescapes_dollar_keys():
|
||||||
|
record = ChangeRecord(ts=TS, a="set", diff={"$$config": 5})
|
||||||
|
assert evaluate(record, {}, _patterns("$config=5"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_path_value_form_requires_same_line():
|
||||||
|
record = ChangeRecord(ts=TS, a="set", diff={"a": {"x": 1}, "b": {"y": 2}})
|
||||||
|
previous = {"a": {"x": 0}, "b": {"y": 0}}
|
||||||
|
assert evaluate(record, previous, _patterns("a.x=1"))
|
||||||
|
# 'a' matches one line's path, '2' another line's value: no match.
|
||||||
|
assert not evaluate(record, previous, _patterns("a=2"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_all_patterns_same_record_any_line():
|
||||||
|
record = ChangeRecord(ts=TS, a="set", diff={"a": {"x": 1}, "b": {"y": 2}})
|
||||||
|
previous = {"a": {"x": 0}, "b": {"y": 0}}
|
||||||
|
assert evaluate(record, previous, _patterns("a.x", "=2"))
|
||||||
|
assert not evaluate(record, previous, _patterns("a.x", "=2", "missing"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_matches_deleted_content_by_previous_value():
|
||||||
|
record = ChangeRecord(ts=TS, a="delete_user", diff={"users": {"$delete": "alice"}})
|
||||||
|
previous = {"users": {"alice": {"email": "alice@example.com"}}}
|
||||||
|
assert evaluate(record, previous, _patterns("alice@example.com"))
|
||||||
|
assert evaluate(record, previous, _patterns("users.alice.email"))
|
||||||
|
assert not evaluate(record, {}, _patterns("alice@example.com"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_highlighter_marks_exactly_the_matched_regions():
|
||||||
|
record = ChangeRecord(
|
||||||
|
ts=TS,
|
||||||
|
a="set",
|
||||||
|
diff={
|
||||||
|
"users": {"alice": {"email": "same@x.com"}, "bob": {"email": "same@x.com"}}
|
||||||
|
},
|
||||||
|
)
|
||||||
|
previous = {
|
||||||
|
"users": {"alice": {"email": "old@x.com"}, "bob": {"email": "same@x.com"}}
|
||||||
|
}
|
||||||
|
hl = evaluate(record, previous, _patterns("users.alice.email"))
|
||||||
|
assert hl is not None
|
||||||
|
# Path-side match: the matched elements are lit, the value is not.
|
||||||
|
assert hl.path("alice", "users.alice") == f"{MARK}alice{UNMARK}"
|
||||||
|
assert hl.value("same@x.com", "users.alice.email") == "same@x.com"
|
||||||
|
# Bob's identical value is a different path: untouched.
|
||||||
|
assert hl.path("bob", "users.bob") == "bob"
|
||||||
|
assert hl.value("same@x.com", "users.bob.email") == "same@x.com"
|
||||||
|
|
||||||
|
hl = evaluate(record, previous, _patterns("=same@x.com"))
|
||||||
|
assert hl is not None
|
||||||
|
# Both lines genuinely match the value: both are marked.
|
||||||
|
assert hl.value("same@x.com", "users.alice.email") == f"{MARK}same@x.com{UNMARK}"
|
||||||
|
assert hl.value("same@x.com", "users.bob.email") == f"{MARK}same@x.com{UNMARK}"
|
||||||
|
assert hl.path("alice", "users.alice") == "alice"
|
||||||
|
|
||||||
|
|
||||||
|
def test_highlighter_merges_overlapping_needles_from_different_patterns():
|
||||||
|
record = ChangeRecord(ts=TS, a="set", diff={"email": "alice@example.com"})
|
||||||
|
hl = evaluate(record, {}, _patterns("alice", "lice@exam"))
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.value("alice@example.com", "email") == (
|
||||||
|
f"{MARK}alice@exam{UNMARK}ple.com"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_highlighter_marks_delete_marker_on_removed_content_match():
|
||||||
|
record = ChangeRecord(ts=TS, a="delete_user", diff={"users": {"$delete": "bob"}})
|
||||||
|
previous = {"users": {"bob": {"email": "bob@example.com"}}}
|
||||||
|
hl = evaluate(record, previous, _patterns("bob@example.com"))
|
||||||
|
assert hl is not None
|
||||||
|
# The matched content is gone: the deletion marker is lit, not the path.
|
||||||
|
assert hl.path("users", "users") == "users"
|
||||||
|
assert hl.path("bob", "users.bob") == "bob"
|
||||||
|
assert hl.delete("✗", "users.bob") == f"{MARK}✗{UNMARK}"
|
||||||
|
assert hl.delete("✗", "users.alice") == "✗"
|
||||||
|
|
||||||
|
# A path-side match lights the genuinely matched elements of the anchor,
|
||||||
|
# and the marker for the element below it.
|
||||||
|
hl = evaluate(record, previous, _patterns("users.bob.email"))
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.path("users", "users") == f"{MARK}users{UNMARK}"
|
||||||
|
assert hl.path("bob", "users.bob") == f"{MARK}bob{UNMARK}"
|
||||||
|
assert hl.delete("✗", "users.bob") == f"{MARK}✗{UNMARK}"
|
||||||
|
|
||||||
|
# A path-side match of one displayed element lights only that element.
|
||||||
|
hl = evaluate(record, previous, _patterns("bob"))
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.path("users", "users") == "users"
|
||||||
|
assert hl.path("bob", "users.bob") == f"{MARK}bob{UNMARK}"
|
||||||
|
|
||||||
|
|
||||||
|
def test_highlighter_meta_marks_action_and_user():
|
||||||
|
record = ChangeRecord(ts=TS, a="create_user", u="admin", diff={"x": 1})
|
||||||
|
hl = evaluate(record, {}, _patterns("create", "ADM"))
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.meta("create_user", "action") == f"{MARK}create{UNMARK}_user"
|
||||||
|
assert hl.meta("admin", "user") == f"{MARK}adm{UNMARK}in"
|
||||||
|
assert hl.meta("extra", "other") == "extra"
|
||||||
|
|
||||||
|
|
||||||
|
def test_matches_snapshot_uses_change_record_matching():
|
||||||
|
state = {"users": {"alice": {"email": "alice@example.com", "age": 30}}}
|
||||||
|
assert matches_snapshot(state, _patterns("users.alice"))
|
||||||
|
assert matches_snapshot(state, _patterns("alice@example"))
|
||||||
|
assert matches_snapshot(state, _patterns("age=30"))
|
||||||
|
assert matches_snapshot(state, _patterns("users.alice", "=30"))
|
||||||
|
assert not matches_snapshot(state, _patterns("bob"))
|
||||||
|
assert not matches_snapshot(state, _patterns("3")) # not a full number match
|
||||||
|
assert not matches_snapshot(state, _patterns("alice", "missing"))
|
||||||
|
assert not matches_snapshot({}, _patterns("alice"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_matches_snapshot_supports_logfmt_prettified_values():
|
||||||
|
state = {"when": 1767225600}
|
||||||
|
|
||||||
|
def logfmt(value, path):
|
||||||
|
return "2026-01-01" if value == 1767225600 else None
|
||||||
|
|
||||||
|
assert matches_snapshot(state, _patterns("2026"), logfmt=logfmt)
|
||||||
|
assert matches_snapshot(state, _patterns("1767225600"), logfmt=logfmt)
|
||||||
|
assert not matches_snapshot(state, _patterns("2026"))
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_record_lines_render_matched_records():
|
||||||
|
"""Matched entries render as one compact record line, change-log style."""
|
||||||
|
state = {
|
||||||
|
"users": {
|
||||||
|
"alice": {"email": "alice@example.com", "age": 30},
|
||||||
|
"bob": {"email": "bob@example.com"},
|
||||||
|
},
|
||||||
|
"counter": 1,
|
||||||
|
}
|
||||||
|
highlighter = evaluate_snapshot(state, _patterns("=alice@example.com"))
|
||||||
|
assert highlighter is not None
|
||||||
|
lines = snapshot_record_lines(state, highlighter)
|
||||||
|
# The whole alice record is shown; bob and counter are not.
|
||||||
|
assert [strip_ansi(line) for line in lines] == [
|
||||||
|
" users.alice = {email: alice@example.com, age: 30}"
|
||||||
|
]
|
||||||
|
assert MARK + "alice@example.com" + UNMARK in lines[0]
|
||||||
|
|
||||||
|
|
||||||
|
def test_snapshot_record_lines_pretty_path_and_value():
|
||||||
|
"""A logfmt-prettified path element matches and is marked, like values."""
|
||||||
|
state = {"users": {"userid002": {"name": "Bob", "role": "editor"}}}
|
||||||
|
|
||||||
|
def logfmt(value, path):
|
||||||
|
return "Bob" if value == "userid002" else None
|
||||||
|
|
||||||
|
highlighter = evaluate_snapshot(state, _patterns("bob"), logfmt=logfmt)
|
||||||
|
assert highlighter is not None
|
||||||
|
lines = snapshot_record_lines(state, highlighter, logfmt)
|
||||||
|
assert len(lines) == 1
|
||||||
|
assert strip_ansi(lines[0]) == " users.Bob = {name: Bob, role: editor}"
|
||||||
|
# Both displayed Bobs — the prettified path element and the name value.
|
||||||
|
assert lines[0].count(MARK) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_entry_dataclass_holds_anchor_for_deletes():
|
||||||
|
entry = _Entry(["a", "b"], 1, "1", anchor=["a"])
|
||||||
|
assert entry.anchor == ["a"]
|
||||||
|
|
||||||
|
|
||||||
|
def _write_db(path, changes, state=None):
|
||||||
|
serializer = JsonSerializer()
|
||||||
|
framer = LineFramer()
|
||||||
|
snapshot = Snapshot(ts=TS, v=1, state=state or {})
|
||||||
|
data = framer.frame_snapshot(serializer.encode(snapshot), record_offset=0)
|
||||||
|
for change in changes:
|
||||||
|
data += framer.frame_change(serializer.encode(change), record_offset=0)
|
||||||
|
path.write_bytes(data)
|
||||||
|
|
||||||
|
|
||||||
|
def _sample_changes():
|
||||||
|
return [
|
||||||
|
ChangeRecord(
|
||||||
|
ts=TS,
|
||||||
|
a="create_alice",
|
||||||
|
u="admin",
|
||||||
|
diff={
|
||||||
|
"users": {
|
||||||
|
"alice": {"email": "alice@example.com", "admin": True},
|
||||||
|
}
|
||||||
|
},
|
||||||
|
),
|
||||||
|
ChangeRecord(
|
||||||
|
ts=TS,
|
||||||
|
a="update_alice",
|
||||||
|
u="admin",
|
||||||
|
diff={"users": {"alice": {"age": 30}}},
|
||||||
|
),
|
||||||
|
ChangeRecord(
|
||||||
|
ts=TS,
|
||||||
|
a="create_bob",
|
||||||
|
u="bob",
|
||||||
|
diff={"users": {"bob": {"email": "bob@example.com"}}},
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _run_cli(tmp_path, capsys, monkeypatch, changes, *args, color=False, state=None):
|
||||||
|
if color:
|
||||||
|
monkeypatch.setenv("FORCE_COLOR", "1")
|
||||||
|
monkeypatch.delenv("NO_COLOR", raising=False)
|
||||||
|
else:
|
||||||
|
monkeypatch.setenv("NO_COLOR", "1")
|
||||||
|
monkeypatch.delenv("FORCE_COLOR", raising=False)
|
||||||
|
path = tmp_path / "test.kantadb"
|
||||||
|
_write_db(path, changes, state=state)
|
||||||
|
code = main([str(path), *args])
|
||||||
|
assert code == 0
|
||||||
|
return capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_filters_records(tmp_path, capsys, monkeypatch):
|
||||||
|
"""Only matching change records are printed."""
|
||||||
|
err = _run_cli(tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "alice")
|
||||||
|
assert "create_alice" in err
|
||||||
|
assert "update_alice" in err
|
||||||
|
assert "create_bob" not in err
|
||||||
|
assert "bob@example.com" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_prints_entire_transaction(tmp_path, capsys, monkeypatch):
|
||||||
|
"""A match prints the whole record, not just the matching line."""
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "users.alice.email"
|
||||||
|
)
|
||||||
|
assert "create_alice" in err
|
||||||
|
# Non-matching lines of the same record are printed too.
|
||||||
|
assert "admin" in err and "true" in err
|
||||||
|
# The update record does not contain the path.
|
||||||
|
assert "update_alice" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_repeated_patterns_must_all_match(tmp_path, capsys, monkeypatch):
|
||||||
|
"""Repeated --grep options are ANDed within the same record."""
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"alice",
|
||||||
|
"--grep",
|
||||||
|
"=alice@example.com",
|
||||||
|
)
|
||||||
|
assert "create_alice" in err
|
||||||
|
# update_alice matches 'alice' but has no email value.
|
||||||
|
assert "update_alice" not in err
|
||||||
|
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"alice",
|
||||||
|
"--grep",
|
||||||
|
"bob",
|
||||||
|
)
|
||||||
|
assert "create_alice" not in err
|
||||||
|
assert "create_bob" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_no_match_exits_zero(tmp_path, capsys, monkeypatch):
|
||||||
|
"""No matching records is not an error; non-matching snapshots are hidden."""
|
||||||
|
err = _run_cli(tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "nobody")
|
||||||
|
assert "snapshot s0" not in err
|
||||||
|
assert "create_alice" not in err
|
||||||
|
assert "create_bob" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_snapshot_prints_only_when_state_matches(
|
||||||
|
tmp_path, capsys, monkeypatch
|
||||||
|
):
|
||||||
|
"""Snapshots are matched against their full state like change records."""
|
||||||
|
state = {"users": {"alice": {"email": "alice@example.com", "age": 30}}}
|
||||||
|
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"alice",
|
||||||
|
state=state,
|
||||||
|
)
|
||||||
|
assert "snapshot s0" in err
|
||||||
|
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"users.alice.age=30",
|
||||||
|
state=state,
|
||||||
|
)
|
||||||
|
assert "snapshot s0" in err
|
||||||
|
|
||||||
|
# A pattern matching nothing in the state suppresses the snapshot.
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"bob",
|
||||||
|
state=state,
|
||||||
|
)
|
||||||
|
assert "snapshot s0" not in err
|
||||||
|
|
||||||
|
# Repeated patterns are ANDed within the snapshot state too.
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"alice",
|
||||||
|
"--grep",
|
||||||
|
"missing",
|
||||||
|
state=state,
|
||||||
|
)
|
||||||
|
assert "snapshot s0" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_snapshot_prints_matched_state(tmp_path, capsys, monkeypatch):
|
||||||
|
"""A matched snapshot shows the matched part of its state, diff-style."""
|
||||||
|
state = {
|
||||||
|
"users": {
|
||||||
|
"alice": {"email": "alice@example.com", "age": 30},
|
||||||
|
"bob": {"email": "bob@example.com"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"=alice@example.com",
|
||||||
|
state=state,
|
||||||
|
)
|
||||||
|
assert "snapshot s0" in err
|
||||||
|
# The matched state line is printed under the snapshot indicator;
|
||||||
|
# unmatched parts of the state (bob) are not shown.
|
||||||
|
assert "alice@example.com" in err
|
||||||
|
assert "bob@example.com" not in err
|
||||||
|
|
||||||
|
# The matched region carries the yellow highlight when color is on.
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"--grep",
|
||||||
|
"=alice@example.com",
|
||||||
|
state=state,
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
assert f"{MARK}alice@example.com{UNMARK}" in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_snapshot_selection_shortcut(tmp_path, capsys, monkeypatch):
|
||||||
|
"""-r sN applies --grep like the replay path: matched state is shown."""
|
||||||
|
state = {
|
||||||
|
"users": {
|
||||||
|
"alice": {"email": "alice@example.com", "age": 30},
|
||||||
|
"bob": {"email": "bob@example.com"},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"-r",
|
||||||
|
"s0",
|
||||||
|
"--grep",
|
||||||
|
"=alice@example.com",
|
||||||
|
state=state,
|
||||||
|
)
|
||||||
|
assert "snapshot s0" in err
|
||||||
|
assert "users.alice" in err
|
||||||
|
assert "alice@example.com" in err
|
||||||
|
assert "bob@example.com" not in err
|
||||||
|
|
||||||
|
# A non-matching pattern suppresses the explicitly selected snapshot.
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_sample_changes(),
|
||||||
|
"-r",
|
||||||
|
"s0",
|
||||||
|
"--grep",
|
||||||
|
"zzz",
|
||||||
|
state=state,
|
||||||
|
)
|
||||||
|
assert "snapshot s0" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_path_value_forms(tmp_path, capsys, monkeypatch):
|
||||||
|
"""The 'path=value', 'path=' and '=value' forms restrict the match side."""
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "users.*.admin=true"
|
||||||
|
)
|
||||||
|
assert "create_alice" in err
|
||||||
|
assert "create_bob" not in err
|
||||||
|
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "users.alice.age="
|
||||||
|
)
|
||||||
|
assert "update_alice" in err
|
||||||
|
assert "create_alice" not in err
|
||||||
|
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path, capsys, monkeypatch, _sample_changes(), "--grep", "=bob@example.com"
|
||||||
|
)
|
||||||
|
assert "create_bob" in err
|
||||||
|
assert "create_alice" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_path_elements_match_in_full(tmp_path, capsys, monkeypatch):
|
||||||
|
"""'users' does not match a 'foousers' element; 'us*' does."""
|
||||||
|
changes = [ChangeRecord(ts=TS, a="trap", diff={"foousers": {"note": "x"}})]
|
||||||
|
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "users")
|
||||||
|
assert "trap" not in err
|
||||||
|
|
||||||
|
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "*users")
|
||||||
|
assert "trap" in err
|
||||||
|
|
||||||
|
|
||||||
|
def _shared_email_changes():
|
||||||
|
return [
|
||||||
|
ChangeRecord(
|
||||||
|
ts=TS,
|
||||||
|
a="create_alice",
|
||||||
|
u="admin",
|
||||||
|
diff={"users": {"alice": {"email": "shared@example.com", "admin": True}}},
|
||||||
|
),
|
||||||
|
ChangeRecord(
|
||||||
|
ts=TS,
|
||||||
|
a="create_bob",
|
||||||
|
u="admin",
|
||||||
|
diff={"users": {"bob": {"email": "shared@example.com"}}},
|
||||||
|
),
|
||||||
|
ChangeRecord(
|
||||||
|
ts=TS,
|
||||||
|
a="delete_bob",
|
||||||
|
u="admin",
|
||||||
|
diff={"users": {"$delete": "bob"}},
|
||||||
|
),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _lines_with(err: str, text: str) -> list[str]:
|
||||||
|
return [line for line in err.splitlines() if text in strip_ansi(line)]
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_highlight_marks_exactly_the_matched_regions(tmp_path, capsys, monkeypatch):
|
||||||
|
"""A path-side match lights only the matched elements, not the value."""
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_shared_email_changes(),
|
||||||
|
"--grep",
|
||||||
|
"users.alice.email",
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
(alice_line,) = _lines_with(err, "shared@example.com")
|
||||||
|
assert f"{MARK}alice{UNMARK}" in alice_line
|
||||||
|
assert f"{MARK}email{UNMARK}" in alice_line
|
||||||
|
# The value itself did not match, so it is not highlighted.
|
||||||
|
assert f"{MARK}shared@example.com{UNMARK}" not in alice_line
|
||||||
|
(header_line,) = _lines_with(err, "users =")
|
||||||
|
assert f"{MARK}users{UNMARK}" in header_line
|
||||||
|
assert "create_bob" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_highlight_value_matches_on_all_matching_lines(
|
||||||
|
tmp_path, capsys, monkeypatch
|
||||||
|
):
|
||||||
|
"""A value-side match lights the value on every line that genuinely matched."""
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_shared_email_changes(),
|
||||||
|
"--grep",
|
||||||
|
"=shared@example.com",
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
lines = _lines_with(err, "shared@example.com")
|
||||||
|
assert len(lines) == 2 # alice's and bob's records both matched
|
||||||
|
for line in lines:
|
||||||
|
assert f"{MARK}shared@example.com{UNMARK}" in line
|
||||||
|
# The deletion of bob matched by its previous (removed) value: the
|
||||||
|
# deletion marker is lit, not the deleted path.
|
||||||
|
(delete_line,) = _lines_with(err, "✗")
|
||||||
|
assert f"{MARK}✗{UNMARK}" in delete_line
|
||||||
|
assert f"{MARK}bob{UNMARK}" not in delete_line
|
||||||
|
assert f"{MARK}users{UNMARK}" not in delete_line
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_highlight_merges_overlapping_matches(tmp_path, capsys, monkeypatch):
|
||||||
|
"""Overlapping matches from different patterns form one continuous mark."""
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_shared_email_changes(),
|
||||||
|
"--grep",
|
||||||
|
"shared",
|
||||||
|
"--grep",
|
||||||
|
"red@exam",
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
lines = _lines_with(err, "shared@example.com")
|
||||||
|
assert len(lines) == 2 # both records genuinely match both patterns
|
||||||
|
for line in lines:
|
||||||
|
assert f"{MARK}shared@exam{UNMARK}ple.com" in line
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_highlight_full_scalar_and_header_fields(tmp_path, capsys, monkeypatch):
|
||||||
|
"""Scalars are marked in full; matched action/user substrings are marked."""
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
_shared_email_changes(),
|
||||||
|
"--grep",
|
||||||
|
"admin=true",
|
||||||
|
"--grep",
|
||||||
|
"create_al",
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
(line,) = _lines_with(err, "shared@example.com")
|
||||||
|
assert f"{MARK}true{UNMARK}" in line
|
||||||
|
(header,) = _lines_with(err, "create_alice")
|
||||||
|
assert f"{MARK}create_al{UNMARK}ice" in header
|
||||||
|
# Only the record matching both patterns is printed.
|
||||||
|
assert "create_bob" not in err
|
||||||
|
assert "delete_bob" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_matches_prettified_and_raw_value_forms():
|
||||||
|
"""With logfmt, both the raw and the prettified form of a value match."""
|
||||||
|
record = ChangeRecord(ts=TS, a="set", diff={"when": 1767225600})
|
||||||
|
|
||||||
|
def logfmt(value, path):
|
||||||
|
return "2026-01-01" if value == 1767225600 else None
|
||||||
|
|
||||||
|
# The prettified form matches; the needle is located in the display text.
|
||||||
|
hl = evaluate(record, {}, _patterns("2026"), logfmt=logfmt)
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.value("2026-01-01", "when") == f"{MARK}2026{UNMARK}-01-01"
|
||||||
|
# The raw form matches too; its needle is absent from the displayed
|
||||||
|
# text, so the whole displayed value is marked.
|
||||||
|
hl = evaluate(record, {}, _patterns("1767225600"), logfmt=logfmt)
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.value("2026-01-01", "when") == f"{MARK}2026-01-01{UNMARK}"
|
||||||
|
# Without logfmt the raw value is matched and marked precisely.
|
||||||
|
hl = evaluate(record, {}, _patterns("1767225600"))
|
||||||
|
assert hl.value("1767225600", "when") == f"{MARK}1767225600{UNMARK}"
|
||||||
|
# Neither form matches.
|
||||||
|
assert evaluate(record, {}, _patterns("1999"), logfmt=logfmt) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_matches_prettified_user():
|
||||||
|
"""The user field matches both the raw id and the logfmt-resolved name."""
|
||||||
|
|
||||||
|
def logfmt(value, path):
|
||||||
|
return "Alice Admin" if path == _USER_PATH else None
|
||||||
|
|
||||||
|
record = ChangeRecord(ts=TS, a="set", u="u123", diff={"x": 1})
|
||||||
|
hl = evaluate(record, {}, _patterns("alice"), logfmt=logfmt)
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.meta("Alice Admin", "user") == f"{MARK}Alice{UNMARK} Admin"
|
||||||
|
# A raw-only user match marks the whole displayed name.
|
||||||
|
hl = evaluate(record, {}, _patterns("u123"), logfmt=logfmt)
|
||||||
|
assert hl is not None
|
||||||
|
assert hl.meta("Alice Admin", "user") == f"{MARK}Alice Admin{UNMARK}"
|
||||||
|
# The action is never prettified.
|
||||||
|
assert hl.meta("set", "action") == "set"
|
||||||
|
|
||||||
|
|
||||||
|
def test_evaluate_without_logfmt_matches_raw_only():
|
||||||
|
record = ChangeRecord(ts=TS, a="set", u="u123", diff={"when": 1767225600})
|
||||||
|
assert evaluate(record, {}, _patterns("1767225600"))
|
||||||
|
assert evaluate(record, {}, _patterns("2026")) is None
|
||||||
|
assert evaluate(record, {}, _patterns("alice")) is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_grep_matches_prettified_forms_with_kanta_object(
|
||||||
|
tmp_path, capsys, monkeypatch
|
||||||
|
):
|
||||||
|
"""End to end: a -k object's logfmt formatter doubles the match surface."""
|
||||||
|
db_path = tmp_path / "test.kantadb"
|
||||||
|
module = tmp_path / "dbmod.py"
|
||||||
|
module.write_text(
|
||||||
|
"from typing import Any\n"
|
||||||
|
"from kanta import Kanta\n"
|
||||||
|
f"kanta = Kanta({str(db_path)!r}, {{}}, type=dict)\n"
|
||||||
|
"@kanta.logfmt\n"
|
||||||
|
"def pretty(value: Any, path: str) -> str | None:\n"
|
||||||
|
" if path == 'when':\n"
|
||||||
|
" return 'Nov 3, 2025'\n"
|
||||||
|
" if path == '$user':\n"
|
||||||
|
" return 'Alice Admin'\n"
|
||||||
|
" return None\n"
|
||||||
|
)
|
||||||
|
changes = [ChangeRecord(ts=TS, a="set", u="u123", diff={"when": 1767225600})]
|
||||||
|
kanta_args = ("-k", str(module))
|
||||||
|
|
||||||
|
# A term matching only the prettified value finds the record.
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path, capsys, monkeypatch, changes, *kanta_args, "--grep", "nov", color=True
|
||||||
|
)
|
||||||
|
(line,) = _lines_with(err, "Nov 3, 2025")
|
||||||
|
assert f"{MARK}Nov{UNMARK} 3, 2025" in line
|
||||||
|
|
||||||
|
# A term matching only the raw value marks the whole prettified display.
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
changes,
|
||||||
|
*kanta_args,
|
||||||
|
"--grep",
|
||||||
|
"1767225600",
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
(line,) = _lines_with(err, "Nov 3, 2025")
|
||||||
|
assert f"{MARK}Nov 3, 2025{UNMARK}" in line
|
||||||
|
|
||||||
|
# The prettified user matches, and the raw id marks the whole display.
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
changes,
|
||||||
|
*kanta_args,
|
||||||
|
"--grep",
|
||||||
|
"alice",
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
(header,) = _lines_with(err, "Alice Admin")
|
||||||
|
assert f"{MARK}Alice{UNMARK} Admin" in header
|
||||||
|
err = _run_cli(
|
||||||
|
tmp_path,
|
||||||
|
capsys,
|
||||||
|
monkeypatch,
|
||||||
|
changes,
|
||||||
|
*kanta_args,
|
||||||
|
"--grep",
|
||||||
|
"u123",
|
||||||
|
color=True,
|
||||||
|
)
|
||||||
|
(header,) = _lines_with(err, "Alice Admin")
|
||||||
|
assert f"{MARK}Alice Admin{UNMARK}" in header
|
||||||
|
|
||||||
|
# Without -k there is no prettified form to match.
|
||||||
|
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "nov")
|
||||||
|
assert not _lines_with(err, "u123")
|
||||||
|
err = _run_cli(tmp_path, capsys, monkeypatch, changes, "--grep", "1767225600")
|
||||||
|
assert _lines_with(err, "u123")
|
||||||
@@ -769,7 +769,11 @@ async def test_transaction_log_false_suppresses_log(tmp_path, format_config, cap
|
|||||||
|
|
||||||
await kanta.close()
|
await kanta.close()
|
||||||
|
|
||||||
info_messages = [r for r in caplog.records if r.levelno == logging.INFO]
|
info_messages = [
|
||||||
|
r
|
||||||
|
for r in caplog.records
|
||||||
|
if r.levelno == logging.INFO and r.name.startswith("kanta.transaction")
|
||||||
|
]
|
||||||
assert not info_messages
|
assert not info_messages
|
||||||
|
|
||||||
|
|
||||||
@@ -787,7 +791,11 @@ async def test_transaction_logdiff_false_logs_header_only(
|
|||||||
|
|
||||||
await kanta.close()
|
await kanta.close()
|
||||||
|
|
||||||
messages = [r.message for r in caplog.records if r.levelno == logging.INFO]
|
messages = [
|
||||||
|
r.message
|
||||||
|
for r in caplog.records
|
||||||
|
if r.levelno == logging.INFO and r.name.startswith("kanta.transaction")
|
||||||
|
]
|
||||||
assert len(messages) == 1
|
assert len(messages) == 1
|
||||||
assert "inc" in messages[0]
|
assert "inc" in messages[0]
|
||||||
assert "counter" not in messages[0]
|
assert "counter" not in messages[0]
|
||||||
@@ -808,7 +816,11 @@ async def test_transaction_log_custom_logger(tmp_path, format_config, caplog):
|
|||||||
|
|
||||||
await kanta.close()
|
await kanta.close()
|
||||||
|
|
||||||
info_messages = [r for r in caplog.records if r.levelno == logging.INFO]
|
info_messages = [
|
||||||
|
r
|
||||||
|
for r in caplog.records
|
||||||
|
if r.levelno == logging.INFO and r.name.startswith("custom.transaction")
|
||||||
|
]
|
||||||
assert len(info_messages) >= 1
|
assert len(info_messages) >= 1
|
||||||
assert "inc" in info_messages[0].message
|
assert "inc" in info_messages[0].message
|
||||||
|
|
||||||
|
|||||||
+39
-13
@@ -38,13 +38,23 @@ def _reset_kanta_loggers():
|
|||||||
logger.handlers.clear()
|
logger.handlers.clear()
|
||||||
|
|
||||||
|
|
||||||
|
def _setup_logging(**kwargs):
|
||||||
|
"""Default kanta logging with the event loggers lifted to INFO.
|
||||||
|
|
||||||
|
Event loggers inherit the root level (WARNING under pytest); output
|
||||||
|
assertions need INFO.
|
||||||
|
"""
|
||||||
|
configure_logging(**kwargs)
|
||||||
|
for name in ("kanta.bootstrap", "kanta.migration", "kanta.transaction"):
|
||||||
|
logging.getLogger(name).setLevel(logging.INFO)
|
||||||
|
|
||||||
|
|
||||||
def _change_event(**kwargs) -> LogEvent:
|
def _change_event(**kwargs) -> LogEvent:
|
||||||
return LogEvent(kind="change", logger=transaction_logger, action="update", **kwargs)
|
return LogEvent(kind="change", logger=transaction_logger, action="update", **kwargs)
|
||||||
|
|
||||||
|
|
||||||
def test_emit_event_falsy_return_stops_chain(capsys):
|
def test_emit_event_falsy_return_stops_chain(capsys):
|
||||||
logging.getLogger("kanta").handlers.clear()
|
_setup_logging()
|
||||||
configure_logging()
|
|
||||||
calls = []
|
calls = []
|
||||||
|
|
||||||
def first(ev):
|
def first(ev):
|
||||||
@@ -60,15 +70,13 @@ def test_emit_event_falsy_return_stops_chain(capsys):
|
|||||||
|
|
||||||
|
|
||||||
def test_emit_event_truthy_return_falls_back_to_default(capsys):
|
def test_emit_event_truthy_return_falls_back_to_default(capsys):
|
||||||
logging.getLogger("kanta").handlers.clear()
|
_setup_logging()
|
||||||
configure_logging()
|
|
||||||
emit_event(_change_event(), [lambda ev: True])
|
emit_event(_change_event(), [lambda ev: True])
|
||||||
assert "update" in capsys.readouterr().err
|
assert "update" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
def test_emit_event_mutation_reaches_later_handlers_and_default(capsys):
|
def test_emit_event_mutation_reaches_later_handlers_and_default(capsys):
|
||||||
logging.getLogger("kanta").handlers.clear()
|
_setup_logging()
|
||||||
configure_logging()
|
|
||||||
calls = []
|
calls = []
|
||||||
|
|
||||||
def first(ev):
|
def first(ev):
|
||||||
@@ -86,8 +94,7 @@ def test_emit_event_mutation_reaches_later_handlers_and_default(capsys):
|
|||||||
|
|
||||||
|
|
||||||
def test_emit_event_handler_error_falls_back_to_default(capsys):
|
def test_emit_event_handler_error_falls_back_to_default(capsys):
|
||||||
logging.getLogger("kanta").handlers.clear()
|
_setup_logging()
|
||||||
configure_logging()
|
|
||||||
|
|
||||||
def boom(ev):
|
def boom(ev):
|
||||||
raise RuntimeError("broken")
|
raise RuntimeError("broken")
|
||||||
@@ -109,8 +116,7 @@ def test_diff_lines_built_lazily(monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_default_emit_created_and_migrated(capsys):
|
def test_default_emit_created_and_migrated(capsys):
|
||||||
logging.getLogger("kanta").handlers.clear()
|
_setup_logging()
|
||||||
configure_logging()
|
|
||||||
emit_event(LogEvent(kind="created", logger=bootstrap_logger, filename="x.kantadb"))
|
emit_event(LogEvent(kind="created", logger=bootstrap_logger, filename="x.kantadb"))
|
||||||
emit_event(
|
emit_event(
|
||||||
LogEvent(
|
LogEvent(
|
||||||
@@ -127,6 +133,23 @@ def test_default_emit_created_and_migrated(capsys):
|
|||||||
assert "🛢️ x.kantadb migrated v0 -> v1: migrate_v1 (rename)" in err
|
assert "🛢️ x.kantadb migrated v0 -> v1: migrate_v1 (rename)" in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_emit_strips_ansi_without_color_support(capsys, monkeypatch):
|
||||||
|
"""NO_COLOR output contains no ANSI codes; FORCE_COLOR keeps them."""
|
||||||
|
_setup_logging()
|
||||||
|
monkeypatch.setenv("NO_COLOR", "1")
|
||||||
|
monkeypatch.delenv("FORCE_COLOR", raising=False)
|
||||||
|
emit_event(_change_event(diff={"counter": 1}))
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "\x1b[" not in err
|
||||||
|
assert "counter" in err
|
||||||
|
|
||||||
|
_setup_logging()
|
||||||
|
monkeypatch.setenv("FORCE_COLOR", "1")
|
||||||
|
monkeypatch.delenv("NO_COLOR", raising=False)
|
||||||
|
emit_event(_change_event(diff={"counter": 1}))
|
||||||
|
assert "\x1b[" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_logemit_receives_transaction_events(tmp_path, format_config):
|
async def test_logemit_receives_transaction_events(tmp_path, format_config):
|
||||||
path = tmp_path / "test.db"
|
path = tmp_path / "test.db"
|
||||||
@@ -242,7 +265,11 @@ async def test_logmigr_failure_does_not_break_open(tmp_path, format_config):
|
|||||||
|
|
||||||
|
|
||||||
@pytest.mark.asyncio
|
@pytest.mark.asyncio
|
||||||
async def test_aborted_transaction_emits_event(tmp_path, format_config, caplog):
|
async def test_aborted_transaction_emits_event(
|
||||||
|
tmp_path, format_config, caplog, monkeypatch
|
||||||
|
):
|
||||||
|
monkeypatch.setenv("FORCE_COLOR", "1")
|
||||||
|
monkeypatch.delenv("NO_COLOR", raising=False)
|
||||||
path = tmp_path / "test.db"
|
path = tmp_path / "test.db"
|
||||||
kanta = make_kanta(path, Data, format_config)
|
kanta = make_kanta(path, Data, format_config)
|
||||||
events = []
|
events = []
|
||||||
@@ -334,8 +361,7 @@ async def test_event_carries_kanta_instance(tmp_path, format_config):
|
|||||||
|
|
||||||
|
|
||||||
def test_header_is_settable_and_used_by_default_emit(capsys):
|
def test_header_is_settable_and_used_by_default_emit(capsys):
|
||||||
logging.getLogger("kanta").handlers.clear()
|
_setup_logging()
|
||||||
configure_logging()
|
|
||||||
|
|
||||||
def restyle(ev):
|
def restyle(ev):
|
||||||
ev.header = f"CUSTOM {ev.action}"
|
ev.header = f"CUSTOM {ev.action}"
|
||||||
|
|||||||
+34
-30
@@ -39,42 +39,52 @@ def _reset_kanta_loggers():
|
|||||||
|
|
||||||
|
|
||||||
def test_configure_logging_defaults():
|
def test_configure_logging_defaults():
|
||||||
kanta_logger = logging.getLogger("kanta")
|
|
||||||
configure_logging()
|
configure_logging()
|
||||||
assert kanta_logger.level == logging.INFO
|
for name in ("kanta.bootstrap", "kanta.migration", "kanta.transaction"):
|
||||||
assert not kanta_logger.propagate
|
logger = logging.getLogger(name)
|
||||||
assert kanta_logger.handlers
|
assert logger.level == logging.INFO # visible without app logging setup
|
||||||
|
assert not logger.propagate
|
||||||
|
assert logger.handlers
|
||||||
|
|
||||||
|
|
||||||
|
def test_configure_logging_debug_lifts_debug_loggers():
|
||||||
|
configure_logging(debug=True)
|
||||||
|
assert logging.getLogger("kanta.bootstrap").level == logging.DEBUG
|
||||||
|
assert logging.getLogger("kanta.migration").level == logging.DEBUG
|
||||||
|
assert logging.getLogger("kanta.transaction").level == logging.INFO
|
||||||
|
|
||||||
|
|
||||||
def test_configure_logging_disables_specific_loggers():
|
def test_configure_logging_disables_specific_loggers():
|
||||||
configure_logging(bootstrap=False, migration=False, transaction=False)
|
configure_logging(bootstrap=False, migration=False, transaction=False)
|
||||||
assert not logging.getLogger("kanta.bootstrap").propagate
|
assert logging.getLogger("kanta.bootstrap").disabled
|
||||||
assert not logging.getLogger("kanta.migration").propagate
|
assert logging.getLogger("kanta.migration").disabled
|
||||||
assert not logging.getLogger("kanta.transaction").propagate
|
assert logging.getLogger("kanta.transaction").disabled
|
||||||
|
|
||||||
|
|
||||||
def test_configure_logging_skiproot_false_leaves_kanta_propagation():
|
def test_configure_logging_skiproot_false_routes_via_root():
|
||||||
kanta_logger = logging.getLogger("kanta")
|
configure_logging(skiproot=False)
|
||||||
kanta_logger.handlers.clear()
|
for name in ("kanta.bootstrap", "kanta.migration", "kanta.transaction"):
|
||||||
configure_logging(bootstrap=False, skiproot=False)
|
logger = logging.getLogger(name)
|
||||||
assert kanta_logger.propagate
|
assert logger.propagate
|
||||||
assert not kanta_logger.handlers
|
assert not logger.handlers
|
||||||
assert not logging.getLogger("kanta.bootstrap").propagate
|
|
||||||
|
|
||||||
|
def _setup_logging(**kwargs):
|
||||||
|
"""Default kanta logging; the event loggers are INFO out of the box."""
|
||||||
|
configure_logging(**kwargs)
|
||||||
|
|
||||||
|
|
||||||
def test_log_change_no_diff(capsys):
|
def test_log_change_no_diff(capsys):
|
||||||
kanta_logger = logging.getLogger("kanta")
|
_setup_logging()
|
||||||
kanta_logger.handlers.clear()
|
|
||||||
configure_logging()
|
|
||||||
log_change("test", {})
|
log_change("test", {})
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "test" in captured.err
|
assert "test" in captured.err
|
||||||
|
|
||||||
|
|
||||||
def test_log_change_appends_extra_string(capsys):
|
def test_log_change_appends_extra_string(capsys, monkeypatch):
|
||||||
kanta_logger = logging.getLogger("kanta")
|
monkeypatch.setenv("FORCE_COLOR", "1")
|
||||||
kanta_logger.handlers.clear()
|
monkeypatch.delenv("NO_COLOR", raising=False)
|
||||||
configure_logging()
|
_setup_logging()
|
||||||
log_change("export", {}, extra="mydb.db")
|
log_change("export", {}, extra="mydb.db")
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "export" in captured.err
|
assert "export" in captured.err
|
||||||
@@ -82,9 +92,7 @@ def test_log_change_appends_extra_string(capsys):
|
|||||||
|
|
||||||
|
|
||||||
def test_log_change_log_diff_false(capsys, monkeypatch):
|
def test_log_change_log_diff_false(capsys, monkeypatch):
|
||||||
kanta_logger = logging.getLogger("kanta")
|
_setup_logging()
|
||||||
kanta_logger.handlers.clear()
|
|
||||||
configure_logging()
|
|
||||||
|
|
||||||
def _boom(*args, **kwargs):
|
def _boom(*args, **kwargs):
|
||||||
raise AssertionError("format_diff should not be called")
|
raise AssertionError("format_diff should not be called")
|
||||||
@@ -97,9 +105,7 @@ def test_log_change_log_diff_false(capsys, monkeypatch):
|
|||||||
|
|
||||||
|
|
||||||
def test_configure_logging_diff_false(capsys):
|
def test_configure_logging_diff_false(capsys):
|
||||||
kanta_logger = logging.getLogger("kanta")
|
_setup_logging(diff=False)
|
||||||
kanta_logger.handlers.clear()
|
|
||||||
configure_logging(diff=False)
|
|
||||||
log_change("update", {"counter": 5}, previous={})
|
log_change("update", {"counter": 5}, previous={})
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
assert "update" in captured.err
|
assert "update" in captured.err
|
||||||
@@ -107,9 +113,7 @@ def test_configure_logging_diff_false(capsys):
|
|||||||
|
|
||||||
|
|
||||||
def test_configure_logging_diff_true_reenables(capsys):
|
def test_configure_logging_diff_true_reenables(capsys):
|
||||||
kanta_logger = logging.getLogger("kanta")
|
_setup_logging(diff=False)
|
||||||
kanta_logger.handlers.clear()
|
|
||||||
configure_logging(diff=False)
|
|
||||||
configure_logging(diff=True)
|
configure_logging(diff=True)
|
||||||
log_change("update", {"counter": 5}, previous={})
|
log_change("update", {"counter": 5}, previous={})
|
||||||
captured = capsys.readouterr()
|
captured = capsys.readouterr()
|
||||||
|
|||||||
@@ -0,0 +1,481 @@
|
|||||||
|
"""Tests for repeated -r ranges: unit window intersection, plain item slicing."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from kanta.replaylog import is_time_spec, parse_time_spec
|
||||||
|
from kanta.serialization import JsonSerializer
|
||||||
|
from kanta.structs import META_FORMAT, ChangeRecord, Snapshot
|
||||||
|
from tests.support import Data, make_kanta
|
||||||
|
|
||||||
|
# The CLI reads files with its own JSON Kanta instance unless -k is given.
|
||||||
|
JSON = ("json", JsonSerializer)
|
||||||
|
|
||||||
|
T0 = datetime(2026, 1, 1, tzinfo=UTC)
|
||||||
|
NOW = datetime(2026, 9, 28, 12, 0, 0, tzinfo=UTC)
|
||||||
|
|
||||||
|
|
||||||
|
def _run_cli(path, *args):
|
||||||
|
from kanta.__main__ import main
|
||||||
|
|
||||||
|
return main([str(path), *args])
|
||||||
|
|
||||||
|
|
||||||
|
def _build(path, increments=3):
|
||||||
|
async def build():
|
||||||
|
kanta = make_kanta(path, Data, JSON)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
for i in range(increments):
|
||||||
|
with kanta.transaction(f"inc{i}", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
asyncio.run(build())
|
||||||
|
|
||||||
|
|
||||||
|
def test_plain_range_restricts_dump_to_last_printed(tmp_path, capsys):
|
||||||
|
"""The JSON dump holds the state after the last printed item."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Items are bootstrap, inc0, inc1, inc2: 1:3 prints inc0 and inc1.
|
||||||
|
assert _run_cli(path, "-r", "1:3", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "bootstrap" not in out.err
|
||||||
|
assert "inc0" in out.err
|
||||||
|
assert "inc1" in out.err
|
||||||
|
assert "inc2" not in out.err
|
||||||
|
# The dump is the state after inc1, the last printed record.
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 2}
|
||||||
|
|
||||||
|
# Inclusive end.
|
||||||
|
assert _run_cli(path, "-r", "0..1") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "bootstrap" in err
|
||||||
|
assert "inc0" in err
|
||||||
|
assert "inc1" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_plain_single_index(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
assert _run_cli(path, "-r", "0") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "bootstrap" in err
|
||||||
|
assert "inc0" not in err
|
||||||
|
|
||||||
|
# Negative index counts from the end of the filtered output.
|
||||||
|
assert _run_cli(path, "-r", "-1") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc2" in err
|
||||||
|
assert "inc1" not in err
|
||||||
|
|
||||||
|
# A single index must exist; out of range is exit 2.
|
||||||
|
assert _run_cli(path, "-r", "99") == 2
|
||||||
|
assert "99" in capsys.readouterr().err
|
||||||
|
|
||||||
|
|
||||||
|
def test_plain_ranges_intersect(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# [1:) and [:3) intersect to items 1 and 2.
|
||||||
|
assert _run_cli(path, "-r", "1:", "-r", ":3") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "bootstrap" not in err
|
||||||
|
assert "inc0" in err
|
||||||
|
assert "inc1" in err
|
||||||
|
assert "inc2" not in err
|
||||||
|
|
||||||
|
# An empty intersection prints nothing, exit 0 (truncation, not error).
|
||||||
|
assert _run_cli(path, "-r", ":1", "-r", "3:") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc" not in err
|
||||||
|
assert "bootstrap" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_plain_applies_after_grep(tmp_path, capsys):
|
||||||
|
"""Plain numbers index into what survives --grep, order-independently."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Grep leaves inc0..inc2; plain 0 picks the first of those.
|
||||||
|
assert _run_cli(path, "--grep", "inc", "-r", "0") == 0
|
||||||
|
first = capsys.readouterr().err
|
||||||
|
assert "inc0" in first
|
||||||
|
assert "inc1" not in first
|
||||||
|
assert "bootstrap" not in first
|
||||||
|
|
||||||
|
assert _run_cli(path, "-r", "0", "--grep", "inc") == 0
|
||||||
|
assert capsys.readouterr().err == first
|
||||||
|
|
||||||
|
# Negative index from the filtered end.
|
||||||
|
assert _run_cli(path, "--grep", "inc", "-r", "-1") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc2" in err
|
||||||
|
assert "inc1" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_unit_specs_intersect(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# r1:r4 = bootstrap..inc1, r2:r5 = inc0..inc2; intersection is inc0, inc1.
|
||||||
|
assert _run_cli(path, "-r", "r1:r4", "-r", "r2:r5", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "bootstrap" not in out.err
|
||||||
|
assert "inc0" in out.err
|
||||||
|
assert "inc1" in out.err
|
||||||
|
assert "inc2" not in out.err
|
||||||
|
# The replay end is the intersected window end (state after inc1).
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 2}
|
||||||
|
|
||||||
|
|
||||||
|
def test_unit_and_plain_combine(tmp_path, capsys):
|
||||||
|
"""Unit window first, then plain slicing over the window's items."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Window r2:r4 yields inc0, inc1; plain item 0 prints only inc0.
|
||||||
|
assert _run_cli(path, "-r", "r2:r4", "-r", "0", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "inc0" in out.err
|
||||||
|
assert "inc1" not in out.err
|
||||||
|
# The dump is the state after the last printed item (inc0).
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 1}
|
||||||
|
|
||||||
|
|
||||||
|
def test_context_expands_after_grep(tmp_path, capsys):
|
||||||
|
"""Context records come from the whole file, free of --grep."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path, increments=5)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
assert _run_cli(path, "--grep", "inc2", "--context", "1") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc1" in err
|
||||||
|
assert "inc2" in err
|
||||||
|
assert "inc3" in err
|
||||||
|
assert "inc0" not in err
|
||||||
|
assert "inc4" not in err
|
||||||
|
assert "bootstrap" not in err
|
||||||
|
|
||||||
|
# Context 0 (default) changes nothing.
|
||||||
|
assert _run_cli(path, "--grep", "inc2") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc1" not in err
|
||||||
|
assert "inc2" in err
|
||||||
|
|
||||||
|
# Context is clamped at the start of the file.
|
||||||
|
assert _run_cli(path, "--grep", "bootstrap", "--context", "2") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "bootstrap" in err
|
||||||
|
assert "inc0" in err
|
||||||
|
assert "inc1" in err
|
||||||
|
assert "inc2" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_context_expands_past_unit_window(tmp_path, capsys):
|
||||||
|
"""Context expands whatever a unit selection would show, past the window."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# -r r2 alone shows only inc0 (the record producing state 2); context 1
|
||||||
|
# expands past the window to bootstrap and inc1, and the dump follows
|
||||||
|
# the last printed record.
|
||||||
|
assert _run_cli(path, "-r", "r2", "--context", "1", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "bootstrap" in out.err
|
||||||
|
assert "inc0" in out.err
|
||||||
|
assert "inc1" in out.err
|
||||||
|
assert "inc2" not in out.err
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 2}
|
||||||
|
|
||||||
|
|
||||||
|
def test_context_expands_snapshot_selection(tmp_path, capsys):
|
||||||
|
"""-r sN --context N shows the snapshot plus its neighbors."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
serializer = JsonSerializer()
|
||||||
|
framer = serializer.framer_cls()
|
||||||
|
out_bytes = bytearray()
|
||||||
|
out_bytes += framer.frame_change(
|
||||||
|
serializer.encode(ChangeRecord(ts=T0, a="inc0", v=0, diff={"counter": 1})),
|
||||||
|
record_offset=len(out_bytes),
|
||||||
|
)
|
||||||
|
snap = Snapshot(ts=T0, v=0, state={"counter": 1}, r=1, k=META_FORMAT)
|
||||||
|
out_bytes += framer.frame_snapshot(
|
||||||
|
serializer.encode(snap), record_offset=len(out_bytes)
|
||||||
|
)
|
||||||
|
out_bytes += framer.frame_change(
|
||||||
|
serializer.encode(ChangeRecord(ts=T0, a="inc1", v=0, diff={"counter": 2})),
|
||||||
|
record_offset=len(out_bytes),
|
||||||
|
)
|
||||||
|
path.write_bytes(bytes(out_bytes))
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Without context, s1 (the file starts with a change, so s0 is l0)
|
||||||
|
# shows just the snapshot and its state.
|
||||||
|
assert _run_cli(path, "-r", "s1", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "snapshot s1" in out.err
|
||||||
|
assert "inc0" not in out.err
|
||||||
|
assert json.loads(out.out) == {"counter": 1}
|
||||||
|
|
||||||
|
# With context, the selection expands past the snapshot line.
|
||||||
|
assert _run_cli(path, "-r", "s1", "--context", "1", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "inc0" in out.err
|
||||||
|
assert "snapshot s1" in out.err
|
||||||
|
assert "inc1" in out.err
|
||||||
|
assert json.loads(out.out) == {"counter": 2}
|
||||||
|
|
||||||
|
|
||||||
|
def test_context_merges_overlaps(tmp_path, capsys):
|
||||||
|
"""Nearby matches share context instead of duplicating records."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path, increments=5)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Grep leaves inc0..inc4; plain 1:3 selects inc1, inc2; context 1
|
||||||
|
# expands to inc0..inc3, each printed exactly once (overlaps merged).
|
||||||
|
assert _run_cli(path, "--grep", "inc", "-r", "1:3", "--context", "1") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
for action in ["inc0", "inc1", "inc2", "inc3"]:
|
||||||
|
assert err.count(action) == 1
|
||||||
|
assert "bootstrap" not in err
|
||||||
|
assert "inc4" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_grep_and_context_shift_dump_to_last_printed(tmp_path, capsys):
|
||||||
|
"""Grep and --context are display filters, but the dump follows the
|
||||||
|
last printed record, so they move it too."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_build(path)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Only inc1 prints; the dump is the state after inc1.
|
||||||
|
assert _run_cli(path, "--grep", "inc1", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "inc0" not in out.err
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 2}
|
||||||
|
|
||||||
|
# Context extends the printout past the match; the dump follows it.
|
||||||
|
assert _run_cli(path, "--grep", "inc1", "--context", "1", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "inc2" in out.err
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 3}
|
||||||
|
|
||||||
|
# Nothing printed: the dump falls back to the range-end state.
|
||||||
|
assert _run_cli(path, "--grep", "zzz", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 3}
|
||||||
|
|
||||||
|
|
||||||
|
def _write_timed(path, entries):
|
||||||
|
"""Write one change record per (timestamp, action, counter) entry."""
|
||||||
|
serializer = JsonSerializer()
|
||||||
|
framer = serializer.framer_cls()
|
||||||
|
out = bytearray()
|
||||||
|
for ts, action, counter in entries:
|
||||||
|
out += framer.frame_change(
|
||||||
|
serializer.encode(
|
||||||
|
ChangeRecord(ts=ts, a=action, v=0, diff={"counter": counter})
|
||||||
|
),
|
||||||
|
record_offset=len(out),
|
||||||
|
)
|
||||||
|
path.write_bytes(bytes(out))
|
||||||
|
|
||||||
|
|
||||||
|
def test_is_time_spec_classification():
|
||||||
|
for spec in [
|
||||||
|
"30s",
|
||||||
|
"15min",
|
||||||
|
"1h",
|
||||||
|
"7d",
|
||||||
|
"2w",
|
||||||
|
"2026-09-28",
|
||||||
|
"2026-09-28T10",
|
||||||
|
"2026-09-28T10:30",
|
||||||
|
"20260928T103045",
|
||||||
|
"2026-09-24/2026-09-28",
|
||||||
|
"2026-09-24/",
|
||||||
|
"/2026-09-28",
|
||||||
|
"2026-13-45", # malformed ISO still classifies as time, then errors
|
||||||
|
]:
|
||||||
|
assert is_time_spec(spec), spec
|
||||||
|
for spec in ["5", "-1", "2:5", "2..5", ":-1", "r3", "s-1", "l10", "v0", "r1:r4"]:
|
||||||
|
assert not is_time_spec(spec), spec
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_time_spec_relative():
|
||||||
|
assert parse_time_spec("1h", NOW) == (NOW - timedelta(hours=1), NOW)
|
||||||
|
assert parse_time_spec("15min", NOW) == (NOW - timedelta(minutes=15), NOW)
|
||||||
|
assert parse_time_spec("30s", NOW) == (NOW - timedelta(seconds=30), NOW)
|
||||||
|
assert parse_time_spec("7d", NOW) == (NOW - timedelta(days=7), NOW)
|
||||||
|
assert parse_time_spec("2w", NOW) == (NOW - timedelta(weeks=2), NOW)
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_time_spec_iso_periods():
|
||||||
|
assert parse_time_spec("2026-09-28", NOW) == (
|
||||||
|
datetime(2026, 9, 28, tzinfo=UTC),
|
||||||
|
datetime(2026, 9, 29, tzinfo=UTC),
|
||||||
|
)
|
||||||
|
assert parse_time_spec("2026-09-28T10", NOW) == (
|
||||||
|
datetime(2026, 9, 28, 10, tzinfo=UTC),
|
||||||
|
datetime(2026, 9, 28, 11, tzinfo=UTC),
|
||||||
|
)
|
||||||
|
assert parse_time_spec("2026-09-28T10:30", NOW) == (
|
||||||
|
datetime(2026, 9, 28, 10, 30, tzinfo=UTC),
|
||||||
|
datetime(2026, 9, 28, 10, 31, tzinfo=UTC),
|
||||||
|
)
|
||||||
|
assert parse_time_spec("20260928T103045", NOW) == (
|
||||||
|
datetime(2026, 9, 28, 10, 30, 45, tzinfo=UTC),
|
||||||
|
datetime(2026, 9, 28, 10, 30, 46, tzinfo=UTC),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_time_spec_iso_ranges():
|
||||||
|
# Inclusive of the end's whole period.
|
||||||
|
assert parse_time_spec("2026-09-24/2026-09-28", NOW) == (
|
||||||
|
datetime(2026, 9, 24, tzinfo=UTC),
|
||||||
|
datetime(2026, 9, 29, tzinfo=UTC),
|
||||||
|
)
|
||||||
|
# Open ends are unbounded.
|
||||||
|
assert parse_time_spec("2026-09-24/", NOW) == (
|
||||||
|
datetime(2026, 9, 24, tzinfo=UTC),
|
||||||
|
None,
|
||||||
|
)
|
||||||
|
assert parse_time_spec("/2026-09-28", NOW) == (
|
||||||
|
None,
|
||||||
|
datetime(2026, 9, 29, tzinfo=UTC),
|
||||||
|
)
|
||||||
|
with pytest.raises(ValueError, match="invalid time"):
|
||||||
|
parse_time_spec("2026-13-45", NOW)
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_time_iso_filters_items(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_write_timed(
|
||||||
|
path,
|
||||||
|
[
|
||||||
|
(datetime(2026, 9, 24, 10, tzinfo=UTC), "inc0", 1),
|
||||||
|
(datetime(2026, 9, 28, 8, tzinfo=UTC), "inc1", 2),
|
||||||
|
(datetime(2026, 9, 28, 23, tzinfo=UTC), "inc2", 3),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# A date denotes the full UTC day; the dump follows the last printed item.
|
||||||
|
assert _run_cli(path, "-r", "2026-09-28", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "inc0" not in out.err
|
||||||
|
assert "inc1" in out.err
|
||||||
|
assert "inc2" in out.err
|
||||||
|
assert json.loads(out.out) == {"counter": 3}
|
||||||
|
|
||||||
|
# Hour precision.
|
||||||
|
assert _run_cli(path, "-r", "2026-09-28T08") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc1" in err
|
||||||
|
assert "inc2" not in err
|
||||||
|
|
||||||
|
# Compact minute precision.
|
||||||
|
assert _run_cli(path, "-r", "20260924T1000") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc0" in err
|
||||||
|
assert "inc1" not in err
|
||||||
|
|
||||||
|
# An inclusive ISO range covers the end's whole period.
|
||||||
|
assert _run_cli(path, "-r", "2026-09-24/2026-09-28") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc0" in err and "inc2" in err
|
||||||
|
|
||||||
|
# Open-ended range.
|
||||||
|
assert _run_cli(path, "-r", "/2026-09-25") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc0" in err
|
||||||
|
assert "inc1" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_time_combines_with_plain_and_grep(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_write_timed(
|
||||||
|
path,
|
||||||
|
[
|
||||||
|
(datetime(2026, 9, 24, 10, tzinfo=UTC), "old", 1),
|
||||||
|
(datetime(2026, 9, 28, 8, tzinfo=UTC), "new_a", 2),
|
||||||
|
(datetime(2026, 9, 28, 9, tzinfo=UTC), "new_b", 3),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# Time filter first, then the plain slice over what remains.
|
||||||
|
assert _run_cli(path, "-r", "2026-09-28", "-r", "0", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "new_a" in out.err
|
||||||
|
assert "new_b" not in out.err
|
||||||
|
assert json.loads(out.out) == {"counter": 2}
|
||||||
|
|
||||||
|
# Time and grep are both restrictive.
|
||||||
|
assert _run_cli(path, "-r", "2026-09-28", "--grep", "new_b") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "new_a" not in err
|
||||||
|
assert "new_b" in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_time_matches_items_not_windows(tmp_path, capsys):
|
||||||
|
"""Non-monotonic timestamps: items match individually, not by position."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_write_timed(
|
||||||
|
path,
|
||||||
|
[
|
||||||
|
(datetime(2026, 9, 28, 12, tzinfo=UTC), "noon", 1),
|
||||||
|
(datetime(2026, 9, 28, 8, tzinfo=UTC), "morning", 2),
|
||||||
|
(datetime(2026, 9, 28, 10, tzinfo=UTC), "ten", 3),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
assert _run_cli(path, "-r", "2026-09-28T10") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "ten" in err
|
||||||
|
assert "noon" not in err
|
||||||
|
assert "morning" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_time_relative(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
now = datetime.now(UTC)
|
||||||
|
_write_timed(
|
||||||
|
path,
|
||||||
|
[
|
||||||
|
(now - timedelta(hours=2), "old", 1),
|
||||||
|
(now - timedelta(minutes=30), "recent", 2),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
assert _run_cli(path, "-r", "1h") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "recent" in err
|
||||||
|
assert "old" not in err
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_time_invalid_is_error(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_write_timed(path, [(T0, "inc0", 1)])
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
assert _run_cli(path, "-r", "2026-13-45") == 1
|
||||||
|
assert "Invalid --range value" in capsys.readouterr().err
|
||||||
@@ -0,0 +1,480 @@
|
|||||||
|
"""Tests for revision numbers: runtime counter, snapshot ``r`` field, CLI rN."""
|
||||||
|
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
from datetime import UTC, datetime, timedelta
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from kanta.exceptions import DataIntegrityError
|
||||||
|
from kanta.serialization import JsonSerializer
|
||||||
|
from kanta.structs import META_FORMAT, ChangeRecord, Snapshot
|
||||||
|
from tests.support import Data, make_kanta, read_last_snapshot
|
||||||
|
from tests.test_rotation import make_clock, read_all, write_history
|
||||||
|
|
||||||
|
T0 = datetime(2026, 1, 1, tzinfo=UTC)
|
||||||
|
|
||||||
|
# The CLI reads files with its own JSON Kanta instance unless -k is given.
|
||||||
|
JSON = ("json", JsonSerializer)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_revision_requires_open(tmp_path, format_config):
|
||||||
|
kanta = make_kanta(tmp_path / "data.kantadb", Data, format_config)
|
||||||
|
with pytest.raises(DataIntegrityError):
|
||||||
|
kanta.rev
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_revision_counts_change_records(tmp_path, format_config):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
kanta = make_kanta(path, Data, format_config)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
# The bootstrap record is the first diff: state is at revision 1.
|
||||||
|
assert kanta.rev == 1
|
||||||
|
|
||||||
|
with kanta.transaction("inc", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
assert kanta.rev == 2
|
||||||
|
|
||||||
|
# A no-op transaction queues no record and does not advance the revision.
|
||||||
|
with kanta.transaction("noop", log=False):
|
||||||
|
pass
|
||||||
|
assert kanta.rev == 2
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_revision_restored_on_reopen(tmp_path, format_config):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
kanta = make_kanta(path, Data, format_config)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
for _ in range(3):
|
||||||
|
with kanta.transaction("inc", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
kanta2 = make_kanta(path, Data, format_config)
|
||||||
|
await kanta2.open(log=False)
|
||||||
|
assert kanta2.rev == 4
|
||||||
|
await kanta2.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_snapshot_carries_revision(tmp_path, format_config):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
kanta = make_kanta(path, Data, format_config)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
for _ in range(2):
|
||||||
|
with kanta.transaction("inc", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
await kanta.flush()
|
||||||
|
kanta.request_snapshot()
|
||||||
|
kanta._impl.maybe_snapshot()
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
snap = read_last_snapshot(path, format_config)
|
||||||
|
assert snap is not None
|
||||||
|
assert snap.r == 3
|
||||||
|
assert snap.k == META_FORMAT
|
||||||
|
|
||||||
|
# Reopening from the snapshot keeps the absolute revision.
|
||||||
|
kanta2 = make_kanta(path, Data, format_config)
|
||||||
|
await kanta2.open(log=False)
|
||||||
|
assert kanta2.rev == 3
|
||||||
|
with kanta2.transaction("inc", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
assert kanta2.rev == 4
|
||||||
|
await kanta2.close()
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_legacy_snapshot_recounted_and_rewritten(tmp_path, format_config):
|
||||||
|
"""Meta migration k1 rewrites a k0 file in full (see kanta/metamigr.py)."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
kanta = make_kanta(path, Data, format_config)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
for _ in range(2):
|
||||||
|
with kanta.transaction("inc", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
await kanta.flush()
|
||||||
|
kanta.request_snapshot()
|
||||||
|
kanta._impl.maybe_snapshot()
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
# Simulate a pre-revision database: rewrite the last snapshot without k,
|
||||||
|
# with a wrong r, then append one more change record.
|
||||||
|
_, serializer_cls = format_config
|
||||||
|
serializer = serializer_cls()
|
||||||
|
framer = serializer.framer_cls()
|
||||||
|
records = read_all(path, format_config)
|
||||||
|
out = bytearray()
|
||||||
|
for rec in records:
|
||||||
|
if isinstance(rec, Snapshot):
|
||||||
|
rec = Snapshot(ts=rec.ts, v=rec.v, state=rec.state, m=rec.m, r=0)
|
||||||
|
out += framer.frame_snapshot(
|
||||||
|
serializer.encode(rec), record_offset=len(out)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
out += framer.frame_change(
|
||||||
|
serializer.encode(rec), record_offset=len(out)
|
||||||
|
)
|
||||||
|
extra = ChangeRecord(ts=T0, a="inc", v=0, diff={"counter": 3})
|
||||||
|
out += framer.frame_change(serializer.encode(extra), record_offset=len(out))
|
||||||
|
path.write_bytes(bytes(out))
|
||||||
|
|
||||||
|
kanta2 = make_kanta(path, Data, format_config)
|
||||||
|
await kanta2.open(log=False)
|
||||||
|
# bootstrap + 2 changes + trailing change = revision 4, despite r=0 above.
|
||||||
|
assert kanta2.rev == 4
|
||||||
|
assert kanta2.data.counter == 3
|
||||||
|
await kanta2.close()
|
||||||
|
|
||||||
|
# The file was rewritten in full: all change records preserved in order,
|
||||||
|
# the legacy intermediate snapshot dropped, and a k1 closing snapshot
|
||||||
|
# carrying the total revision appended.
|
||||||
|
records = read_all(path, format_config)
|
||||||
|
changes = [r for r in records if isinstance(r, ChangeRecord)]
|
||||||
|
assert [c.a for c in changes] == ["bootstrap", "inc", "inc", "inc"]
|
||||||
|
snapshots = [r for r in records if isinstance(r, Snapshot)]
|
||||||
|
assert len(snapshots) == 1
|
||||||
|
assert isinstance(records[-1], Snapshot)
|
||||||
|
closing = snapshots[0]
|
||||||
|
assert closing.k == META_FORMAT
|
||||||
|
assert closing.r == 4
|
||||||
|
assert closing.state["counter"] == 3
|
||||||
|
|
||||||
|
# The next open trusts the stored r (file unchanged this time).
|
||||||
|
content = path.read_bytes()
|
||||||
|
kanta3 = make_kanta(path, Data, format_config)
|
||||||
|
await kanta3.open(log=False)
|
||||||
|
assert kanta3.rev == 4
|
||||||
|
await kanta3.close()
|
||||||
|
assert path.read_bytes() == content
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_legacy_migration_preserves_leading_snapshot(tmp_path, format_config):
|
||||||
|
"""A legacy file starting with a snapshot keeps it, re-stamped k1."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_, serializer_cls = format_config
|
||||||
|
serializer = serializer_cls()
|
||||||
|
framer = serializer.framer_cls()
|
||||||
|
snap_ts = T0 - timedelta(days=40)
|
||||||
|
snap_m = T0 - timedelta(days=41)
|
||||||
|
leading = Snapshot(ts=snap_ts, v=3, state={"users": {}, "counter": 7}, m=snap_m)
|
||||||
|
out = bytearray()
|
||||||
|
out += framer.frame_snapshot(serializer.encode(leading), record_offset=0)
|
||||||
|
for i in range(2):
|
||||||
|
rec = ChangeRecord(ts=T0, a=f"inc{i}", v=3, diff={"counter": 8 + i})
|
||||||
|
out += framer.frame_change(serializer.encode(rec), record_offset=len(out))
|
||||||
|
path.write_bytes(bytes(out))
|
||||||
|
|
||||||
|
kanta = make_kanta(path, Data, format_config)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
# Only the in-file history is countable: 2 change records.
|
||||||
|
assert kanta.rev == 2
|
||||||
|
assert kanta.data.counter == 9
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
records = read_all(path, format_config)
|
||||||
|
assert isinstance(records[0], Snapshot)
|
||||||
|
assert records[0].k == META_FORMAT
|
||||||
|
assert records[0].r == 0
|
||||||
|
assert records[0].ts == snap_ts
|
||||||
|
assert records[0].m == snap_m
|
||||||
|
assert records[0].v == 3
|
||||||
|
assert records[0].state == {"users": {}, "counter": 7}
|
||||||
|
changes = [r for r in records if isinstance(r, ChangeRecord)]
|
||||||
|
assert [c.a for c in changes] == ["inc0", "inc1"]
|
||||||
|
assert isinstance(records[-1], Snapshot)
|
||||||
|
assert records[-1].k == META_FORMAT
|
||||||
|
assert records[-1].r == 2
|
||||||
|
assert records[-1].state["counter"] == 9
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_rotation_of_legacy_file_migrates(tmp_path, format_config):
|
||||||
|
"""Rotation and meta migration coincide: rotation already produces k1
|
||||||
|
snapshots with revisions counted from pre-rotation record positions."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
_, serializer_cls = format_config
|
||||||
|
serializer = serializer_cls()
|
||||||
|
framer = serializer.framer_cls()
|
||||||
|
|
||||||
|
def change(action, day, counter):
|
||||||
|
return ChangeRecord(
|
||||||
|
ts=T0 + day * timedelta(days=1), a=action, v=0, diff={"counter": counter}
|
||||||
|
)
|
||||||
|
|
||||||
|
legacy_snap = Snapshot(
|
||||||
|
ts=T0 - 40 * timedelta(days=1), v=0, state={"users": {}, "counter": 1}, r=0
|
||||||
|
)
|
||||||
|
out = bytearray()
|
||||||
|
out += framer.frame_change(
|
||||||
|
serializer.encode(change("bootstrap", -41, 0)), record_offset=len(out)
|
||||||
|
)
|
||||||
|
out += framer.frame_change(
|
||||||
|
serializer.encode(change("day-40", -40, 1)), record_offset=len(out)
|
||||||
|
)
|
||||||
|
out += framer.frame_snapshot(
|
||||||
|
serializer.encode(legacy_snap), record_offset=len(out)
|
||||||
|
)
|
||||||
|
out += framer.frame_change(
|
||||||
|
serializer.encode(change("day-20", -20, 2)), record_offset=len(out)
|
||||||
|
)
|
||||||
|
out += framer.frame_change(
|
||||||
|
serializer.encode(change("day-5", -5, 3)), record_offset=len(out)
|
||||||
|
)
|
||||||
|
path.write_bytes(bytes(out))
|
||||||
|
|
||||||
|
cell = [T0]
|
||||||
|
kanta = make_kanta(path, Data, format_config, retention=30 * timedelta(days=1))
|
||||||
|
kanta.clock(make_clock(cell))
|
||||||
|
await kanta.open(log=False)
|
||||||
|
assert kanta.rev == 4
|
||||||
|
assert kanta.data.counter == 3
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
# The rotation's leading snapshot is k1 with the pre-rotation revision.
|
||||||
|
records = read_all(path, format_config)
|
||||||
|
assert isinstance(records[0], Snapshot)
|
||||||
|
assert records[0].k == META_FORMAT
|
||||||
|
assert records[0].r == 2
|
||||||
|
assert [r.a for r in records if isinstance(r, ChangeRecord)] == [
|
||||||
|
"day-20",
|
||||||
|
"day-5",
|
||||||
|
]
|
||||||
|
|
||||||
|
# No further rewriting on the next open.
|
||||||
|
content = path.read_bytes()
|
||||||
|
kanta2 = make_kanta(path, Data, format_config)
|
||||||
|
await kanta2.open(log=False)
|
||||||
|
assert kanta2.rev == 4
|
||||||
|
await kanta2.close()
|
||||||
|
assert path.read_bytes() == content
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_legacy_snapshot_readonly_recounts_without_writing(
|
||||||
|
tmp_path, format_config
|
||||||
|
):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
kanta = make_kanta(path, Data, format_config)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
with kanta.transaction("inc", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
# Inject a legacy snapshot (no k marker) at the start; the whole file is
|
||||||
|
# re-framed with a single framer so binary checksums stay valid.
|
||||||
|
_, serializer_cls = format_config
|
||||||
|
serializer = serializer_cls()
|
||||||
|
framer = serializer.framer_cls()
|
||||||
|
legacy = Snapshot(ts=T0, v=0, state={"users": {}, "counter": 0}, r=0)
|
||||||
|
out = bytearray()
|
||||||
|
out += framer.frame_snapshot(serializer.encode(legacy), record_offset=0)
|
||||||
|
for rec in read_all(path, format_config):
|
||||||
|
out += framer.frame_change(serializer.encode(rec), record_offset=len(out))
|
||||||
|
path.write_bytes(bytes(out))
|
||||||
|
content = path.read_bytes()
|
||||||
|
|
||||||
|
kanta2 = make_kanta(path, Data, format_config)
|
||||||
|
await kanta2.open(readonly=True, log=False)
|
||||||
|
assert kanta2.rev == 2
|
||||||
|
await kanta2.close()
|
||||||
|
assert path.read_bytes() == content
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.mark.asyncio
|
||||||
|
async def test_rotation_preserves_revisions(tmp_path, format_config):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
await write_history(path, format_config, days=[-40, -20, -5])
|
||||||
|
|
||||||
|
cell = [T0]
|
||||||
|
kanta = make_kanta(path, Data, format_config, retention=30 * timedelta(days=1))
|
||||||
|
kanta.clock(make_clock(cell))
|
||||||
|
await kanta.open(log=False)
|
||||||
|
# bootstrap + 3 changes = revision 4 before and after the rotation.
|
||||||
|
assert kanta.rev == 4
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
records = read_all(path, format_config)
|
||||||
|
leading = records[0]
|
||||||
|
assert isinstance(leading, Snapshot)
|
||||||
|
# One change was dropped (bootstrap + day-40 are before the cutoff
|
||||||
|
# position... bootstrap is a change record too: 2 records before).
|
||||||
|
assert leading.r == 2
|
||||||
|
assert leading.k == META_FORMAT
|
||||||
|
|
||||||
|
kanta2 = make_kanta(path, Data, format_config)
|
||||||
|
await kanta2.open(log=False)
|
||||||
|
assert kanta2.rev == 4
|
||||||
|
with kanta2.transaction("inc", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
assert kanta2.rev == 5
|
||||||
|
await kanta2.close()
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_revision_snapshot_base_file(tmp_path, capsys):
|
||||||
|
"""File = snapshot {r: 2} + one change; the two producing changes are gone."""
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
serializer = JsonSerializer()
|
||||||
|
framer = serializer.framer_cls()
|
||||||
|
snap = Snapshot(
|
||||||
|
ts=T0, v=0, state={"foo": 1, "bar": 1}, r=2, k=META_FORMAT
|
||||||
|
)
|
||||||
|
out = bytearray()
|
||||||
|
out += framer.frame_snapshot(serializer.encode(snap), record_offset=0)
|
||||||
|
change = ChangeRecord(ts=T0, a="add_baz", v=0, diff={"baz": 1})
|
||||||
|
out += framer.frame_change(serializer.encode(change), record_offset=len(out))
|
||||||
|
path.write_bytes(bytes(out))
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# r1's producing change is gone and no snapshot holds state 1: error.
|
||||||
|
assert _run_cli(path, "-r", "r1") == 2
|
||||||
|
assert "r1" in capsys.readouterr().err
|
||||||
|
|
||||||
|
# r2 prints the snapshot line (it literally carries r: 2) and its state;
|
||||||
|
# revisions are shown on change records, not on snapshot indicators.
|
||||||
|
assert _run_cli(path, "-r", "r2", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "snapshot s0" in out.err
|
||||||
|
assert "r2" not in out.err
|
||||||
|
assert json.loads(out.out) == {"foo": 1, "bar": 1}
|
||||||
|
|
||||||
|
# r3 is the change, replayed on the snapshot base; state includes baz.
|
||||||
|
assert _run_cli(path, "-r", "r3", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "add_baz" in out.err
|
||||||
|
assert " r3" in out.err
|
||||||
|
assert json.loads(out.out) == {"foo": 1, "bar": 1, "baz": 1}
|
||||||
|
|
||||||
|
# Single index and half-open range agree.
|
||||||
|
assert _run_cli(path, "-r", "r2") == 0
|
||||||
|
single = capsys.readouterr().err
|
||||||
|
assert _run_cli(path, "-r", "r2:r3") == 0
|
||||||
|
assert capsys.readouterr().err == single
|
||||||
|
|
||||||
|
|
||||||
|
def _run_cli(path, *args):
|
||||||
|
from kanta.__main__ import main
|
||||||
|
|
||||||
|
return main([str(path), *args])
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_revision_ranges(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
|
||||||
|
async def build():
|
||||||
|
kanta = make_kanta(path, Data, JSON)
|
||||||
|
await kanta.open(log=False)
|
||||||
|
for i in range(3):
|
||||||
|
with kanta.transaction(f"inc{i}", log=False) as data:
|
||||||
|
data.counter += 1
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
asyncio.run(build())
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# A revision selects the record that produced that state, with the
|
||||||
|
# state: r1 and r1:r2 both show only the bootstrap and output state 1.
|
||||||
|
assert _run_cli(path, "-r", "r1") == 0
|
||||||
|
single = capsys.readouterr().err
|
||||||
|
assert "bootstrap" in single
|
||||||
|
# Change record headers carry the revision of the state they produce.
|
||||||
|
assert " r1" in single
|
||||||
|
assert "inc0" not in single
|
||||||
|
|
||||||
|
assert _run_cli(path, "-r", "r1:r2") == 0
|
||||||
|
assert capsys.readouterr().err == single
|
||||||
|
|
||||||
|
assert _run_cli(path, "-r", "r1", "-o", "-") == 0
|
||||||
|
assert json.loads(capsys.readouterr().out) == {"users": {}, "counter": 0}
|
||||||
|
|
||||||
|
# r1..r2 (inclusive end) adds the next change and outputs state 2.
|
||||||
|
assert _run_cli(path, "-r", "r1..r2", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "inc0" in out.err
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 1}
|
||||||
|
|
||||||
|
# r0 is not meaningful: no record produces the empty initial state.
|
||||||
|
assert _run_cli(path, "-r", "r0") == 2
|
||||||
|
assert "r0" in capsys.readouterr().err
|
||||||
|
|
||||||
|
# Range by revision.
|
||||||
|
assert _run_cli(path, "-r", "r2:r4") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "inc0" in err
|
||||||
|
assert "inc1" in err
|
||||||
|
assert "inc2" not in err
|
||||||
|
|
||||||
|
# Negative revisions are rejected.
|
||||||
|
assert _run_cli(path, "-r", "r-1") == 1
|
||||||
|
assert "negative" in capsys.readouterr().err
|
||||||
|
|
||||||
|
# Out-of-range single revision is a range error (exit 2).
|
||||||
|
assert _run_cli(path, "-r", "r99") == 2
|
||||||
|
assert "r99" in capsys.readouterr().err
|
||||||
|
|
||||||
|
# The same bounds as a range truncate Python-style instead: an empty
|
||||||
|
# display and the closest known state, not an error.
|
||||||
|
assert _run_cli(path, "-r", "r99:r100", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "bootstrap" not in out.err
|
||||||
|
assert "inc" not in out.err
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 3}
|
||||||
|
|
||||||
|
# Only the end beyond the last revision: truncated to end of file.
|
||||||
|
assert _run_cli(path, "-r", "r3:r99", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "inc1" in out.err
|
||||||
|
assert "inc2" in out.err
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 3}
|
||||||
|
|
||||||
|
|
||||||
|
def test_cli_revision_after_rotation(tmp_path, capsys):
|
||||||
|
path = tmp_path / "data.kantadb"
|
||||||
|
|
||||||
|
async def build():
|
||||||
|
await write_history(path, JSON, days=[-40, -20, -5])
|
||||||
|
cell = [T0]
|
||||||
|
kanta = make_kanta(
|
||||||
|
path, Data, JSON, retention=30 * timedelta(days=1)
|
||||||
|
)
|
||||||
|
kanta.clock(make_clock(cell))
|
||||||
|
await kanta.open(log=False)
|
||||||
|
await kanta.close()
|
||||||
|
|
||||||
|
asyncio.run(build())
|
||||||
|
capsys.readouterr()
|
||||||
|
|
||||||
|
# The leading snapshot holds revision 2 (bootstrap + day-40). Since the
|
||||||
|
# change that produced state 2 is gone, r2 shows the snapshot itself and
|
||||||
|
# outputs the snapshot state.
|
||||||
|
assert _run_cli(path, "-r", "r2", "-o", "-") == 0
|
||||||
|
out = capsys.readouterr()
|
||||||
|
assert "snapshot s0" in out.err
|
||||||
|
assert "r2" not in out.err
|
||||||
|
assert "day-20" not in out.err
|
||||||
|
assert json.loads(out.out) == {"users": {}, "counter": 1}
|
||||||
|
|
||||||
|
# r1 has no record in this file: a single index does not truncate.
|
||||||
|
assert _run_cli(path, "-r", "r1") == 2
|
||||||
|
assert "r1" in capsys.readouterr().err
|
||||||
|
|
||||||
|
# A range truncates like Python slicing instead: r1:r3 starts at the
|
||||||
|
# beginning of the file and stops before the r3 record, showing just the
|
||||||
|
# leading snapshot.
|
||||||
|
assert _run_cli(path, "-r", "r1:r3") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "snapshot s0" in err
|
||||||
|
assert "r2" not in err
|
||||||
|
assert "day-20" not in err
|
||||||
|
|
||||||
|
# Absolute revisions reach the retained change records.
|
||||||
|
assert _run_cli(path, "-r", "r3:r4") == 0
|
||||||
|
err = capsys.readouterr().err
|
||||||
|
assert "day-20" in err
|
||||||
|
assert "snapshot" not in err
|
||||||
|
assert "day-5" not in err
|
||||||
Reference in new issue
Block a user