Durable embedded apps¶

This guide covers running KGLite as the embedded database behind an application — the open → mutate → reopen lifecycle, persistence on close, and crash-safe durable writes via a write-ahead log (WAL).

If you only build a graph, query it, and throw it away, you don’t need any of this — KnowledgeGraph() plus Data Loading is enough. Reach for this guide when the graph is long-lived state your app reopens across runs: an agent’s memory, a knowledge base that accretes facts, a service that accepts writes between restarts.

The lifecycle entry points¶

Call	What it does
`kglite.open(path)`	Load-or-create. Opens the graph at `path` if it exists, creates a fresh one bound to `path` if it doesn’t. The database-style entry point.
`kglite.load(path)`	Load an existing `.kgl` file (or disk-mode directory). Raises `kglite.FileError` if missing, `kglite.FileFormatError` if corrupt (see below).
`g.save(path=None, *, fsync=True)`	Write a full checkpoint, atomically and durably. With no `path`, saves back to the remembered path.
`g.to_bytes()` / `kglite.from_bytes(data)`	Serialize/deserialize the graph to/from a `.kgl` byte buffer — own the write (object storage, a pipe, a checksum) instead of a filesystem path.
`g.close()`	Checkpoint, release persistence ownership, and retain a detached mutable graph. A failed checkpoint keeps ownership for retry.
`with kglite.open(...) as g:`	Checkpoint on clean exit; skip checkpoint on exception. Both detach the retained graph. Already committed WAL writes remain recoverable.

After close() or context exit, the retained graph keeps its data, query configuration and CDC stream, but no remembered path, WAL writer or writer lease. It remains queryable and mutable; those later writes are private. save() without a path refuses, while save(path) explicitly persists the detached snapshot using the existing unlocked snapshot-save contract. Held read snapshots remain readable without blocking the next writer. Transactions begun under the ended owner cannot commit their writes back into it; use a new transaction on the detached graph or reopen the database for further durable work.

Every save() is atomic and torn-proof, even in non-durable mode: it writes to a sibling temp file and atomically renames it over the target, so a crash mid-save can never leave a half-written .kgl — a reader always sees the old file or the complete new one. With fsync=True (default) the file and its directory are flushed to physical storage before returning; pass fsync=False to skip that flush for speed in a hot loop (still atomic). This removes the temp-file + os.replace + dir-fsync dance consumers used to hand-roll.

Corrupt-file detection is typed. load() / from_bytes() raise kglite.FileFormatError (a subclass of kglite.KgError) on a corrupt, truncated, or wrong-format input, and kglite.FileError on a missing file — so a disposable-cache consumer can branch “corrupt → rebuild from source” vs “missing → create new” cleanly, without a broad except IOError.

Detection does not rely on the damage happening to be structurally invalid. Every section of a .kgl — topology, each node type’s columns, embeddings, time series, secondary labels, vector index — is written with a checksum and verified on load, so a bit that flips in storage or transit produces an error naming the damaged section rather than a graph that loads with quietly different data. Files written by older versions lack the checksums and load unchanged; files written by this version load on older versions.

The thread that ties these together is the remembered path: open() and load() record where the graph came from, so a later bare save() — or the context manager’s auto-save — writes back without you re-specifying the target.

import kglite

# First run: file doesn't exist → fresh graph, bound to "app.kgl".
with kglite.open("app.kgl") as g:
    g.cypher("CREATE (:Person {id: 1, name: 'Alice'})")
# clean exit → auto-saved to app.kgl

# Next run: file exists → loaded back.
with kglite.open("app.kgl") as g:
    g.cypher("CREATE (:Person {id: 2, name: 'Bob'})")
    print(g.cypher("MATCH (p:Person) RETURN count(p) AS n").scalar())  # 2

The default: crash-safe¶

open() is durable by default. Every committed mutation is fsync’d before the call returns, so a mutation that has returned survives a hard crash.

g = kglite.open("app.kgl")
g.cypher("CREATE (:Order {id: 1001, total: 49.90})")   # fsync'd before this returns

You get this without asking for it because it is what makes an embedded database trustworthy: the alternative default silently loses every write since the last explicit save() whenever a process dies.

The default is the strongest level, not the only one. If power-loss safety is more than your application needs, durable="normal" keeps the log and drops only the per-commit barrier — see Choosing a durability level.

Choosing a durability level¶

durable names what a committed mutation survives. It uses SQLite’s synchronous vocabulary, and the levels are stated as guarantees rather than as syscalls — the syscall differs by platform, the guarantee does not.

`durable=`	A committed mutation survives…	Per-commit cost
`"full"` (or `True`) — default	process crash, OS crash, power loss	one barrier
`"normal"`	process crash — `kill -9`, panic, OOM-kill	no barrier
`"off"` (or `False`)	nothing since the last `save()`	no log

True and False are accepted spellings of "full" and "off", so existing code keeps working unchanged.

`"normal"` — the process-crash level¶

g = kglite.open("app.kgl", durable="normal")
g.cypher("CREATE (:Order {id: 1001, total: 49.90})")   # logged, not barriered

The frame is handed to the kernel with a plain write before the call returns. The page cache belongs to the kernel, not to your process, so the commit survives your process dying by any means — an uncaught exception, kill -9, the OOM killer. What it does not survive is the kernel dying: an OS crash or a power cut loses commits made since the last save().

That is the right trade for most applications, because a crashing process is the failure that actually happens and a power cut is the one you keep backups for. It is also the level to reach for when per-commit barrier latency is shaping your write throughput — "normal" writes the same log frame and skips only the barrier.

`"off"` — no log at all¶

When the graph is rebuildable from source data — a bulk load, a derived index, a scratch analysis — logging buys nothing:

g = kglite.open("kb.kgl", durable="off")
g.add_nodes(df, node_type="Topic", unique_id_field="id")   # nothing logged
g.save()          # one explicit checkpoint at the end
g.close()

What that is not: crash-safe. A snapshot is written only when you call save()/close() or the context manager exits cleanly. If the process is killed mid-session (kill -9, power loss, an unhandled crash before the next save()), the work since the last checkpoint is gone.

Taking a power-safe point on demand: `sync()`¶

"normal" skips the per-commit barrier — but you can take that barrier whenever it matters:

g = kglite.open("app.kgl", durable="normal")

def handle_request(payload):
    g.cypher("CREATE (:Event $props)", params={"props": payload})

handle_request(...)
g.sync()      # everything committed so far now survives power loss too

sync() writes no checkpoint and truncates nothing — it only makes the existing log durable, which is why it is the right granularity for “flush at the end of a request” or “flush before shutdown”. A full save() republishes the entire graph and is far more expensive.

Under "full" it returns immediately (every commit was already barriered). On a graph with no log it raises ValueError rather than silently doing nothing, because a caller who believes they bought power-safety and got nothing is the failure that costs data.

How durability works¶

With the default durable="full", every committed mutation is appended to a <path>-wal sidecar and barriered to stable storage before the call returns. durable="normal" writes the log without that per-commit barrier; use sync() when a power-safe point is required.

How it fits together:

Each mutation → one WAL frame, written before the call returns. Under "full" the frame is also barriered to stable storage per commit; that barrier adds device latency to the engine and WAL-encoding work (see “Cost and tuning” below). Under "normal" the same frame is encoded and written but not barriered.
save() → writes a full checkpoint (.kgl) and truncates the WAL. The checkpoint is the new baseline; the WAL starts empty again.
open(...) → loads the last checkpoint, then replays any WAL frames written since it, reconstructing the exact committed state — including work that was never checkpointed because the process crashed.

So the on-disk state is always “last checkpoint + replayable tail”, and reopen folds the two back together automatically.

WAL format compatibility¶

The current writer uses WAL format 4; the reader decodes formats 2–4. Format 4 records complete node state and the complete property-map multiplicity of each affected parallel-relationship group. This changes the WAL sidecar, not the .kgl checkpoint format. A readable older WAL header is upgraded before new frames are appended. Readers limited to formats 2/3 refuse the format 4 header, rather than interpreting unfamiliar frames as a torn tail.

Legacy formats 2/3 lack a discriminator for individual parallel relationships. If a surviving legacy edge action matches multiple checkpoint relationships, recovery refuses the ambiguity instead of choosing one. A later format 4 group snapshot can supply the complete final state. This does not reconstruct parallel members already lost by an earlier replay or information absent from the old log. Unambiguous legacy operations remain readable.

Durable adoption also refuses duplicate exact (primary type, id) identities before taking ownership or changing the WAL. This admission check does not change the existing Cypher CREATE identity policy.

A surviving uncheckpointed frame that stores a raw legacy endpoint reference is also refused before replay, graph mutation, torn-tail repair, or WAL truncation. Unlike a complete checkpoint, a WAL frame has no originating graph view from which a physical slot can be resolved safely. Frames at or below the checkpoint LSN are already represented by the complete snapshot and are skipped as residue. Move the refused sidecar aside only when deliberately discarding those uncheckpointed commits; otherwise recover with a compatible older build and write a clean checkpoint first. KGLite does not guess the former target.

Crash recovery in practice¶

import os

# Process A — commits, then dies hard before any save().
g = kglite.open("app.kgl", durable=True)
g.cypher("CREATE (:Person {id: 1, name: 'Alice'})")   # committed + fsync'd
g.cypher("CREATE (:Person {id: 2, name: 'Bob'})")     # committed + fsync'd
os._exit(1)   # hard crash — no save(), no clean close

# Process B — reopen recovers both, from the WAL.
g = kglite.open("app.kgl", durable=True)
assert g.cypher("MATCH (p:Person) RETURN count(p) AS n").scalar() == 2
g.save()   # checkpoint: fold the WAL into a fresh .kgl, truncate the log

Both rows survive the crash even though save() was never called in process A — they were fsync’d to the WAL at commit time, and reopen replayed them.

Choosing the mode¶

KGLite has three persistence postures for an embedded app. Pick by what you’re optimising for:

You want…	Use	Trade-off
Every committed write to survive a hard crash	`open(path)` (the default)	One barrier per commit; reopen is O(graph) (loads the whole graph).
Committed writes to survive a crashing process, cheaply	`open(path, durable="normal")`	No barrier per commit; an OS crash or power cut loses work since the last `save()`. Call `sync()` for a power-safe point.
Maximum write throughput on rebuildable data	`open(path, durable="off")`	Nothing logged; a crash loses work since the last checkpoint.
Crash safety on a graph that outgrows RAM	`open(path, storage="mapped")`	Same per-commit WAL guarantee; property columns spill to mmap. The `.kgl` records the mode, so every later `open(path)` comes back mapped without repeating the argument — and passing `storage="mapped"` on a memory-saved graph converts it. See Choosing a storage mode.
100 M+ nodes (Wikidata-scale), cheap cold-open	`open(path, storage="disk")`	Paged mmap, lazy load; no per-commit WAL — durability is your `save()` calls.

The first three are in-memory — the whole graph lives in RAM, which is what makes traversal and multi-hop queries fast. Durability adds crash-safety on top of that model without changing the in-memory read path.

If your app is simply growing, reach for mapped, not disk. mapped is the larger-than-RAM mode that keeps this guide’s guarantee: it is durable by default and its crash recovery is kill-9 tested alongside in-memory. disk is a different trade — a Wikidata-scale exploration mode whose commit boundary is an explicit save(), not a logged write (see the Limitations below). Choosing disk because a graph got big means giving up per-commit crash safety you did not have to give up.

If cold-open latency is what hurts, the storage mode is the lever — not the log. Reopening a .kgl decodes its whole payload before the first query answers, and that cost scales with the file. It is one serialized payload rather than an addressable layout, so there is nothing to defer; and a mapped .kgl is read back by deserializing into a memory backend and then swapping it onto the mapped one, which is why a mapped reopen costs what a memory reopen costs. storage="disk" is the only mode that changes that shape: a disk directory is already in its query-ready layout, so opening it maps files instead of decoding a payload — measured 6.5x faster to reopen at ~400k edges, and an external evaluation measured ~28x at 10.5M, so treat the small number as the floor rather than the rate. The price is the one above: no per-commit WAL, durability is your save() calls. Keeping the log and checkpointing regularly bounds replay, which is a different cost from decoding the checkpoint — see Cost and tuning.

Serving concurrent reads¶

A KnowledgeGraph is single-owner — don’t share one instance across threads while a thread mutates it (that raises a clear RuntimeError). For a read-heavy server, take an immutable snapshot with g.freeze() → a FrozenGraph that shares the data via an O(1) clone and serves cypher() from many threads at once, lock-free. When the data changes, build/reload, freeze() again, and swap the snapshot in. See Concurrency for the full model.

snapshot = g.freeze()
# hand `snapshot` to N reader threads — concurrent, lock-free
snapshot.cypher("MATCH (o:Order) RETURN count(o)")

Durability and shared concurrent writes don’t combine in one handle. A Session (graph.session() / kglite.open_session(...)) serves shared reads and serialized writes, but its execute() writes land on a working copy visible only through that session — reachable by neither the log nor the owning graph’s save(). A Session write against a durable graph therefore raises, rather than applying a mutation nothing can persist; reads are unaffected. For a durable app, keep writes on the durable KnowledgeGraph itself (there they are serialized and fsync’d) and use freeze() snapshots for concurrent reads. After the source ends ownership, a retained Session without CDC may write to its private state; those writes are not logged to the former source path. Sessions sharing a CDC stream remain read-only because their independent mutations must not appear as changes to the unchanged source graph. Use the source KnowledgeGraph for captured writes. Reach for Session writes without WAL/CDC when you need shared concurrent writes but not persistence or capture authority. See Concurrency for the full model.

Cost and tuning¶

"full" adds a barrier per commit. Device latency can dominate small writes, but it is only one component of their cost. WAL encoding scales with the affected node state and the complete parallel-relationship groups sharing the same type, source and target. Updating one member records the final property maps of every member in that group; finding them also traverses adjacency. Ordinary reads do not write or barrier the WAL.
"normal" skips only the barrier. It retains the same state capture, encoding and log writes as "full", including the cost of large parallel groups. It preserves process-crash recovery; sync() supplies a power-safe point when needed.
On macOS, "full" buys more than SQLite’s default does. KGLite’s barrier is F_FULLFSYNC, which flushes the drive’s own write cache; SQLite’s default synchronous=FULL issues a plain fsync, which on macOS does not. The guarantees are therefore not the same thing measured differently — KGLite’s default is the stronger one, and it costs accordingly.
Batch where you can. One cypher() that creates 1,000 nodes is one fsync; 1,000 separate cypher() calls are 1,000 fsyncs. Group related mutations into a single statement (or a transaction — see Transactions and sessions) when they logically commit together. Within a committed batch, each affected relationship group contributes one final snapshot, even when several mutations touched it.
Checkpoint to bound recovery time. Reopen replays every WAL frame since the last save(). Replay is fast (frames are folded into net per-entity state and the index rebuilt once), but a periodic save() keeps the WAL short and recovery near-instant for write-heavy, rarely-restarted services.

Limitations¶

Not available for storage="disk". A disk graph commits by publishing an immutable generation, so its durability boundary is that publish rather than a logical log; reconciling a replayed frame against a published generation needs a generation-aware log this release does not have. open(path, storage="disk") therefore opens non-durable, and both durable="full" and durable="normal" raise ValueError rather than pretending — the levels are not uniform across storage modes, because the blocker here is the commit boundary itself and not barrier strength. storage="disk" supports only durable="off". The in-memory default and storage="mapped" support every level — if you want crash safety on a graph that outgrew RAM, mapped is the answer.

What disk mode does guarantee is worth stating exactly, because it is stronger than “no crash safety” and is kill-9 tested (crates/kglite/tests/disk_crash_guarantee.rs):

A crash loses exactly the mutations made since the last save(), and nothing else. The graph reopens at the last published generation, complete and uncorrupted — never at a partially-written one.

Between save() calls, disk-mode mutations live only in the process’s heap overlay; nothing is written, so nothing can be half-written. The publish itself is crash-atomic: the staged snapshot is fsync’d, renamed into place, and only then does an atomically-replaced CURRENT pointer select it, so a crash mid-publish leaves the previous generation selected. No acknowledged commit is ever lost — the acknowledgement point is your save() call.

Budget for the checkpoint’s cost before you sprinkle save() calls. Every disk save() writes a complete new generation and the superseded ones are retained, so N checkpoints leave N full copies of the graph on disk. That is deliberate — readers hold their generation mmap’d and must not have it deleted underneath them — but there is no retention policy yet, so checkpoint on a schedule you have the disk budget for, and prune old generations/gen_* directories yourself once no reader is using them.
What the log carries. Nodes, edges, labels; every declaration you make about them — the identity-field spellings an add_nodes call names (unique_id_field / node_title_field), set_parent_type, define_ontology / clear_ontology, create_index / drop_index and their range and composite siblings, CREATE CONSTRAINT / DROP CONSTRAINT, set_spatial, set_schema_version; and the two bulk payloads — timeseries channels (set_timeseries, set_time_index, add_ts_channel, add_timeseries, add_nodes(timeseries=…)) and embeddings (set_embeddings, add_embeddings, embed_texts, import_embeddings, copy_embeddings_from, and the build_vector_index declaration). A recovered graph is queryable by your own column names, with your indexes built, your constraints enforced, your series readable and your vectors searchable — carrying the model id and per-node text hashes, so a following embed_texts(mode='changed') re-embeds nothing. What replay does not restore is derived state a load rebuilds anyway; the HNSW topology is rebuilt from the replayed vectors rather than logged, because it addresses store slots that replay renumbers.
Bulk payloads make bulk frames. A timeseries load logs the series it produced, so a 10 000-node × 365-day × 3-channel add_timeseries writes one frame of roughly 129 MB, assembled in memory before a single write. That is ~3.6× cheaper per source row than the node rows the log already carries, and far inside the 4 GiB frame cap, but it is a real transient cost on a Wikidata-scale ingest. durable="off" (or loading before open(…, durable=…)) skips it.
A with block is not a transaction. Each mutation commits as it runs, so an exception inside the block does not undo mutations that already returned — they are recovered on the next open(). Use begin() when you want discard-on-error (Transactions and sessions).