Value projection — how RETURN materialises¶

Companion to bolt-implementation.md Phase A.1. Reference for reviewers of the executor / projection code, and for implementers of future bindings (Bolt server in crates/kglite-bolt-server, Arrow exporter, Polars adapter).

This page documents what happens between a Cypher RETURN clause and a Python dict in your hand. It explains the Value enum, the distinction between transient and materialised graph values, the serialised .kgl format, and the contract that bindings consume.

The `Value` enum¶

Defined at crates/kglite/src/datatypes/values.rs. Fourteen variants today (Phase A.1 added five):

Variant	What it carries	Bolt PackStream analogue
`Null`	nothing	NULL (`0xC0`)
`Boolean(bool)`	true / false	BOOLEAN (`0xC2`/`0xC3`)
`Int64(i64)`	signed 64-bit integer	INT (sized)
`Float64(f64)`	IEEE 754 double	FLOAT (`0xC1`)
`UniqueId(u32)`	node-id-style integer	INT (cast to i64)
`String(String)`	UTF-8	STRING (sized)
`DateTime(NaiveDate)`	calendar date (no time-of-day yet)	Struct `0x44` (Date)
`Point { lat, lon }`	2D geographical point	Struct `0x58` (Point2D, srid=4326)
`Duration { months, days, seconds }`	Neo4j-shape calendar duration	Struct `0x45` (Duration)
`NodeRef(u32)`	transient internal handle (see below)	— never crosses the boundary
`Node(Box<NodeValue>)`	materialised node `(id, labels, properties)`	Struct `0x4E` (Node)
`Relationship(Box<RelValue>)`	materialised rel `(id, start, end, type, properties)`	Struct `0x52` (Relationship)
`Path(Box<PathValue>)`	materialised path `{nodes, relationships}`	Struct `0x50` (Path)
`List(Vec<Value>)`	ordered heterogeneous list	LIST (sized)
`Map(BTreeMap<String, Value>)`	string-keyed map (deterministic order)	MAP (sized)

Phase A.1 / C5 bumped the .kgl format from v3 → v4 to mark the enum extension. Old v3 files are rejected by the v4 binary with a clear “rebuild your graph” error — no read-compat path.

`NodeRef` vs `Node` — transient vs materialised¶

Value::NodeRef(u32) and Value::Node(Box<NodeValue>) look similar but serve different roles in the executor:

NodeRef(idx) — a transient internal handle carrying the petgraph NodeIndex. Used by intermediate stages (WITH, UNWIND, collect() inputs) to preserve node identity without cloning property data. Never user-visible; never persisted.
Node(Box<NodeValue>) — a materialised graph value with the full (id, labels, properties) triple. Built at projection time when the executor needs to hand a node value to a consumer: RETURN n, collect(n), nodes(p), the WITH n chain that carries n forward, etc.

The transition happens in evaluate_expression(Expression::Variable) at crates/kglite/src/graph/languages/cypher/executor/expression.rs:

if let Some(&idx) = row.node_bindings.get(name) {
    if let Some(node_value) = materialize_node_value(idx, self.graph) {
        return Ok(Value::Node(Box::new(node_value)));
    }
    // Tombstone path — DELETE-then-RETURN-in-same-query
    return Ok(Value::Node(Box::new(NodeValue { id: idx.index() as u32, labels: vec![], properties: BTreeMap::new() })));
}

The tombstone arm preserves Cypher’s “count-of-matched-rows” semantics across MATCH ... DELETE n RETURN count(n) — the binding survives deletion but materialising the node would return None; the tombstone keeps count(n) non-Null without faking data.

The projection flow¶

End-to-end for MATCH (n:Person {id: 'alice'}) RETURN n:

parser           AST: RETURN Variable("n")
  │
  ▼
planner          MATCH binds n → NodeIndex 42 in result_row.node_bindings
                 RETURN n is a per-row projection expression
  │
  ▼
executor         per row:
  │                evaluate_expression(Variable("n"), row)
  │                  → row.node_bindings.get("n") = Some(42)
  │                  → materialize_node_value(42, graph)
  │                      = NodeValue { id: 42, labels: ["Person"],
  │                                    properties: BTreeMap {...} }
  │                  → Value::Node(Box::new(node_value))
  │                projected.insert("n", Value::Node(...))
  │
  ▼
CypherResult     rows: [[ Value::Node(...) ]]
  │
  ▼
Python boundary  preprocess_values_owned wraps each Value in
                 PreProcessedValue::Plain (single-variant enum
                 post-Phase A.1 / C7a)
  │
  ▼
py_out::value_to_py(py, &Value::Node(node_val)) →
   PyDict { "id": 42, "labels": ["Person"], "properties": {...} }

The materialize_node_value helper (executor/helpers.rs) is the canonical entry point. It’s backend-aware: in memory mode it reads properties via NodeData::property_iter; in mapped / disk modes properties live in the column store, so it walks graph .get_node_type_metadata(node_type) and reads each property via resolve_node_property (which knows the column-aware path). The parametrised tests in tests/test_value_variants.py run every projection assertion 3× (memory / mapped / disk) to keep the behaviours in lockstep.

At the Python boundary¶

crates/kglite-py/src/datatypes/py_out.rs::value_to_py recursively converts the fourteen Value variants to Python objects. The shapes consumers see:

>>> g.cypher("MATCH (n:Person) RETURN n LIMIT 1").to_list()
[{"n": {"id": 42, "labels": ["Person"], "properties": {"id": "alice", "title": "Alice", "type": "Person", "age": 30, "city": "Oslo"}}}]

>>> g.cypher("MATCH ()-[r:KNOWS]->() RETURN r LIMIT 1").to_list()
[{"r": {"id": 0, "start": 42, "end": 43, "type": "KNOWS", "properties": {"since": 2015}}}]

>>> g.cypher("MATCH p = shortestPath((a:Person)-[*]->(b:Person)) RETURN p LIMIT 1").to_list()
[{"p": {"nodes": [...], "relationships": [...]}}]

>>> g.cypher("MATCH (n:Person) RETURN labels(n) AS L LIMIT 1").to_list()
[{"L": ["Person"]}]                  # native list, not '["Person"]'

>>> g.cypher("MATCH (n:Person) RETURN properties(n) AS P LIMIT 1").to_list()
[{"P": {"id": "alice", "title": "Alice", "age": 30, "city": "Oslo"}}]

What changed from pre-A.1: RETURN n used to return Value::String(node.title) — a plain title string. labels(n) used to return a JSON-encoded string '["Person"]' that a Python-side inference hack auto-parsed back into a list (based on the leading [). Both surfaces are now native; the inference hack is deleted.

The lazy ResultView path is preserved: when the planner flags a terminal RETURN as lazy_eligible, per-cell materialisation runs on Python access (cached via Mutex<Vec<Option<Vec<PreProcessedValue>>>>). The cell shape got richer post-A.1 (one Box<NodeValue> per node projection, vs. one u32 pre-A.1) — see the Performance note at the bottom of docs/history/bolt-implementation.md.

In `.kgl` files¶

The .kgl v4 format (Phase A.1 / C5) is a binary container:

[0..4]    Magic: b"RGF\x04"
[4..8]    core_data_version: u32 LE (== 2 for the post-A.1 layout)
[8..12]   metadata_length: u32 LE
[12..N]   JSON metadata (column schemas, section sizes, all config)
[section] topology.zst — graph structure without node properties
[section] columns_<Type>.zst — packed property columns per type
[section] embeddings.zst (optional)
[section] timeseries.zst (optional)

Value serialises via serde with a discriminant tagged by variant position. The order in crates/kglite/src/datatypes/values.rs is intentionally stable for the first 9 variants (Null=0 .. Duration=8) so any future enum changes append at the end (variants 9..=14 today). The format-version + magic-byte bump in C5 makes the break explicit: a v3 binary cannot read a v4 file (unknown discriminants 9..=13 mid-stream), and the v4 binary refuses v3 files outright with a documented error rather than silently misinterpreting.

The tests/test_phase4_parity.py::test_kgl_v3_golden_hash byte-level tripwire fires on any drift in the saved layout; the test_kgl_v3_file_rejected_with_clear_error test pins the hard-break error message.

For binding implementers¶

If you’re writing a binding that consumes CypherResult from Rust — the Bolt server (crates/kglite-bolt-server, Phase C), an Arrow exporter, a Polars adapter, a JNI bridge, anything that reads Vec<Vec<Value>> and produces a downstream shape — your value-mapping layer is responsible for the 15 variants.

The reference table at the top of this page maps each variant to its Bolt PackStream analogue. For other targets:

Arrow / Polars: scalars map to their typed columns directly. List → a LargeList column; Map → a Struct column when the key set is fixed at the column level, otherwise a Map column. Node / Relationship / Path → either a Struct column or a JSON-string fallback (the agent ecosystem expects dict-shaped output; the structured form is preferred where the target supports it).
C ABI (kglite-c, shipped 0.10.3): Cypher result rows serialize as JSON-string blobs via kglite_cypher_result_rows_json. Each binding parses the JSON with its language’s stdlib (Go’s encoding/json, JS’s JSON.parse, etc.) — same row-shape rules apply. A future v2 may add a tagged-union accessor for performance-critical row-by-row consumption; the JSON-at-boundary path is fine for the common-case query sizes. See docs/rust/c-abi.md §7.

The Value::type_name() -> &'static str method (added in C7a, at crates/kglite/src/datatypes/values.rs) returns the canonical PascalCase variant name — useful for binding-side dispatch tables. The impl Display gives a debug-shaped string suitable for log lines and error messages; for wire serialisation, use the per-binding mapping explicitly.

What this is NOT¶

Not the public API surface. That’s kglite::api::* (crates/kglite/src/lib.rs).
Not a serialisation spec for the .kgl format. The versioned binary container and zstd layout details live in crates/kglite/src/graph/io/file.rs; this doc just names the format-version boundaries.
Not a guide to writing new scalar functions. That’s covered inline in crates/kglite/src/graph/languages/cypher/executor/scalar_functions/ — search for the pattern of an existing function and mirror it.

Value projection — how RETURN materialises¶

The Value enum¶

NodeRef vs Node — transient vs materialised¶