Hybrid Search

engrava’s search_hybrid() combines up to five scoring signals into a single ranked result list.

Signal Model

Default weights sum to 1.0. When a signal is unavailable (e.g. neither recency reference nor a cycle provider can resolve recency, no embeddings → vector skipped), its weight is redistributed proportionally across active signals.

The table’s defaults apply on every construction path. SqliteEngravaCore(conn, ...) with no search_config argument resolves them exactly like a store built with SearchConfig() or with from_config() and no override: each reads SearchConfig’s own field defaults. An explicit per-call weight, or a SearchConfig built with a different value, still wins. search_reflections_only() takes no per-call recency_weight override.

Keyword query syntax (FTS)

The keyword signal — and the search_fts() method that exposes it directly — runs your text against an SQLite FTS5 index. engrava normalises the query before handing it to FTS5, with two modes that switch automatically on what you type:

Bare queries are matched with OR. A plain natural-language query like what was my sister doing is treated as a bag of words joined with OR, so a document matches when it shares any word. BM25’s IDF weighting then ranks the documents that share the most distinctive words first, so common function words (what, was, my) carry little weight and need no stopword list or stemmer — this works in any language.

# Bare query -> OR-matched: finds docs sharing any content word, best-ranked first
hits = await store.search_fts("what was my sister doing", top_k=10)

Expert syntax is preserved unchanged. If your query uses FTS5 operators, it is passed through as written:

• quoted phrases — "machine learning" matches the exact phrase;
• uppercase booleans — AND, OR, NOT (must be uppercase) compose terms, e.g. python AND NOT snake;
• prefix — a trailing * does prefix matching, e.g. neur*;
• column filters — essence: and content: restrict a term to that column, e.g. content:berlin.

Punctuation never raises. Unsafe characters split a token into separate terms rather than breaking the query: a contraction like sister's becomes sister OR s, so it still matches a stored sister's dog. Pasting a URL or a timestamp is safe too — only the real essence: / content: column filters are honoured, so http://example.com and 12:30 are treated as ordinary search terms. When a normalised full-text expression is a genuinely malformed FTS5 query, engrava logs a warning, increments the read-only fts_match_failure_count counter, and retries once through the bare normalisation (unsafe characters dropped, wildcards collapsed to legal prefixes, any exposed AND/OR/NOT phrase-quoted so FTS5 cannot read it as an operator), which is always a valid MATCH; the FTS arm returns that query’s matches — an empty set when the sanitised query matches nothing.

Graceful Degradation

• FTS5 unavailable or empty query → FTS skipped.
• query_vector is None and no embedding provider → vector skipped.
• No explicit current_cycle, no configured cycle provider, and no recency_now → recency skipped.
• priority_weight is 0.0 → priority skipped.
• graph_weight is 0.0 → the weighted graph signal is skipped (see the Performance note on graph_expansion_enabled for what this does and does not stop).
• All signals disabled → a query-less window of up to top_k rows ordered by updated_cycle, honouring the same filters and archived exclusion as the arms. With no cycle reference, every row scores a flat 0.0. A reference supplied at recency_weight=0.0 — current_cycle or recency_now — is accepted but inert here too: the weight gates recency on this query-less path the same way it does everywhere else, so the row still scores a flat 0.0 rather than a decayed one. When only FTS and vector are unusable, that same path pre-orders the window by whichever recency axis is still active.

Two recency axes

Recency ranks along two separately-typed axes, and a query picks exactly one. Passing neither explicit reference uses a configured cycle provider when one is present; without one, recency stays off.

Both use the same exponential half-life decay; only the clock differs.

Why the second axis exists. A consumer with no cadence writes everything at cycle 0, where the cognitive-cycle axis is degenerate: every row has the same age, so recency ranks nothing. The transaction-time axis makes “recently written” rankable on exactly that store — the common case for externally-written memories.

# Rank by how recently each memory was written, relative to a caller-supplied
# "now", with a 24-hour freshness half-life.
from datetime import UTC, datetime

result = await store.search_hybrid(
    "incident timeline",
    recency_now=datetime.now(UTC).isoformat(),   # the caller owns "now"
    recency_weight=0.4,
    recency_now_half_life=86400,                 # seconds; default 604800 (7 days)
)

The rules worth knowing before you wire it:

• The caller owns “now”. recency_now is a caller-supplied ISO-8601 instant; engrava’s core reads no wall clock when ranking, so retrieval stays deterministic and replayable — the same store plus the same recency_now yields the same ranking.
• Naive values are read as UTC. The host timezone is never consulted.
• Malformed input raises — once the axis is on. With a recency_now given, a malformed instant, or a non-positive recency_now_half_life, raises InvalidRecencyArgumentError. Both checks sit behind recency_now, so a recency_now_half_life passed without one is not validated and not applied: it is silently ignored, along with the axis it was meant to tune.
• A row with no usable timestamp scores minimum. If its updated_at / created_at is missing or malformed it is treated as maximally old rather than dropped.
• Explicit beats passive. An explicit recency_now takes precedence over a configured cycle_provider — the provider is not consulted. Supplying both explicit references, current_cycle and recency_now, raises RecencyModeConflictError.
• The units never mix. A wall-clock age is never subtracted from a cognitive cycle. To use transaction-time recency on a store that has a cycle provider, pass recency_now and omit an explicit current_cycle.

Archived thoughts leave default retrieval

Thoughts whose lifecycle_status is ARCHIVED are excluded from the default retrieval candidate set — on search_hybrid(), recall(), search_fts(), search_similar(), and the query-less fallback alike. This is an eligibility filter in the same class as the existing expired-row filter, not a scoring change: a thought you archived stops surfacing without being deleted. This is the retrieval side of Forgetting — an opt-in, off-by-default hygiene loop — but the exclusion applies to any archived row, whether or not that loop is enabled.

# Search the archive for this call only — the rows stay ARCHIVED.
result = await store.search_hybrid("old incident", include_archived=True)

# Or bring one back for good.
restored = await store.restore_thought(thought_id)

Two boundaries are worth knowing before you rely on it:

• It is not a global filter, but it is not lifecycle-blind either. list_thoughts() and count_thoughts() are not ranked retrieval, so they return an archived row by default — unless that row is also expired, in which case the same default expiry exclusion everything else gets applies to it too; pass include_expired=True to include an archived-and-expired row there. Filter on lifecycle_status= yourself to exclude archived rows from these two entirely.
• It does not cover retired reflections. A REFLECTION retired because its whole source cluster left ACTIVE is held out by a separate freshness rule, so it stays excluded even under include_archived=True.

See Data Lifecycle for the full archive-vs-delete mechanics.

Graph-Aware Ranking

The graph signal uses 1-hop-weighted neighbour boost. If a candidate thought’s graph neighbours also match the query, the candidate receives a boost proportional to the neighbour’s semantic score and the connecting edge weight.

Algorithm

candidate_scores = { C: max(fts_score[C], vector_score[C]) for C in fusion pool }
For each candidate C in the fusion pool:
  neighbours = top max_neighbors outgoing/incoming edge matches, ranked by
               edge.weight DESC, then edge.edge_id ASC
               # a self-loop matches once in each direction
  For each (edge, neighbour):
    neighbour_base = candidate_scores.get(neighbour, 0.0)  # 0 if neighbour is off-pool
    boost[C] += edge.weight * neighbour_base * graph_edge_decay
final_score[C] += graph_weight * boost[C]

A neighbour outside the fusion pool contributes 0, not its own fts/vector score — only pool members’ base scores propagate.

Key properties:

• Only semantic scores propagate — priority, recency, and graph scores are excluded from neighbour_base to prevent hub-cascade effects.
• No new candidates — graph signal re-ranks existing results; it does not add thoughts to the result set.
• Capped selection — outgoing and incoming matches are ranked by edge.weight DESC, then edge_id ASC, before the max_neighbors cap is applied; a self-loop contributes one match in each direction.

Configuration

search:
  default_graph_weight: 0.0          # opt-in (0.0 = disabled)
  graph_edge_decay: 0.5              # decay factor for 1-hop distance
  max_neighbors_per_candidate: 5     # safety cap

Per-query override:

result = await store.search_hybrid(
    "python async",
    graph_weight=0.1,
    graph_edge_decay=0.3,
)

Performance

graph_weight=0.0 (default) disables the weighted graph signal above — it does not by itself guarantee zero graph queries, because graph_expansion_enabled (on by default) separately triggers a one-hop expansion query whenever the fused pool contains at least one top-ranked REFLECTION candidate. Set graph_expansion_enabled=False too if you need to guarantee no graph queries run. When the weighted signal is active, the candidate pool it runs over is the fused fts_top_k + vector_top_k semantic-base set (default 50 each, and each raised to top_k when smaller, so it can be far larger than the final top_k; passing collapse_key then multiplies both pools by collapse_pool_factor — default 4 — before this pool is built), queried in 450-wide chunks (SQLite’s bound-parameter ceiling makes one query per chunk necessary once the pool is large). Each chunk’s query itself is bounded in SQL to at most max_neighbors_per_candidate rows per candidate returned — via a ROW_NUMBER() OVER (PARTITION BY candidate_id ...) ranking window over that candidate’s edges, not a limit applied in Python after fetching everything — so the total rows returned across all chunks is bounded by (fts_top_k + vector_top_k) × max_neighbors_per_candidate — up to collapse_pool_factor times larger when collapse_key is set — not the size of the full adjacency. See Performance for the full cost breakdown.

The SQL-side rank window bounds the rows that cross back into Python, not the underlying scan-and-sort cost: SQLite still has to gather and order every matching edge for a candidate inside the query before the per-candidate rank cutoff drops the excess, so a genuinely heavily-connected hub is still expensive to rank at the database layer — the window changes where the truncation happens, not whether the full edge set for that candidate is touched.

Observability

When the graph signal contributes to at least one candidate, "graph" appears in HybridSearchResult.backends_used.

Per-Query Overrides

All weights can be overridden per call via keyword arguments:

result = await store.search_hybrid(
    "quantum computing",
    query_vector=embedding,
    fts_weight=0.4,
    vector_weight=0.4,
    recency_weight=0.1,
    priority_weight=0.05,
    graph_weight=0.05,
    current_cycle=42,
)

Configuration Reference

See Configuration for the full YAML reference of SearchConfig fields.

Querying Reflections

After DreamingExtension.run_consolidation() runs its clustering phase, ThoughtType.REFLECTION meta-thoughts exist in the store. Four controls govern how hybrid search handles them — and one of the four is on by default, so read reflection_topk_cap below before tuning the other three.

include_reflections (default True)

When False, REFLECTION thoughts are excluded from search_hybrid() results. Useful when you want raw observations / insights without higher-order aggregates:

result = await store.search_hybrid(
    "machine learning",
    query_vector=embedding,
    include_reflections=False,
)

reflection_boost (default SearchConfig.reflection_boost = 1.0)

When REFLECTIONs are included, their final score is multiplied by this factor. The default 1.0 leaves REFLECTIONs on equal footing with regular thoughts while scoring; raise it above 1.0 to give high-level abstractions a modest upranking so they surface for broad queries without dominating narrow ones. Equal scoring is not the last word on the result window: reflection_topk_cap runs afterwards and can evict a REFLECTION that scored its way in.

# Stronger boost -- reflections rank near the top for broad queries
result = await store.search_hybrid(
    "patterns in memory",
    query_vector=embedding,
    reflection_boost=1.5,
)

# Disable boost -- reflections compete on equal footing
result = await store.search_hybrid(
    "specific fact",
    query_vector=embedding,
    reflection_boost=1.0,
)

Configure the default in YAML:

search:
  reflection_boost: 1.0   # applies when reflection_boost not overridden per-call

reflection_topk_cap (default SearchConfig.reflection_topk_cap = 0.3)

This one is active unless you turn it off. It bounds the fraction of the final top-K window REFLECTIONs may occupy, and it is applied after fusion, boosting and de-fragmentation — so it reshapes a window the scores had already decided.

At the default 0.3 with top_k=10, REFLECTIONs get at most int(10 * 0.3) = 3 slots. If more than that survive into the window, the lowest-scoring excess are removed and the freed slots are backfilled with off-list non-REFLECTION candidates from just below the cut. The eviction is unconditional: if there are not enough off-list candidates to backfill with, the excess REFLECTIONs still go, the window comes back shorter, and engrava logs a warning.

search:
  reflection_topk_cap: 0.3   # fraction of top_k; 1.0 disables the cap

Unlike include_reflections and reflection_boost, this is configuration only — search_hybrid() takes no per-call override, so a store built without a SearchConfig runs on the 0.3 default. Set it to 1.0 to let REFLECTIONs hold as many slots as they score into.

Every query reports what the cap did:

result = await store.search_hybrid("patterns in memory", query_vector=embedding)
result.reflections_evicted   # 0 => the ranked results are exactly what scoring produced

search_reflections_only()

Convenience helper that returns only REFLECTION thoughts, scored by cosine similarity to the query vector (plus optional recency blend). Designed for queries like “what themes exist in my memory?”:

result = await store.search_reflections_only(
    "recurring ideas about learning",
    query_vector=embedding,
    top_k=5,
    current_cycle=42,   # optional recency blend
)
for thought_id, score in result.results:
    ref = await store.get_thought(thought_id)
    print(ref.content)  # JSON with member_ids + keywords

Key difference from search_hybrid(include_reflections=True): search_reflections_only() fetches all REFLECTIONs directly from the store (no pagination gap) and scores them purely by cosine similarity to the query. It does not compete against regular thoughts for result slots.