indexing

What it does

A collection is a set of encrypted records in a noy-db vault, and query()/lazyQuery() are its chainable, in-hub read builders — .where(field, op, value) filters records (including in-list membership tests), and .orderBy(field) sorts them, all running after decryption, never against the raw store. The indexing service adds a secondary-index layer under those builders: a fast lookup structure keyed by one or more record fields, so .where(), in-list filters, and .orderBy() don't have to touch every record. Without an index, a query is a linear scan over the collection's in-memory cache (eager mode) or a walk of the backing store (lazy mode); with an index declared on the field, the same query becomes an O(1) hash lookup or an O(log n) sorted-range scan.

Where the index lives depends on the collection's hydration mode:

• Eager-mode collections (prefetch: true, the default) hold the index as an in-memory mirror. It's built from the cache on load and updated incrementally on every put/delete; it never touches the store, and rebuilds itself from the record cache on restart.
• Lazy-mode collections (prefetch: false) can't keep the whole collection in memory, so the index has to be durable: each indexed field gets one _idx/<field>/<recordId> side-car per record — an extra encrypted entry, stored next to the main record, holding just that field's value. lazyQuery() reads these side-cars to resolve a candidate set of record ids without decrypting the whole collection.

Because side-car writes are a separate store call from the main record write, they can drift out of sync if the store is ever modified out-of-band (a restored backup, a manual edit). reconcileIndex() and the reconcileOnOpen option exist to detect and repair that drift — see Usage below.

When you need it

• A collection with more than ~5,000 records
• Queries that filter by stable fields (status, region, clientId)
• orderBy on a hot field
• Lazy-mode collections (memory-bounded; query() won't work without an index — use lazyQuery())

Prerequisites

• None standalone — withIndexing() plus per-collection indexes declarations accelerate eager queries.
• For lazyQuery() on a lazy collection: lazyStrategy: withLazy() must also be passed, and every field the query touches must be declared in indexes — see lazy.

Opt in

import { createNoydb } from '@noy-db/hub'
import { toMemory } from '@noy-db/to-memory'
import { withIndexing } from '@noy-db/hub/indexing'

const db = await createNoydb({
  store: toMemory(),
  user: 'alice',
  indexingStrategy: withIndexing(),
})

Declare which fields to index per collection:

const invoices = vault.collection<Invoice>('invoices', {
  indexes: ['clientId', 'status', 'date'],
})

indexes accepts a field name (single-field index), or { fields: [...] } / a plain array (a composite index over the field tuple, in order).

Usage

Backfill after a bulk import. rebuildIndexes() throws away whatever index state exists and rebuilds it from the current records — the in-memory mirror in eager mode, every _idx/<field>/<recordId> side-car in lazy mode. Use it after adding a new indexed field to a collection that already holds data, or to recover from widespread drift:

// 10,000 invoices were imported before `indexes: ['clientId']` was declared.
await invoices.rebuildIndexes()
// → resolves once every side-car (lazy mode) or the in-memory mirror
//   (eager mode) reflects the current records

const paid = await invoices.lazyQuery()
  .where('clientId', '==', 'c-42')
  .orderBy('date', 'desc')
  .limit(20)
  .toArray()
// → [
//     { id: 'inv-118', clientId: 'c-42', date: '2026-06-30', amount: 4200, status: 'paid' },
//     { id: 'inv-091', clientId: 'c-42', date: '2026-05-14', amount: 1800, status: 'paid' },
//     …
//   ]

Detect drift without repairing it. reconcileIndex(field, { dryRun: true }) compares every _idx/<field>/* side-car against the live records and reports what's out of sync, without writing anything:

const report = await invoices.reconcileIndex('clientId', { dryRun: true })
// → {
//     field: 'clientId',
//     missing: ['inv-042'],              // indexable, but no side-car exists
//     stale: ['_idx/clientId/inv-999'],  // side-car points at a vanished/changed record
//     applied: 0                         // dryRun: true never writes
//   }

Call it again with { dryRun: false } (the default) to write the fixes; applied then reports how many writes actually happened.

Auto-reconcile on open. Rather than calling reconcileIndex() by hand, a lazy-mode collection can run it automatically the first time its persisted-index mirror loads (its first lazyQuery()/query() call after the vault opens):

const invoices = vault.collection<Invoice>('invoices', {
  prefetch: false,
  cache: { maxRecords: 1000 },
  indexes: ['clientId'],
  reconcileOnOpen: 'auto',
})

db.on('index:reconciled', (e) => {
  console.log(e)
  // → { vault: 'acme', collection: 'invoices', field: 'clientId',
  //     missing: [], stale: ['_idx/clientId/inv-999'], applied: 1, skipped: false }
})

API reference

withIndexing()

function withIndexing(): IndexStrategy

Builds the strategy object that lights up indexing for a Noydb instance. Pass it to createNoydb({ indexingStrategy }) — see Opt-in above. With no indexingStrategy, every collection behaves as described in "Behavior when NOT opted in" below.

collection.lazyQuery()

lazyQuery(): LazyQuery<T>

Builds a chainable, indexed read against a lazy-mode collection (prefetch: false). Every field touched by .where(...) or .orderBy(...) must be declared in indexes. Terminals are .toArray(), .first(), and .count(), all Promise-returning since candidate records are decrypted from the store on demand.

const row = await invoices.lazyQuery()
  .where('clientId', '==', 'c-42')
  .first()
// → { id: 'inv-118', clientId: 'c-42', date: '2026-06-30', amount: 4200, status: 'paid' }

Throws IndexRequiredError if .toArray()/.first()/.count() is called with a filtered or ordered field that isn't declared in indexes; throws synchronously (before any request) if the collection is eager-mode, or if no index is declared at all.

collection.rebuildIndexes()

rebuildIndexes(): Promise<void>

Rebuilds every declared index from scratch — not incremental. See the backfill example under Usage above.

collection.reconcileIndex(field, opts?)

reconcileIndex(
  field: string,
  opts?: { dryRun?: boolean },
): Promise<{ field: string; missing: string[]; stale: string[]; applied: number }>

Lazy mode only. Compares the _idx/<field>/* side-cars for one declared field against the live records and reports (and, unless dryRun: true, repairs) the drift — see the dry-run example under Usage above. Throws in eager mode (there's no durable side-car to drift), and throws if field isn't declared in indexes.

reconcileOnOpen

Auto-reconcile policy for a lazy-mode collection's persisted indexes, evaluated the first time its index mirror loads.

Use 'auto' for unattended long-lived processes (servers, workers); leave it 'off' (or use 'dry-run') for human-operated apps and surface a manual "rebuild indexes" action instead.

Behavior when NOT opted in

• Eager-mode collections: where('field', '==', value) falls back to a linear scan; indexes: [...] is ignored
• Lazy-mode collections: lazyQuery() throws, pointing at @noy-db/hub/indexing
• rebuildIndexes() is a no-op — there is no index state to rebuild
• reconcileIndex() always throws: in eager mode because reconcile only applies to durable side-cars; in lazy mode because indexing isn't enabled

Pairs well with

• joins (always-core, no opt-in needed) — the join planner already gets O(1) per-row lookups from the collection's own primary-key cache, independent of whether withIndexing() is enabled; this service doesn't change join performance
• aggregate — groupBy dispatches through the index when present
• lazy — required for any non-trivial query against a prefetch: false collection

Edge cases & limits

• Side-car writes happen after the main record write succeeds, as a separate store call — they are not part of the same transaction. If a side-car put/delete fails, the main record write still succeeded; the collection emits an index:write-partial event ({ vault, collection, id, action, error }, where error is an IndexWriteFailureError carrying recordId/field/op/cause) instead of throwing. reconcileIndex(field) finds and repairs what that event flagged.
• Side-car drift can also occur if the store is modified out-of-band (a restored backup, a manual edit). Use reconcileOnOpen: 'auto' for unattended servers, 'dry-run' for human-operated apps that surface a "rebuild" button.

See also

• Query basics — the where/orderBy pipeline indexing accelerates
• aggregate — groupBy dispatches through the index when present
• lazy — bounded-memory prefetch: false collections, which lazyQuery() requires an index for
• partitioning — planned query scoping by time range/region/tenant; a different way to avoid scanning the whole collection than a secondary index
• Proven by showcase 11-with-indexing.