search — full-text full-text scan and lexical retrieval

This service is in preview: the API on this page is usable but may still change before it stabilizes.

What it is

Search moves keyword lookup — full-text search, typeahead, "find records matching these words" — out of your application code and into the database, without weakening zero-knowledge (noy-db's core guarantee: the store only ever holds ciphertext, and no server or storage backend can decrypt it): every match is computed client-side, after decryption, and nothing about the query or the matched terms is written to the store.

There are two ways to search by keyword, both covered on this page:

• collection.search() — a zero-setup scan: decrypt the collection and rank matches with BM25 on every call. No index to build or keep warm; the right default for most collections.
• collection.retrieve() — an indexed path: build an in-memory inverted index once (client-side, never written to the store by default), then serve ranked, multi-field hits with snippet extraction. Optionally persist the index as an encrypted blob so it survives across sessions.

collection.retrieve() also has a mode option for searching by meaning instead of exact words ('semantic') or combining both ('hybrid') — those modes need an embeddings config and are documented on the search — AI page, which this page's lexical mode feeds into.

Both search() and retrieve() require eager mode (prefetch: true, the default) — they read the fully-decrypted in-memory cache and throw in lazy mode (prefetch: false).

Prerequisites

• None for search() — it scans whatever field you name at call time.
• For indexed retrieve() / warmIndex(): a textIndexes allowlist declared on the collection — calling retrieve() without one throws.
• For warmIndex(): eager mode (the default); pre-building on a lazy collection throws.
• For semantic / hybrid modes: an embeddings encoder config — see search — AI.

Opt in

Search and retrieval are gated behind a withSearch() capability you pass to createNoydb(). Without it, collection.search(), collection.retrieve(), collection.similarTo(), collection.warmIndex(), and collection.flushIndex() all throw SearchNotEnabledError.

import { createNoydb, withSearch } from '@noy-db/hub'

const db = await createNoydb({
  user: 'alice',
  secret: 'alice-secret-2026',
  searchStrategy: withSearch(),
})
const vault = await db.openVault('books')

interface Invoice { id: string; title: string; statusLabel: string; notes: string; category: string; attachmentName: string }

// search() needs nothing else — it scans whatever field you name at call time.
// retrieve() additionally needs an explicit opt-in allowlist of fields to index:
const invoices = vault.collection<Invoice>('invoices', {
  textIndexes: ['title', 'statusLabel', 'attachmentName'],
})

withSearch() is dynamically imported behind the scenes, so the search/retrieval engine stays out of your bundle until you opt in.

Usage

collection.search(field, query, opts) — scan mode

Ranks records by relevance against a tokenized query, scoring in memory with BM25 on every call.

const hits = await invoices.search('title', 'overdue invoice')
// → [{ id, score, record }, …]  (best first, OR of query terms)

await invoices.search('title', 'overdue invoice', { match: 'all' })
// → same shape, AND of query terms — a record must contain both "overdue" and "invoice"

await invoices.search('title', 'mee', { prefix: true })
// → same shape — the last term ("mee") is matched as a prefix, e.g. "meeting"

await invoices.search('title', 'invoice', { limit: 10 })
// → same shape, capped to the top 10 by score

Scan mode decrypts the collection in memory and scores client-side, so nothing searchable is written to the store — it adds no leakage beyond what reading the collection already does. It's the safe default for sensitive fields (names, notes). Cost is O(n) per query. If the collection also carries large fields you never search, moving them to a satellite collection keeps them off this decrypt path.

Results are descending by score. Deleted or forgotten records never appear. Full option and return-shape reference: see API reference below.

collection.retrieve(query, opts) — lexical index

Builds an in-memory inverted index on first call (or on collection open, if warmIndexOnOpen: true), then scores query terms with multi-field BM25 and returns ranked hits with per-hit snippet extraction. Requires the collection to declare textIndexes (as configured above); calling retrieve() without one throws.

// Multi-field ranked search, over the `invoices` collection from Opt in:
const hits = await invoices.retrieve('overdue invoice')
// → RetrieveHit[]  (best first)
//   { id, score, rank, field, snippet, locale?, record? }

// Prefix autocomplete (typeahead):
const ac = await invoices.retrieve('over', { prefix: true, limit: 5 })
// → same shape, capped to the top 5 hits, last term ("over") matched as a prefix

// Include the full decrypted record:
const detail = await invoices.retrieve('overdue', { includeRecord: true })
// → same shape, each hit additionally carries `record: Invoice`

Full option and return-shape reference: see API reference below.

API reference

collection.search(field, query, opts?)

search<T>(field: string, query: string, opts?: SearchOptions): Promise<SearchResult<T>[]>

Scan-mode BM25 search over one named field, decrypting and scoring in memory on every call — see Usage above.

SearchOptions:

SearchResult<T>:

interface SearchResult<T> {
  readonly id: string
  readonly score: number   // BM25, descending
  readonly record: T
}

collection.retrieve(query, opts?)

retrieve<T>(query: string, opts?: RetrieveOptions<T>): Promise<RetrieveHit<T>[]>

Indexed, multi-field BM25 retrieval with snippet extraction, over the fields declared in textIndexes — see Usage above. Throws if the collection has no textIndexes configured, and throws in lazy mode (prefetch: false).

RetrieveOptions<T>:

RetrieveHit<T>:

interface RetrieveHit<T> {
  readonly id: string       // record key
  readonly score: number    // BM25, descending
  readonly rank: number     // 1-based position, monotonic with score order
  readonly field: string    // winning field (highest per-field BM25)
  readonly snippet: string  // ±window chars around the best match
  readonly locale?: string  // set for i18n-text and dictionary-key hits
  readonly record?: T       // only when includeRecord: true
}

rank matters beyond display order: it's what fuseRetrieval uses to combine ranked lists (from different retrieval modes, or from different vaults) without comparing raw scores across lists that aren't on the same scale.

textIndexes

vault.collection<Invoice>('invoices', {
  textIndexes: ['title', 'notes', 'category', 'attachmentName'],
  warmIndexOnOpen: true,   // auto-build on collection open
})

An opt-in allowlist of field paths (supports nested paths and []-wildcard arrays). Only fields in this list enter the index — collections with no textIndexes pay zero overhead, and retrieve() throws without one.

collection.warmIndex()

warmIndex(): Promise<void>

Pre-builds the lexical index instead of paying the build cost on the first retrieve(). See warmIndex() and warmIndexOnOpen below for its exact behavior in lazy mode and with no textIndexes.

collection.flushIndex()

flushIndex(): Promise<void>

Force-persists the in-memory index immediately. Only meaningful with textIndexPersist: true — a no-op otherwise. See persistence below.

Other exports

• SearchEntry<T> — the { id, record } input-entry shape scan mode builds internally from the eager cache before scoring; not something you construct yourself.
• SearchStrategy — the interface withSearch() implements (search, retrieve, similarTo, warmIndex, flushIndex, embedOnWrite). Advanced/internal: most code never touches it directly, it just passes withSearch() as searchStrategy to createNoydb().
• NO_SEARCH — the disabled-sentinel SearchStrategy every collection uses until you opt in; every method on it throws SearchNotEnabledError.
• tokenize(text) — the standalone scan-mode tokenizer search() uses internally (NFKC-normalize, lowercase, split on Unicode letter/number runs — no Intl.Segmenter, so it does not word-segment Thai/CJK): tokenize('Hello World') → ['hello', 'world'].

Field-type matrix

Combining retrieve() and where():

const textHits = new Set((await invoices.retrieve('overdue')).map((h) => h.id))
const rows = invoices.query().where('amount', '>', 1000).run()
const combined = rows.filter((r) => textHits.has(r.id))

(RetrieveOptions.within does the same intersection in one call — see the table above.)

i18n tokenizer — Intl.Segmenter

The lexical index's tokenizer uses Intl.Segmenter (granularity: 'word'), then NFKC-normalizes, lowercases, and keeps only word-like segments. This correctly segments Thai, CJK, and other scripts written without inter-word spaces — a run of Thai text is split into individual words instead of collapsing into one token.

A custom tokenizer is pluggable via the Tokenizer type:

import type { Tokenizer } from '@noy-db/hub'
// Tokenizer = (text: string) => string[]

search()'s scan-mode tokenizer is simpler — NFKC-normalize, lowercase, split on Unicode letter/number runs, no Intl.Segmenter — so for Thai/CJK fields prefer retrieve().

warmIndex() and warmIndexOnOpen

The index is lazy by default — built on the first retrieve() call. To pre-build it:

// On-demand pre-build:
await invoices.warmIndex()

// Auto-build on collection open (set in collection options):
vault.collection<Invoice>('invoices', {
  textIndexes: ['title'],
  warmIndexOnOpen: true,
})

warmIndex() on a lazy collection (prefetch: false) throws — pre-building requires eager mode. warmIndex() on a collection with no textIndexes configured is a silent no-op instead: there's no index to build, so it returns immediately without throwing and without building anything.

Session-local by default, or persisted with textIndexPersist

The lexical index is session-scoped and in-memory unless you opt into persistence:

• It's built from the already-decrypted eager cache — no extra store reads.
• By default it's never written to the store — zero added leakage.
• On a new session it rebuilds on the first retrieve() / warmIndex() call.
• A write (put / delete) marks the index dirty; the next retrieve() triggers a full rebuild.

Set textIndexPersist: true to keep the index warm across sessions and devices instead of rebuilding it every time:

const persisted = vault.collection<Invoice>('invoices', {
  textIndexes: ['title', 'notes'],
  textIndexPersist: true,   // persist the index as an opaque encrypted blob
  warmIndexOnOpen: true,    // combine with warm load for an instant first retrieve()
})

With textIndexPersist: true:

• The in-memory index is serialized and encrypted under the collection's DEK — the same key that encrypts every record body. The store sees one opaque ciphertext blob at _ftindex/<collection>: no terms, postings, or plaintext, and no per-term addressability the way a store-side term index would have.
• On the next retrieve() / warmIndex() in a new session, the blob is loaded and checked against a fingerprint ({ count, maxVersion } derived cheaply from the eager cache: record count, and the highest _v version stamp across all records). A matching fingerprint means the blob is used as-is — a cold load with no re-tokenizing. A mismatch (a write happened on another device or tab) triggers a rebuild and re-persist.
• Writes mark the index dirty and schedule a debounced flush (about one second after the last write by default) — typically one blob write per burst of writes, not one per record.
• flushIndex() force-persists the index immediately — call it before saving a checkpoint or before a known idle window:

  await invoices.flushIndex()
  

• Close-flush: db.close() triggers a best-effort flush of any pending dirty index, so the next session opens against the freshest blob.
• forget() teardown: vault.forget(subject) crypto-shreds matching records and also deletes the _ftindex blob, because the blob is encrypted under the same collection DEK and would otherwise still decrypt the forgotten subject's indexed terms. The next retrieve() rebuilds from the post-forget record set. If the blob couldn't be deleted (for example, the store is offline at shred time), that's reported as residue in the forget() return value — the same residue-tracking pattern used for the record-level index side-cars.

Persistence adds no leakage beyond the size of that one blob: the store can't distinguish an index write from a record write by its content, the fingerprint itself lives inside the encrypted envelope, and a collection with textIndexPersist: false (the default) writes nothing extra at all.

Limits & gotchas

See also

• search — AI — semantic (mode: 'semantic') and hybrid (mode: 'hybrid') retrieval, similarTo(), and fuseRetrieval(), all part of the same withSearch() opt-in.
• Lazy — the prefetch: false mode that search(), retrieve(), and warmIndex() all throw under.
• Satellites — split large non-searched fields off the decrypt path that scan mode reads.
• Forget — the erasure path this page's persisted-index teardown plugs into.
• Enclave — the collection DEK that encrypts the persisted index blob.
• Proven by showcases 111-scan-search, 122-with-retrieve, and 123-persisted-index.