derivations

withDerivation() keeps computed data in sync with its source: a deterministic function projects one source record into one or more typed outputs, encrypted the same way as everything else in a vault. It is one of three related derived-data services registered on createNoydb; the other two have their own pages — materialized-views persist the result of an entire query as a live collection, and overlay-views let an operator override specific rows. All three share one cycle detector (checked once when a vault opens) and one rule for internal cleanup: system-issued deletes (tombstones, refresh-driven row removal) bypass your own onDelete guards — the write-time rules a collection can register to reject a delete — so housekeeping never deadlocks a rule you wrote for humans.

What it does

withDerivation lets a vault declare a deterministic function that turns one source record into one or more typed outputs — for example, a PDF source row that derives extracted metadata and searchable text into two separate collections. The derive function runs on plaintext, after the source collection's data-encryption key (DEK) is unwrapped, and every output is re-encrypted with the same DEK before it reaches the store — a storage backend that only ever sees ciphertext cannot reconstruct the derivation graph by listing records.

Reach for it when you're splitting a large record into faster-to-read projections, keeping side data encrypted under the same key as its source, mixing cheap eager outputs with expensive lazy ones, or need a bulk recompute path after a strategy change (vault.deriveAll).

Prerequisites

• None for same-collection derivations — withDerivation() plus the strategy declarations on this page.
• For FK-keyed triggers (triggerBy): the trigger field must be indexed, which requires indexingStrategy: withIndexing() — see indexing.

Opt in

import { createNoydb } from '@noy-db/hub'
import { toMemory } from '@noy-db/to-memory'
import { withDerivation } from '@noy-db/hub/derivations'

const pdfDerivation = withDerivation<Pdf, { meta: PdfMeta; text: PdfText }>({
  source: 'pdfs',
  deterministic: true,
  outputs: {
    meta: { shape: 'record', collection: 'pdf-meta' },
    text: { shape: 'record', collection: 'pdf-text' },
  },
  derive: (pdf) => ({
    meta: { len: pdf.body.length, pages: pdf.pages.length },
    text: { content: extractText(pdf.body) },
  }),
  lifecycle: 'eager', // or 'lazy'
})

const db = await createNoydb({
  store: toMemory(),
  user: 'alice',
  derivationStrategies: [pdfDerivation],
})

Basic usage

Writing a source record fires derive inside the same transaction (lifecycle: 'eager'); the outputs are readable immediately through their own collections:

const vault = await db.openVault('docs')

await vault.collection<Pdf>('pdfs').put('doc-1', {
  id: 'doc-1',
  body: 'Annual report body text',
  pages: 12,
})

await vault.collection('pdf-meta').get('doc-1')
// → { len: 24, pages: 12, _derivedFrom: { source: 'pdfs', sourceId: 'doc-1', sourceVersion: 1, derivedAt: '2026-05-18T00:00:00.000Z', strategyHash: 'sha256-...' } }

await vault.collection('pdf-text').get('doc-1')
// → { content: 'Annual report body text', _derivedFrom: { source: 'pdfs', sourceId: 'doc-1', sourceVersion: 1, derivedAt: '2026-05-18T00:00:00.000Z', strategyHash: 'sha256-...' } }

API

derive(source, ctx) and the read-only vault facade

The derive function receives a second argument: a DerivationContext carrying the same read-only vault facade guards see as ctx.vault. Use it to fetch sibling records without denormalising them onto the source row.

withDerivation<Allocation, { receipt: Receipt }>({
  source: 'allocations',
  deterministic: true,
  outputs: { receipt: { shape: 'record', collection: 'receipts' } },
  derive: async (alloc, ctx) => {
    const payment = await ctx.vault.collection<Payment>('payments').get(alloc.paymentId)
    const bill = await ctx.vault.collection<Bill>('bills').get(alloc.billId)
    return {
      receipt: {
        id: alloc.id,
        paymentId: alloc.paymentId,
        issuedAt: payment!.paymentDate,
        clientId: bill!.clientId,
        appliedAmount: alloc.appliedAmount,
      },
    }
  },
  lifecycle: 'eager',
})

ctx.vault exposes .get(id), .list(), and .query() — no write capability is reachable from the facade. The strategy hash incorporates derive.toString(), so the function source pins the inputs; whatever sibling reads happen inside derive must be deterministic given the same source record (the consumer's responsibility — the hash does not fingerprint sibling-record content).

Available on both lifecycles: the eager dispatch and the lazy resolve-on-read paths both pass the same facade.

sources

sources: ['a', 'b'] declares sibling collections whose writes re-fire this derivation. A write to any declared sibling re-runs derive against the PRIMARY source record at the SAME id (a silent no-op if no primary record exists there). Siblings participate in cycle detection.

withDerivation<Allocation, { receipt: Receipt }>({
  source: 'allocations',
  sources: ['payments', 'bills'],  // sibling writes re-derive at the same id
  deterministic: true,
  outputs: { receipt: { shape: 'record', collection: 'receipts' } },
  derive: async (alloc, ctx) => ({ /* … */ }),
  lifecycle: 'eager',
})

This retires the hand-rolled "poke a sibling" pattern (writing a no-op back to the source to force a re-derive when a referenced record changed).

FK-keyed triggers — reverse-denormalization

sources[] re-fires at the same id. triggerBy re-fires keyed by a foreign key: a write to a PARENT collection fans out to every source record whose FK matches the parent's id. The canonical use is keeping a denormalized field on the children in sync with a parent change.

withDerivation<Sale, { self: Sale }>({
  source: 'sales',
  deterministic: true,
  // A write to buyers re-derives every sale where sale.buyerId === buyer.id.
  triggerBy: [{ collection: 'buyers', on: 'buyerId' }], // `on` = the FK ON the source
  // Self-write: patch ONLY buyerName back onto the sale (field-level provenance).
  outputs: { self: { shape: 'record', collection: 'sales', denorm: ['buyerName'] } },
  derive: async (sale, ctx) => {
    const b = await ctx.vault.collection<Buyer>('buyers').get(sale.buyerId)
    return { self: { ...sale, buyerName: b?.companyName ?? null } }
  },
  lifecycle: 'eager',
})

• Self-write + denorm. A record output whose collection equals the source is a self-write and MUST declare denorm: [...]. Only those fields are merged onto the raw stored record — the rest of the user's record (and any i18n maps) is never clobbered, and no whole-record _derivedFrom tag is added (the record stays user-owned). The self-write edge is exempt from cycle detection; a value-equality guard breaks the recursion: when the patch changes nothing, no write is issued.
• Fan-out cost. The match runs through the FK index when the source collection has an index on on (declared via indexes: [...] with indexingStrategy: withIndexing() enabled — see indexing); otherwise it scans the collection (O(N) — fine for small child sets; index the FK to scale). An optional maxFanout on the trigger entry (default 64) throws DerivationCapExceededError before any write.
• triggerBy.collection must differ from source. Give it either on or match.

Composite matches — match

Since 0.7.0-pre.12 a trigger entry can key on several fields at once (noy-db#1249). match is a list of { from, to } pairs, read as a conjunction: a source record matches when every pair satisfies String(source[to]) === String(written[from]).

// Re-derive every sale for the same buyer AND the same region.
triggerBy: [{ collection: 'buyers', match: [{ from: 'id', to: 'buyerId' }, { from: 'region', to: 'region' }] }]

from: 'id' reads the written record's id, so the on form above is the single-pair special case — on: 'buyerId' normalizes to match: [{ from: 'id', to: 'buyerId' }]. on is unchanged and still supported; nothing that uses it needs rewriting. Both sides of a pair are coerced to strings, and a pair whose value is neither string nor number fails to match rather than throwing.

One declared hop — via

Since 0.7.0-pre.16 a match pair may resolve through an intermediate collection (noy-db#1277), which makes a relationship expressible where the two collections share no field at all:

// Bills carry `entityId`; disbursements carry `clientId`. The client record relates them.
match: [
  { from: 'clientId', to: 'entityId', via: { collection: 'clients', take: 'id', on: 'entityId' } },
  { from: 'cycle', to: 'cycle' },
]

A source matches when the intermediate — the record in via.collection whose via.take equals written[from] — carries a via.on equal to source[to].

The alternative to a hop is a denormalised key on one side. That is a second copy of something the vault can already resolve, and a partial backfill goes quiet on exactly the oldest, least-audited rows — the silent staleness match exists to remove, handed back one layer down.

Cost and edges: hop resolution is one lookup per written record (take: 'id' is a direct get), never one per candidate row. A dangling hop matches nothing rather than throwing. maxFanout caps the hop fan-out as it does the direct form, and the intermediate's edge enters the cycle-detection graph.

When the fan-out fires — updates and deletes

Two behaviours changed with match, and they apply to the on form too:

• An update fans out on the old match ∪ the new one. Changing a matched field re-derives the records addressed by the previous value as well as the new one, so nothing is left silently stale. ⚠️ A write that does not thread the prior record — a sync-applied wave write, a tiers restore — degrades to the new tuple only. That is the previous behaviour, chosen over guessing at an old key.
• A delete fans out on the tombstoned record's values. Deletes previously fired no triggers at all. The fan-out runs after the delete commits, so a maxFanout cap or a strict: true failure can surface from delete() — a call site that never used to throw for derivation reasons.

Match targets are checked at registration

vault.collection() validates every from / to against the collection's declared fields, so a provable typo throws there rather than matching nothing forever. Two refinements land in 0.7.0-pre.13 (noy-db#1266):

A virtual field is evaluated on read and never stored, so a matcher pointed at one reads every candidate row and matches nothing, permanently — the exact failure the guard exists to prevent, reached through it. It is refused rather than supported because matching a virtual field turns an indexed narrow into a full scan; use mode: 'materialized'. The virtual check runs even when the field list is unreadable, unlike the typo check beside it: "declared but never stored" is provable from the declaration alone.

withRollup(options)

withRollup is the inverse of a join: instead of reading children on demand, the parent carries a maintained summary folded from its children by foreign key. It recomputes on child insert, update, AND delete — gap-free.

import { withRollup } from '@noy-db/hub/derivations'

withRollup<Sale, Buyer>({
  from: 'sales',          // child collection (the trigger)
  key: 'buyerId',         // FK on the child → parent id
  into: 'buyers',         // parent collection
  field: 'revenueByYear', // field on the parent to maintain
  compute: (sales) => groupSumByYear(sales, 'total'),
})

• On a from write/delete the parent at id child[key] is recomputed: compute(children where child[key] === parentId) is patched onto parent[field]. A parent write also recomputes its own aggregate, so a parent created after its children still fills in.
• A child that RE-PARENTS recomputes both parents since 0.7.0-pre.13 (noy-db#1257). Moving a child from A to B previously recomputed only B and left A's aggregate holding a number that was correct before the move — a stale total reads as data rather than as an error, so nothing prompts anyone to re-check it. ⚠️ Same qualification as the triggerBy fan-out: a write that does not thread a prior record — a sync-applied wave write, a tiers restore — degrades to new-parent-only rather than guessing at the old key.
• Only field is patched onto the parent's raw record (other fields and i18n maps untouched); a value-equality guard suppresses no-op writes.
• The aggregate gathers children through the FK index when from is indexed on key, else a scan.
• Eager-only in this slice. Desugars to a derivation carrying a rollup marker; dispatch handles it without the executor.

Why not a materialized view? An MV aggregates children into a parallel collection (buyer-totals.get(id)); it cannot write the aggregate onto a field of the parent buyers record. withRollup does. vault.forget() does not auto-re-derive (it bypasses the put path) — recompute explicitly if a forgotten record must drop out of an aggregate.

optional

Declare an output as optional: true to let derive return null for that key:

outputs: {
  receipt: { shape: 'record', collection: 'receipts', optional: true },
},
derive: (alloc) => ({
  receipt: alloc.servicesNetPortion > 0
    ? { id: alloc.id, paymentId: alloc.paymentId, appliedAmount: alloc.appliedAmount }
    : null,
})

Semantics:

• null (or undefined) for an optional output → no write fires.
• If a previous derivation emitted an output at this id, it's deleted (a tombstone for derived data). The eager path captures the prior envelope on the active transaction context so the delete rolls back alongside the source op on transaction failure.
• A never-emitted optional output is a silent no-op.
• Returning null for a required output (default — no optional flag) still throws DerivationOutputShapeError.

Same behavior on eager and lazy lifecycles, and through vault.deriveAll().

Tombstone-vs-onDelete composition

A tombstone is a system-internal delete: the derivation engine revoking its own prior emission because the source flipped to the "no output" branch. User onDelete guards registered on the output collection are not consulted on tombstones — the same way amendments bypass user-facing hooks. If they fired, a consumer registering both (a) an optional: true derivation and (b) an onDelete: () => throw on the output collection would deadlock: every flip-to-null source write would block on the user's own append-only rule.

Concretely: if you ship a paymentAllocation → receipt derivation with optional: true, AND receipts.onDelete: throw for legal-document immutability, the tombstone bypass keeps both coherent. Users can no longer manually delete a receipt (good), and the system can still revoke its own prior emission when the underlying allocation restructures (also good).

The bypass is scoped to the system-internal delete path (Collection._internalDelete); a user-initiated Collection.delete on the same record still fires onDelete normally.

lifecycle

• eager — derive runs synchronously inside the source-write transaction. Outputs are written via the same Collection.put pipeline. Recommended for small, fast derive functions.
• lazy — source-write marks output ids stale; first read of any output triggers the derive. Recommended when the derive is expensive and most sources are written without being read.

strict

Default (strict: false): per-output failures are isolated. Other outputs commit; the failed output is absent from the store and re-attempts on next vault.deriveAll.

strict: true: a single output failure rolls back the source write (only when wrapped in withTransactions). Use for outputs that must remain consistent with the source.

vault.deriveAll(collection)

Re-derive every record in the named source collection. Useful after a strategy change (the strategy hash mismatch forces a recompute on next visit, but deriveAll is the explicit bulk path):

const { derived, failed } = await vault.deriveAll('pdfs')
// → { derived: 42, failed: 0 }

_derivedFrom

Every derived record carries:

{
  "_derivedFrom": {
    "source": "pdfs",
    "sourceId": "abc",
    "sourceVersion": 3,
    "derivedAt": "2026-05-18T00:00:00.000Z",
    "strategyHash": "sha256-..."
  }
}

strategyHash (a hash of the source name, the output keys, and the derive function's own source) is the mechanism for detecting strategy drift: a record whose hash doesn't match the current strategy is recomputed by vault.deriveAll. sourceVersion is stamped 0 on records produced by vault.deriveAll() or by lazy resolution — the source record's real version isn't threaded through those two paths yet.

Composition order

Derivations fire after the store write and ledger append. A guard that blocks a source write also blocks the derivation that would have fired from it.

Collection.put
  1. Permission check
  2. GuardRegistry.check
  3. Encrypt + store.put
  4. Ledger append
  5. DerivationRegistry           ← this doc

Errors

All extend NoydbError:

• DerivationCycleError(path[]) — graph contains a cycle
• DerivationDepthError(limit, attempted) — cascade exceeded maxDepth
• DerivationOutputUnknownError(collection) — output collection unknown
• DerivationOutputShapeError(outputKey, detail) — derive returned wrong shape
• DerivationCapExceededError(key, count, max) — a triggerBy fan-out or array-shape output exceeded maxFanout

Behavior when NOT opted in

• createNoydb({ derivationStrategies: [...] }) is the only way to register derivations; without it, no derive function fires
• vault.deriveAll(collection) is a no-op — it returns { derived: 0, failed: 0 } rather than throwing, since there is nothing registered to run
• DerivationCycleError / DerivationDepthError / DerivationOutputUnknownError / DerivationOutputShapeError / DerivationCapExceededError are still importable from @noy-db/hub/derivations but are never thrown by core

Pairs well with

• guards — guards run before encryption; a blocked source write short-circuits the derivation that would have fired
• transactions — strict: true derivations require withTransactions so output failures can roll back the source write
• history — every committed derivation output fires its own ledger entry through the standard Collection.put pipeline

Edge cases & limits

Zero-knowledge guarantee

The derive function executes after DEK unwrap, on plaintext. The store never sees plaintext. Outputs are encrypted with the same DEK as the source before they reach the store. _derivedFrom metadata lives inside the encrypted payload, not in the plaintext envelope.

Cycle detection

Cyclic graphs (A → B → A, self-loops, etc.) are rejected when a vault opens (db.openVault), with DerivationCycleError. The graph is the union of every strategy's source + output collection set.

Behavior notes

• Lazy stale-tracking is in-memory only. A vault close-and-reopen loses pending stale flags. Derived records with a matching strategyHash are treated as fresh; vault.deriveAll() is the explicit recompute path after strategy changes.
• sourceVersion: 0 on bulk recomputes. vault.deriveAll() and the lazy-resolution path stamp _derivedFrom.sourceVersion = 0 because the envelope version isn't threaded through those two paths.
• Recursion guard. The dispatch hook in Collection.put skips when the incoming record carries a _derivedFrom field — a defense-in-depth check alongside the openVault cycle validator.

Not yet built

The following are outside what withDerivation / withRollup do today — there is no API surface for them yet:

• Cache-tier storage backends as a derivation target
• Built-in derivers (PDF text extraction, image processing, etc.) — you bring your own derive function
• Scheduled / cron-style refresh (only eager and lazy exist)
• Non-deterministic derivations with persistence — deterministic: true is required
• External or sandboxed derivation runtimes
• Publishing a derivation's output to a public CDN
• Streaming materialized views for continuous/unbounded result sets
• Persisting the lazy stale-flag map across a vault close
• Threading the real per-record source version through bulk recompute paths

See also

• materialized-views — persist an entire query as a live, auto-refreshed collection (sibling derived-data service)
• overlay-views — operator row-level overrides on a source or MV (sibling derived-data service)
• guards — write-time rules; system-internal deletes from derivations bypass them
• transactions — required for strict: true rollback on derivations
• history — every derived write still appends its own ledger entry
• Package tests plus showcase 80-with-derivation.showcase.test.ts
• Proven by showcases 80-with-derivation and 104-reverse-denorm.