noy-db
stable

ndjson

Newline-delimited JSON export, streamed record-by-record for data pipelines and jq.

@noy-db/as-ndjson · pnpm add @noy-db/as-ndjson Streaming plaintext export — one JSON record per line.

What it does

as-ndjson is built for data pipelines, warehouse ingestion, and jq processing rather than human consumption. Besides the format instance asNdjson() (export and import through the vault's format port), it ships pipe(vault, sink, options) and stream(vault, options) — streaming exports with O(one record) memory, so a large collection never buffers whole — and a fromString() reader that parses newline-delimited JSON back into an import plan.

The vault needs formatsStrategy: withFormats() (from @noy-db/hub/as) for asNdjson(), toString() and download() — since 0.10.0-pre.2 the last two go through the format port, and without the strategy they throw FormatsNotEnabledError. stream() and pipe() stay on exportStream and do not need it. write() (Node) requires acknowledgeRisks: true, like every on-disk plaintext writer.

Usage

import { asNdjson } from '@noy-db/as-ndjson'

const ndjson = await vault.export(asNdjson(), { collections: ['invoices'] })
// {"id":"inv-1","client":"Globex","amount":1500}
// {"id":"inv-2","client":"Acme","amount":2400}

import { pipe } from '@noy-db/as-ndjson'

await pipe(vault, sink, { collections: ['invoices'], acknowledgeRisks: true })

import { fromString } from '@noy-db/as-ndjson'

const plan = await fromString(vault, ndjson, { collection: 'invoices' })
// → AsNDJSONImportPlan { plan: VaultDiff, policy: 'merge', apply() }

See also

json

noydb