stable

Email archive

Model MIME emails as one queryable record plus blob slots — the record/blob decision matrix and the ingest flow.

Audience: apps that ingest raw MIME email (.eml) and need to query metadata, serve attachments, and keep forensic originals — inboxes, archives, compliance systems. Bundle: core + withBlobs().

You receive raw MIME emails (.eml files) and want three things at once: query the metadata (from, to, subject, thread) as structured data, serve the bodies and attachments as files, and keep the original bytes for forensic re-parsing. This recipe shows how to lay that out in noy-db — an encrypted, offline-first document store where structured data lives in queryable records and binary payloads live in blob slots attached to those records.

It assumes you know the basics of vaults and collections and uses the blobs service for attachments.

Setup

Blobs are an opt-in service: pass blobsStrategy: withBlobs() to createNoydb. Everything else in this recipe is core.

import { createNoydb } from '@noy-db/hub'
import { withBlobs } from '@noy-db/hub/blobs'
import { toFile } from '@noy-db/to-file'

const db = await createNoydb({
  store: toFile({ dir: './mail-data' }),
  user: 'me',
  secret: 'user-supplied secret',
  blobsStrategy: withBlobs(),        // enables collection.blob(id)
})

const vault = await db.openVault('mail')

Short answer

An email is ONE entity with many facets. Model it as one Email record plus N+2 blob slots on that same record:

emails/<email-id>        ← the record (structured fields)
  └─ blob slots:
     ├─ 'raw'            ← original .eml (always)
     ├─ 'body-html'      ← if HTML part exists
     ├─ 'body-text'      ← if plaintext body is > 10 KB
     └─ 'att-N-<file>'   ← one blob per attachment

Don't split into separate emails + attachments collections unless you cross ~50 000 emails. The attachment manifest lives on the email record; the bytes are blob slots on that same record. noy-db's content-address deduplication (each blob is stored under an eTag — HMAC-SHA-256(blob key, plaintext bytes)) stores the same PDF attached to 100 emails as one blob on disk.

The decision matrix — which part lives where

Part	Size	Query-needed?	→ Where
`from`, `to`, `cc`, `subject`, `receivedAt`	tiny	yes	record
`messageId`, `threadId`, `inReplyTo`, `references`	tiny	yes (threading)	record
`labels`, `flags` (read / starred / archived / replied)	tiny	yes	record
`bodyPreview` (first ~200 chars)	small	no (display)	record
`bodyText` plaintext	1–50 KB typical	rarely	record if < 10 KB, blob if larger
`bodyHtml`	often 100 KB+	no (display)	blob slot `body-html`
Raw `.eml`	50 KB – few MB	no (forensic)	blob slot `raw`
Attachments (PDFs, images, nested `.eml`)	10 KB – 100 MB	no (download)	blob slot per attachment
`attachments[]` metadata (filename, mime, size, slot name)	tiny	yes	record
`extracted` (AI summary, topics, action items)	small	yes	record

One rule: if you ever .where() or .orderBy() on it, put it on the record. Otherwise it's a blob. Keep records small so the in-memory vault stays fast; blobs lazy-load.

Concrete record shape

interface Address {
  email: string
  name?: string
}

interface Email {
  id: string                      // ULID — stable identity
  messageId: string               // RFC 5322 Message-ID header (dedup)
  threadId: string                // derived — see "Threading"

  inReplyTo?: string
  references?: string[]

  from: Address
  to: Address[]
  cc?: Address[]
  bcc?: Address[]
  subject: string
  receivedAt: string              // ISO-8601 — Received: header timestamp
  sentAt?: string                 // ISO-8601 — Date: header

  labels?: string[]
  flags?: {
    read?: boolean
    starred?: boolean
    replied?: boolean
    archived?: boolean
  }

  bodyPreview: string             // first ~200 chars for listing UIs
  bodyText?: string               // plaintext ONLY if < 10 KB
  bodyTextInBlob?: boolean        // true → fetch blob slot 'body-text'
  bodyHtmlInBlob?: boolean        // true → fetch blob slot 'body-html'

  attachments?: Array<{
    filename: string
    mimeType: string
    sizeBytes: number
    contentId?: string            // cid:... reference for inline images
    slotName: string              // blob slot holding the bytes
  }>

  extracted?: {
    summary?: string
    topics?: string[]
    sentiment?: number
    actionItems?: string[]
    invoiceNumber?: string        // when the email IS an invoice
  }
}

Note there is no eTag on the manifest: the content address is slot metadata, maintained by the blobs service. Read it with emails.blob(id).list() when you need it (see Shared documents).

Ingest flow

MIME parsing itself is out of noy-db's scope — use any .eml parser and adapt its output to this minimal shape:

// The shape your MIME parser must produce.
interface ParsedEmail {
  headers: Record<string, string | undefined>
  from: Address
  to: Address[]
  cc?: Address[]
  subject: string
  receivedAt: string              // ISO-8601, from the top Received: header
  text?: string
  html?: string
  attachments: Array<{
    filename: string
    contentType: string
    size: number
    contentId?: string
    bytes: Uint8Array
  }>
}

import { readFile } from 'node:fs/promises'
import { generateULID, type Collection } from '@noy-db/hub'
import { parseEmail } from './your-eml-parser'   // MIME → ParsedEmail

/** Filename → safe slot-name fragment. */
function slug(filename: string): string {
  return filename.toLowerCase().replace(/[^a-z0-9.]+/g, '-')
}

async function ingestEml(emails: Collection<Email>, emlBytes: Uint8Array): Promise<string> {
  const parsed = parseEmail(emlBytes)
  const id = generateULID()

  const attachments = parsed.attachments.map((a, i) => ({
    filename: a.filename,
    mimeType: a.contentType,
    sizeBytes: a.size,
    contentId: a.contentId,                         // for cid: inline images
    slotName: `att-${i}-${slug(a.filename)}`,
  }))

  const record: Email = {
    id,
    messageId: parsed.headers['message-id'] ?? id,
    threadId: await deriveThreadId(emails, parsed),
    inReplyTo: parsed.headers['in-reply-to'],
    references: parsed.headers['references']?.split(/\s+/),
    from: parsed.from,
    to: parsed.to,
    cc: parsed.cc,
    subject: parsed.subject,
    receivedAt: parsed.receivedAt,
    bodyPreview: (parsed.text ?? '').slice(0, 200),
    bodyText: parsed.text && parsed.text.length < 10_000 ? parsed.text : undefined,
    bodyTextInBlob: (parsed.text?.length ?? 0) >= 10_000,
    bodyHtmlInBlob: !!parsed.html,
    attachments,
  }
  await emails.put(id, record)

  // Always store the raw .eml — forensic source of truth
  await emails.blob(id).put('raw', emlBytes, { mimeType: 'message/rfc822' })

  if (record.bodyTextInBlob) {
    await emails.blob(id).put('body-text', new TextEncoder().encode(parsed.text!))
  }
  if (record.bodyHtmlInBlob) {
    await emails.blob(id).put('body-html', new TextEncoder().encode(parsed.html!))
  }

  for (const [i, att] of attachments.entries()) {
    const src = parsed.attachments[i]!
    await emails.blob(id).put(att.slotName, src.bytes, {
      mimeType: att.mimeType,
      filename: att.filename,
    })
  }

  return id
}

const emails = vault.collection<Email>('emails')
const emlBytes = new Uint8Array(await readFile('./inbox/q2-invoice.eml'))
const id = await ingestEml(emails, emlBytes)
// → '01J1ZC4Q4T9GQ0V4Y3F2M8KDXP'

When to flip to a secondary attachments collection

Add vault.collection<AttachmentIndex>('attachments_index') as a manually maintained secondary collection (not built with the materialized-views service — you write the sync logic yourself, e.g. inside ingestEml), not a source of truth, when:

You cross ~50 000 emails and scan+filter queries for "find all PDFs" become slow.
You have retention rules per attachment type (e.g. "medical images purge after 7 years, legal agreements retain indefinitely") — easier to run lifecycle policies on a dedicated collection.
You want attachments searchable independent of email state (e.g. search deleted emails' attachments for compliance).