RAG (retrieval-augmented generation)

Retrieval-Augmented Generation: instead of hoping the model remembers your docs, you retrieve relevant passages at call-time + put them in the prompt. The model answers from what you handed it, not from training data.

Voltro's RAG primitives live in three places:

• Storage — vector columns + HNSW indexes (Vector columns). Index-accelerated on postgres; MariaDB runs the distance operators natively; the other dialects store vectors but fall back to a sequential scan.
• Embedding generation — embed(text) from @voltro/ai + the vectorEmbedding() mixin
• Retrieval helpers — nearestNeighbours(...), hybridSearch(...) on ctx.store, rerank(...) from @voltro/ai

#The minimal RAG pipeline

// apps/api/database/docs.entity.ts
import { table, id, text, vectorEmbedding } from '@voltro/database'
import { tenant } from '@voltro/plugin-multitenancy'

export const docs = table('docs', {
  id:   id(),
  body: text(),
}).with(
  vectorEmbedding({
    from:       'body',
    model:      'text-embedding-3-small',
    dimensions: 1536,
  }),
  tenant(),
)

The mixin:

• Adds an embedding: vector(1536) column with an HNSW index.
• On INSERT, the runtime calls embed(body) (from @voltro/ai) + stores the vector.
• On UPDATE of body, re-embeds.

You write body. The vector handles itself.

Embeddings default to mock. Out of the box embed uses a deterministic, key-free mock provider (it hashes the text into a stable vector — reproducible, but not semantic). The model: 'text-embedding-3-small' string is stored but ignored by the mock. To get real embeddings, set AI_EMBED_PROVIDER=openai (or voyage / cohere) and install that provider's package (e.g. @ai-sdk/openai). See Providers.

#Querying

// apps/api/tools/searchDocs.tool.tsx
import { defineTool } from '@voltro/ai'
import { Schema } from 'effect'

export const searchDocs = defineTool({
  name:        'search-docs',
  description: 'Search the user knowledge base. Returns up to 5 passages.',
  input:       Schema.Struct({ query: Schema.String }),
  output:      Schema.Array(Schema.Struct({
    body:     Schema.String,
    href:     Schema.String,
    distance: Schema.Number,
  })),
})

export default async ({ query }, ctx) => {
  const results = await ctx.store.select('docs')
    .nearestNeighbours(query, 5)                  // embeds `query`, limit 5
    .all()

  return results.map((r) => ({
    body:     r.body,
    href:     `/docs/${r.id}`,
    // `distance` is a DISTANCE — lower = closer. Keep the runtime field
    // name so consumers don't sort it backwards (a "score" would imply
    // higher = better). Invert it explicitly if you want a similarity.
    distance: r.distance,
  }))
}

That's it. The runtime:

1. Embeds the query string (via @voltro/ai's embed).
2. Runs ORDER BY embedding <=> $1 LIMIT 5 (via the HNSW index on postgres; sequential scan elsewhere).
3. Returns rows with a distance field.

#Wiring into an agent

// apps/api/agents/help.agent.tsx — descriptor (browser-safe)
import { defineAgent } from '@voltro/ai/agent'
import { Schema } from 'effect'

export const help = defineAgent({ name: 'help', input: Schema.Struct({ prompt: Schema.String }) })

// apps/api/agents/help.agent.server.tsx — executor (server-only)
import { defineAgentExecutor } from '@voltro/ai'
import { help } from './help.agent'
import { searchDocs } from '../tools/searchDocs.tool'

export default defineAgentExecutor(help, {
  system: `You are a helpful support agent. When the user asks how to do
something, ALWAYS call search-docs first to ground your answer in the
docs. Then summarise + cite the relevant passage.`,
  tools:  { searchDocs },
})

The two agent files are all you write — the framework synthesizes help.send + help.messages (codegen-typed for the client). The model decides when to call search-docs; the system prompt nudges it strongly — "always call first" is usually enough.

#Chunking strategy

Voltro doesn't ship a markdown chunker — that's app-specific. The right strategy depends on your data:

Data	Chunk by	Why
Markdown docs	H2 sections (≤2k tokens each)	Self-contained units; preserves heading context.
API references	Function / type	Each row IS the chunk.
Long-form articles	Sliding window with 200-token overlap	Preserves cross-paragraph context.
Code	File / function	Each row IS the chunk.
Customer support tickets	Per-ticket	Each row IS the chunk.

For markdown chunking, a 30-line helper is enough:

const chunkBySection = (md: string): string[] => {
  const out: string[] = []
  let current = ''
  for (const line of md.split('\n')) {
    if (line.startsWith('## ') && current) {
      out.push(current)
      current = line
    } else {
      current += '\n' + line
    }
  }
  if (current) out.push(current)
  return out
}

Ingest:

for (const chunk of chunkBySection(md)) {
  await ctx.store.insert('docs', { body: chunk })
  // The vectorEmbedding mixin handles the embedding side-effect.
}

#Hybrid search

Pure vector similarity misses exact-match queries ("does X support Y" — the keyword "Y" is more reliable than its embedding). Combine vector + full-text:

import { hybridSearch } from '@voltro/database'

const results = await ctx.store.select('docs').use(hybridSearch({
  vector: { col: 'embedding', query },
  fts:    { indexName: 'docsBody', query },
  alpha:  0.6,    // 0 = pure FTS, 1 = pure vector
})).limit(5).all()

The FTS clause narrows the candidate set; the vector clause ranks it, fused with Reciprocal Rank Fusion. For a knowledge base it routinely beats either alone — for technical docs, lean toward alpha=0.4-0.6 (FTS-weighted).

#Re-ranking

For top-shelf retrieval quality, run a re-ranker over the top-N hits. rerank ships in @voltro/ai:

import { rerank } from '@voltro/ai'

const candidates = await searchDocs(query, 20)
const reranked = yield* rerank({
  query,
  documents: candidates,
  getText:   (d) => d.body,
  provider:  'cohere',
  model:     'rerank-english-v3.0',
  topN:      5,
})
const top5 = reranked.map((r) => r.document)

Re-rankers are slower than vector search but much more accurate. Use the cheap vector search to narrow to ~20 candidates, then the rerank model to pick the top 5. Total latency: ~150-300ms vs. 50ms for vector-only. The default provider: 'mock' scores by lexical overlap — deterministic and key-free for tests; the cohere / voyage providers resolve their SDK lazily (install the provider package to use them).

#Citing sources

The model needs source info in its context to cite:

const docs = await searchDocs(query, 5)
const context = docs.map((d, i) => `[${i + 1}] ${d.body}\n(source: ${d.href})`).join('\n\n')

const messages = [
  { role: 'system', content: `Sources:\n${context}\n\nAnswer using ONLY these sources. Cite as [1], [2], etc.` },
  { role: 'user',   content: input.question },
]

For UI that links each citation: parse [1], [2] patterns out of the model's output + map back to docs[0].href, docs[1].href. The agent SDK doesn't do this automatically — it's render-layer work.

#Tenant isolation

The vectorEmbedding() mixin + tenant() mixin compose correctly:

const docs = table('docs', {
  id:   id(),
  body: text(),
}).with(
  vectorEmbedding({ from: 'body', dimensions: 1536 }),
  tenant(),
)

Searches across docs are automatically tenant-scoped. The runtime AND-merges the tenant filter before the ANN order/limit, so Tenant A's queries never surface Tenant B's vectors — even though the vectors live in the same column.

#Cost considerations

Embedding cost (only with a REAL provider configured — the default mock provider is free and offline):

• text-embedding-3-small (OpenAI, AI_EMBED_PROVIDER=openai): roughly $0.02 per 1M tokens.
• voyage-3 (AI_EMBED_PROVIDER=voyage): roughly $0.12 per 1M tokens.

Check the provider's current pricing — these are rough figures.

Storage cost:

• 1536-dim float32: ~6KB/row + ~3KB HNSW overhead = ~9KB/row.
• 10k docs: ~100MB. Cheap.
• 1M docs: ~10GB. Plan around it.

Re-embedding cost (rebuilding the column with a new model) = full corpus × embedding cost. Pick your model + dimensions deliberately.

#When NOT to use RAG

• The data fits in the context window. If you have 50 docs and Claude can hold 200k tokens, just stuff them all in. Simpler, more accurate.
• The data IS the prompt. For "translate this paragraph", you don't need retrieval.
• You need exact lookups. RAG returns "similar" results; if you need "exactly this customer's order", use a regular query.

RAG is for when the corpus is too big for context + the answer is in a small slice of it.