Capstan

Capstan is a B2B AI support co-pilot, live at getcapstan.vercel.app. A company creates a workspace, invites teammates with owner, admin, or agent roles, and uploads its help docs, which are chunked, embedded, and indexed into Postgres with pgvector by durable background jobs. Customer questions arrive through a one-line embeddable widget or an org-scoped REST API; for each one, Capstan retrieves the most relevant passages and has Claude draft a reply that cites the exact source chunks it used, with a retrieval-confidence score. A human agent reviews, edits, and sends every reply. Around that core sits a full SaaS shell: Stripe subscriptions with plan-gated limits, an analytics dashboard, rate limiting, and an evaluation harness for the AI pipeline, all deployed on Vercel and Neon with CI.

Selvik covered commerce, but it left the SaaS-specific engineering undemonstrated: multi-tenancy, subscription billing, background job infrastructure, and a production-shaped AI feature rather than a bolt-on. The AI part has a specific failure mode I wanted to engineer against: support bots that answer confidently whether or not they actually know. And LLM features are notoriously hard to test, so the project needed a real answer to 'how do you know it works?' beyond clicking around.

Multi-tenancy is hand-rolled rather than delegated to a hosted auth product, because it is the point: organizations, memberships, and roles in Prisma, with every query scoped by organization and every server action re-checking membership and role. The AI pipeline runs as durable Inngest steps: heading-aware chunking, Voyage embeddings into pgvector, then retrieval, a confidence guardrail, and a Claude call constrained to structured JSON that must map its claims onto numbered source chunks. The public surface is treated as hostile: hashed API keys in secret and publishable variants, per-conversation access tokens so a widget key can never enumerate other customers' conversations, and Postgres-backed rate limiting. Billing is Stripe Checkout and the Customer Portal, synced through signature-verified webhooks with an idempotency ledger.

The defining decision is that the AI escalates instead of guessing: below a similarity threshold the question goes straight to a human before any LLM call, and the model can also flag its own sources as insufficient, turning hallucination from an embarrassment into a routed workflow. Tenant isolation returns 404 rather than 403 for other orgs' resources, so the API never leaks what exists. Webhook idempotency uses event ids as a primary key, so Stripe's at-least-once delivery can never double-apply a plan change. And the eval harness splits what has a right answer from what is taste: retrieval hit-rate on a golden question set and guardrail behaviour are hard CI assertions, while citation quality and LLM-judged groundedness run advisory.

This project consolidated what production AI engineering actually involves beyond the model call: retrieval quality measured with evals rather than vibes, guardrails that fail toward humans, and citations as a first-class product feature rather than an afterthought. On the SaaS side, it was a masterclass in server-authoritative thinking, with entitlements, roles, and tenant boundaries all enforced at the data layer, and in operational plumbing: idempotent webhooks, durable job retries with idempotent writes, and a deploy pipeline that migrates the database before every build. It also sharpened judgement about real-world API constraints, from embedding rate limits to designing key formats that are safe to embed in a customer's page.