Under the hood
How Whaler is built.
Whaler started in April 2026 as a landing page with a good story and no product underneath it. By July it was a working AI meeting assistant in production: a desktop client capturing both sides of a live meeting, an agent loop turning transcripts into evidence-backed records, and a retrieval layer that remembers every prior conversation.
One engineer wrote all of it. This page is the part a product page usually hides: the architecture, the decisions, and what each one cost.
1
Engineer
79
Commits, Apr → Jul 2026
152
Unit tests, all offline
~14k
Lines of TypeScript
The system
Desktop — Electron, Windows
- —Native window
- —Mic capture → the advisor
- —System loopback → the client
- —Thin client: ships no secrets
Cloud — Next.js on Vercel
- —Planner–executor agent loop
- —Hybrid RAG retrieval
- —Real-time assist state machine
- —Meeting processing + persistence
Data & models
- —Postgres + pgvector (Supabase)
- —Claude — planner + extraction
- —OpenAI — transcription, embeddings
- —Deterministic compliance rules
The agent loop
- 1
The planner reads the transcript and decides what to do with it. It owns the conversation and the stopping condition — not a fixed pipeline.
- 2
extract_structured_facts runs as an executor with its own inner LLM call. Output is parsed against a Zod schema; a validation failure is re-prompted with the specific error, up to three attempts.
- 3
The planner judges whether a compliance review is warranted — a semantic call. An administrative request ('email me a PDF of my statement') gets skipped, which is the interesting case: it reasons about the content of an instruction, not merely its presence.
- 4
flag_compliance_issues runs the rule engine over the extracted facts — regex, zero tokens, every flag carrying the field and phrase that triggered it.
- 5
search_meeting_history is available throughout so the planner can pull in prior meetings when the current one references them.
- 6
The loop stops on end_turn, an iteration cap, or a per-tool call limit — a planner stuck on one tool gets an error result rather than a blank cheque.
You can read one complete run — input, output, flags, evidence, timings and token counts — on the sample report page.
Decisions, and what they cost
Every one of these could have gone the other way. The trade-off line is the part that matters.
Planner–executor split, not one mega-prompt
An outer loop owns the conversation and decides which tool to call and when to stop. Tools execute: one makes its own inner LLM call, one runs a pure rule engine, one queries the vector store.
Trade-off — More round-trips and latency than a single prompt that does everything — bought back as a loop that stays simple no matter how many tools get added, and tools that can be tested in isolation.
One Zod schema doing three jobs
The same schema is the LLM contract (via z.toJSONSchema), the runtime validator (safeParse), and the inferred TypeScript type. Every decision, action item, and instruction must carry a verbatim evidence_quote — the model cannot assert a fact it can't quote.
Trade-off — Upfront schema rigidity, and the model occasionally fights the shape. In exchange: guaranteed structure and provenance everywhere downstream, and a structural hallucination guard rather than a prompt asking nicely.
Self-healing extraction instead of a hard failure
When output fails validation, the Zod error is formatted into a readable path: message list and pushed back as a follow-up turn — 'your output failed validation, fix and resend.' Three attempts, then a typed failure rather than a throw.
Trade-off — Extra tokens on the failure path. The retry cost is tracked in telemetry rather than hidden, because a retry loop you can't see the price of is a bug waiting to happen.
Not every tool should be an LLM call
Fact extraction is LLM-backed. Compliance flagging is a deterministic rule engine — zero tokens, microsecond latency, unit-testable, and analyzable for coverage. The planner mixes both without caring how either works.
Trade-off — Rules are narrow and brittle where an LLM is flexible. That narrowness is informative: when a flag doesn't fire, the reason is inspectable, which is not true of a model that just didn't mention it.
Two retrievers, fused on rank
Client-history search runs pgvector cosine similarity over OpenAI embeddings and Postgres full-text ts_rank in parallel, then fuses with Reciprocal Rank Fusion — 1/(k+rank), so the incomparable score scales never have to be normalized. Chunking is speaker-turn aware, so a retrieved chunk rarely starts mid-sentence.
Trade-off — Two indexes and a fusion step to maintain, for retrieval that survives either retriever's blind spots. Measured with a retrieval eval (hit@k, MRR) over public QMSum fixtures rather than assumed.
The real-time pipeline is a pure state machine
Functional core, imperative shell: each new turn folds into rolling state and emits side-effect requests; the shell runs the LLM and DB work and feeds results back in. Summaries update incrementally with a periodic full recompute for drift; re-extraction is gated by a valve (N new turns or a keyword hit) with monotonic dedup.
Trade-off — More moving parts than 're-summarize after every turn' — repaid in token cost and in cadence logic that's offline-testable, which is why most of the 152 tests can exist at all.
Prompt caching measured before it was used
Caching is a prefix match with a per-model minimum; a prefix under it is silently not cached. Segment sizes were measured with a free countTokens probe before placing breakpoints. Two breakpoints — one cross-run on the system block, one rolling on the newest tool result — so a request never exceeds two regardless of iteration count.
Trade-off — Breakpoint bookkeeping in the loop. Worth it because the alternative is a cache-control call that looks right and quietly does nothing.
Secrets never ship in the desktop build
The Windows desktop client is Electron and deliberately thin: it captures microphone and system-loopback audio (two channels, so the transcript is speaker-labeled by construction) and loads the hosted app. All keys live in server-side env on Vercel.
Trade-off — The desktop app is useless offline and roughly doubles transcription cost by running two streams. In return, who-said-what is a property of the capture rather than a diarization model's guess — and the .exe is not a credential to be extracted.
What it deliberately is not
The compliance layer is a small rule engine — not an implementation of CIRO or PIPEDA. It catches a defined set of moments well and says nothing about the rest.
No model fine-tuning — prompt engineering and structured outputs only. The interesting problems here were the loop and the retrieval, not the weights.
Single-advisor workspace — auth and multi-tenancy are the next infra step; the user_id column is already threaded through, deliberately deprioritized until distribution needs it.
Narrow on purpose — the goal was depth on a few hard problems over surface area on many easy ones. Feature breadth was the easier and less interesting path.
How it's kept honest
152 unit tests, none of them touching a network
Rank fusion, speaker-aware chunking, the compliance rules, the real-time state machine's valves and dedup, schema contracts, Markdown export. The functional-core design is what makes this possible — the parts worth testing are pure functions. GitHub Actions runs them plus a production build on every push.
Evals for the parts tests can't pin
A retrieval eval scores hit@k and MRR over public QMSum fixtures; an agent eval scores extraction against expected structure. Unit tests pin the deterministic logic; the evals measure the parts that are probabilistic by nature — and the difference between the two is the point.
Stack
- Language & framework
- TypeScript · Next.js (App Router) · React · Tailwind v4 + shadcn/ui
- AI
- Anthropic Claude Sonnet 4.5 (planner + extraction) · OpenAI Realtime/Whisper (transcription) · text-embedding-3-small
- Data
- Postgres + pgvector on Supabase · Drizzle ORM · hybrid vector + full-text retrieval with RRF
- Desktop
- Electron (Windows), dual-channel audio capture with echo cancellation
- Quality
- Zod v4 contracts · Vitest (152 offline tests) · GitHub Actions CI · retrieval + agent eval harness
- Infra
- Vercel (push to main deploys) · server-side secrets only
Built by
Donny Feng
Software developer · Vancouver, B.C.
Four and a half years building enterprise software in C#/.NET and SQL Server before founding Whaler Technology and taking this product from an empty repository to production alone — architecture, agent design, retrieval, desktop client, tests, CI, and the design of the page you're reading.
Currently open to software engineering roles in Vancouver or remote in Canada. Happy to walk through any decision on this page — including the ones that turned out to be wrong.