Build time: parse founding docs into 108 chunks along their own semantic seams → embed each with all-MiniLM-L6-v2 → L2-normalize → ship as a 451 KB JSON. Runtime: the browser loads the same quantized ONNX model, embeds the query, and scores it. The upgraded path runs three signals — dense vector similarity, BM25 keyword, and a cross-encoder reranker — then returns top passages with citations.

Founding documents carry their own boundaries: each grievance is one accusation, each amendment one right, each section one power. Chunking on those seams means every chunk is a complete thought with a citation a human recognizes — "Amendment IV" with full text, not "chunk 47 starting mid-sentence." The transferable rule: find the document's atomic unit first — contracts have clauses, API docs have endpoints, runbooks have steps.

Once every vector is unit length, cosine similarity reduces to a plain dot product — no arccos, no division at query time. Search becomes a single multiply-and-add loop.

108 passages × 384 dimensions = 41,472 multiplications — under a millisecond on any device. A sorted array is the right structure when the corpus fits in memory; no vector database and no approximate-nearest-neighbor index needed. You reach for ANN/HNSW only once the corpus outgrows memory.

Dense vectors catch meaning, BM25 catches exact terms (names, numbers), and a ms-marco-MiniLM cross-encoder reranks the merged candidates by scoring query-passage pairs directly. Hybrid + rerank beats any single signal on recall — dense alone misses exact-term matches, keyword alone misses paraphrases.

Yes — there's an eval harness (eval/run-eval.js) over a labeled question set. The upgrade moved measured recall from 93.3% to 97.8%, with the multi-hop slice at 100%. I also tested a bigger embedding model (bge-base) and rejected it on the numbers — measured, not assumed. "I measured and rejected an option" is the senior signal.

In the curated path the AI only does retrieval and question-matching — the short answers and 108 passage explainers are hand-written at build time, so nothing generates text from scratch. The optional browser-side synthesis model (LaMini-Flan-T5-77M) is constrained to the retrieved passages as its only input, so it can't introduce facts that aren't there, and its output is labeled "computed."

27 curated Q&A pairs embedded alongside the corpus. The query is scored against them; above a 0.60 cosine threshold a human-written modern-English short answer shows above the founders' passages. The threshold is empirical: paraphrases of covered questions score 0.63–0.84, off-topic queries stay under 0.20, so the gap kills false positives.

Transformers.js runs a quantized ONNX build of the model (22.9 MB), cached in the browser's Cache API after the first visit. The key line is env.backends.onnx.wasm.proxy = true, which moves inference into a web worker. Without it, ONNX ran on the main thread and blocked for the whole embedding pass.

Lighthouse mobile the night before launch. First, render-blocking Google Fonts (fixed with the media="print" / onload swap). That exposed a second issue: Total Blocking Time jumped to 4,500 ms because the blank screen had been hiding ONNX inference on the main thread. Moving inference to a worker dropped TBT under 10 ms. Lesson: optimizing one metric can reveal another — sequence matters.

Marginal cost per query is zero, at 10 users or 10 million, because each visitor brings their own compute. The heavy bytes (22.9 MB model, ~11 MB runtime) come from third-party CDNs; the Vercel origin serves ~600 KB per cold visit, and the model caches after first load. Free tier covers ~170,000 cold visits/month.

Private data (the whole index ships to every visitor), large corpora (the index download outgrows its welcome), or when you need generated prose of real quality (a 77M-param browser model isn't GPT-4). It's for small, public, read-heavy corpora — docs, policy texts, FAQs. Matching the tool to the problem is the actual engineering judgment.

PROBE. Six stages, each a component with one job: a principle emits probes carrying a rubric, the target model answers, a judge scores the exchange against the rubric, a metric aggregates, a report bundles the evidence. Because each stage sits behind an interface, I can test them in isolation.

Same reason you trust any validated model — you check it against known cases first. The judge has a reliability check: feed it two identical answers and it must score them equal, or none of its other numbers mean anything. That's gated as a real integration test before the judge grades anything.

v1 measured fairness by how far two answers diverged from each other. That broke for groundedness — one answer checked against given facts, not two answers compared — and for prompt-injection and refusal. The choice was bolt on special cases or find the shape that fits all four. I unified it: every principle emits probes carrying a rubric, and the judge answers one question, assess(rubric, exchange) → 0–1. Four principles became one machine with different rubrics. Lesson: when an abstraction strains, unify it, don't wrap it in conditionals.

The whole engine is one contract: evaluate(principle, target). The target doesn't have to be a chatbot — a classical-ML evaluator (credit, fraud, pricing) implements the same contract and plugs in without touching the pipeline. Honest caveat: that's a designed seam, not shipped in v1. Keeping that qualifier on every time is what makes the rest credible.

The Vercel serverless deploy failed twice. I read the actual traceback each time — fixed the package bundling and the requirements location. I also switched an async job model to a synchronous endpoint, because serverless functions don't share memory between requests, so the job state was vanishing.

The target and judge are behind interfaces, so the core suite swaps in fakes — zero network calls, runs in under a second. The one live-judge test is gated behind an API key. That's what lets me trust the numbers: the plumbing is proven deterministically, and only the model call is non-deterministic.

Existing tools give you a benchmark score. They don't give a governance owner a named principle, a threshold set in advance, and an audit bundle. The framing — policy to control — is the contribution, not the model call.

LLM behavior only in v1, deliberately scoped. Four worked principles: gender fairness, groundedness, prompt-injection resistance, refusal-appropriateness. 32 passing tests, 1 gated live-judge integration test, core suite under 1 second with zero network calls, groundedness ships with a 0.85 pass threshold.

Those are governance lifecycles — frame, classify, assess, design controls, measure, prove. PROBE isn't a competitor; it's the instrument you plug into the measure step. A framework says you must measure and evidence a control. It doesn't produce the number. PROBE does — the score and the evidence bundle for one principle at a time.

Fair challenge, and I say it plainly: it's a designed and documented extension point, not shipped code. I built the contract so it can exist, because a measurement framework that could only ever look at chatbots would be measuring the wrong thing in a bank — the high-risk models are credit, fraud, and pricing.

Real travel problem: Paris's 2025–26 fare reform split one ticket into separate metro and bus products, so every English tourist guide went out of date at once. I built a four-tab React app — what to buy, card-rejected help, troubleshooting, and a free-form AI assistant. It stays current for free: a Puppeteer scraper runs each morning in GitHub Actions, reads the official Ile-de-France prices, and writes a JSON file that only commits when something changed, which redeploys the site. The Gemini assistant sits behind a serverless function so the key never ships to the browser.

Fares change rarely, so per-request freshness isn't worth a server or DB. A committed JSON file is free to host, gives me an audit trail in git history, and the daily cron keeps it current enough. The cost is a rebuild per change, which is fine at this cadence.

Layered extraction — labeled text first (Full price / Reduced price), then price-class CSS selectors, then a catch-all euro-amount regex. Each page is wrapped in its own try/catch, the workflow uses continue-on-error, and a confidence flag only promotes to "High" when the primary metro price actually parses. It degrades to defaults instead of crashing.

To keep the API key server-side. Calling Gemini from React would bake the key into the browser bundle. The function is a thin proxy that validates the request and forwards it.

The proxy shipped as an open, unauthenticated endpoint: no origin check, no rate limit, no prompt-size cap, and it forwarded the client-supplied prompt verbatim. Anyone could POST arbitrary prompts and drain the paid Gemini quota. The fix is origin/referer check + rate limit + max-length cap + building the prompt server-side instead of trusting the client body. Catching that in my own code is the instinct you'd want on someone touching bank systems.

No — it's single-turn. Chat history is kept in React state for display but isn't sent back; each call sends the system prompt plus the current question. I left multi-turn out for simplicity and cost, and I call that out as a known limitation rather than hiding it.

Right now it's concatenated with the user question client-side into one prompt string. The cleaner version uses Gemini's dedicated system-instruction field and builds the prompt server-side — which also closes the injection gap from trusting the client's text.

The regex scraper against the French site. If they restructure the fare pages or change wording, extraction returns null, confidence stays Medium, and the app falls back to defaults. It's the most fragile dependency — the fix is structured selectors or an official data export, plus a CI test that fails loudly when the primary price is null instead of silently degrading.

Honest gap: the static system prompt even tells the model not to give specific numbers, so fare answers come from Gemini's training data, not the scraped fares.json. The highest-leverage fix is injecting the live fares into the prompt — a small RAG-style grounding step that turns two disconnected features into one authoritative product.

It's a client-rendered CRA, so fetching the URL returns basically just the title — no content or Open Graph for crawlers. For a discovery-dependent tourist tool that's the biggest miss; the fix is prerender/SSR (react-snap, or a framework with SSG) plus real meta and OG tags.