Trust Answer Engine

I kept returning to one line from the brief: the trust site has a lot of useful information, but may not necessarily be enough. Sometimes the answer means talking to someone or finding a document that is not on the trust site. That was the moment the no-document question became the real problem for me.

I mapped the review around enterprise buyers in the Canva, Notion, Samsung class. They will not accept certificates alone; their internal policy requires a completed questionnaire on file, so every enterprise deal produces a security review. At the ceiling, a questionnaire has 800 questions. Those questions repeat heavily across buyers, then funnel to one person who answers them manually with an AI chatbot in the loop.

What stopped me was the question with no documented answer. Today that means walking to an engineer, getting a sentence, pasting it into a spreadsheet, and then losing it when the next buyer asks the same thing. I wanted the design to make that answer durable instead of letting the knowledge evaporate as soon as it was used.

“Answer once, never again.”

I did not see “add an LLM” as the missing piece. A chatbot is already in the loop and the problem remains. The useful system needs grounding, citation, confidence gating, routing to the right human, and capture of what that human said.

On my offline run, 800 questions became 84 distinct concepts because buyers ask the same things in different words. I keyed the answer bank on concept, never on the question string. That lets one governed answer carry across buyers instead of making the work start over with every workbook.

I kept 299 in view as a useful result, not an embarrassing one. Roughly a third of the workbook goes to a human, and the system treats that as the correct outcome when the evidence is not strong enough to answer on its own.

“Unanswerable is a first-class outcome, and a valuable one , it’s a product input.”

I made selection the core safety property because a model-authored sentence could quietly become a buyer-facing security claim. The model never writes that text. The wording lives in a knowledge-base file as a canonical ## Answer body. An agent’s job is to select which concepts a question maps onto, and a second read-only pass verifies the claims against their cited sources. Merging and rendering are deterministic code, so the final wording is not improvised at answer time.

I keep the selecting agents bounded to Read, Grep and Glob. They have no write, shell, or web access, and they work within bounded turns and a bounded budget. Their structured output is checked against the real catalogue paths, so a hallucinated concept path fails validation instead of producing a confident wrong answer.

“Optimise for never being wrong, not for throughput.”

I can recover from over-escalation, even when it is annoying. A confidently wrong security answer is a misrepresentation to a customer, in writing, with the company’s logo on it.

I built the answer bank as a versioned bundle of 61 concept files across ten categories, access control, AI governance, compliance, data handling, deployment, encryption, governance, personnel, resilience, tenancy. A concept’s file path is its identity. Each file carries frontmatter and three body sections: the canonical answer, its scope and limits, and internal notes that never go to a buyer.

The evidence tiers I used, and the rule behind them

I keep buyer citations at the artifact level: a direct link, or “available under NDA via the trust centre,” or “available on request.” Page and section numbers are stripped. Internal citations keep the source id, the concept path, the exact claim text and the tier. That detail belongs in my audit trail, not in a buyer’s spreadsheet.

I use freshness as a gate, not a badge. When a concept’s stale_after date has passed, it stops auto-answering and becomes a draft, even if everything else about it is perfect. Stale cited evidence must wait for a human to re-verify it before it can answer for me.

I enforce the honesty at composition time. When a question maps onto several concepts, the merge takes the worst posture, the highest deal-risk severity and the earliest staleness date across all of them. No model writes connective prose between the canonical bodies. I concatenate them in match-strength order, then append scope, limits and compensating controls when the posture requires them.

I let posture decide the cell, and code decide posture

The five checks I require before anything answers a buyer by itself

• ✓I require human verification of the concept; machine-only verification is not enough.
• ✓The concept status must be stable, not draft.
• ✓It must be fresh, with its stale_after date strictly in the future.
• ✓Every claim must be entailed by its cited sources, with a non-zero claim count.
• ✓Its support must be tier 1–3. Tier 4 or 5 can support escalation only.

If any one of those checks fails while governed support exists, I return a draft for a human to read. If there is no governed support at all, I escalate. I store no numeric trust score. Trust is derived at read time from the records, so it cannot drift out of sync with them.

Before I let any row be answered, I sweep the whole knowledge base for conflicts. Real trust surfaces disagree with themselves, and whichever version lands in a buyer’s spreadsheet can become the official answer. I wanted that problem visible before it reaches the workbook.

The sweep extracts quantities, counts, durations, retention and deletion windows, RTO/RPO-shaped values, and compares sources that measure the same thing. It also compares assertions lexically by subject, flags geographic conflicts such as US versus EU residency, shared versus dedicated infrastructure claims, and drift between governed material and marketing copy.

When I do have to resolve a conflict automatically, I use a conservative rule, always: longer durations, smaller capacities and counts. I never resolve toward the value that helps close the deal.

Once I reached the no-document case, the system had to do something useful with it. When no document answers the question, I do not guess or silently drop the row. I pick an owner, ask in Slack, and turn the reply into a durable asset.

The modal asks for more than prose: how certain the person is (certain, fairly sure, or needs someone else), whether the answer can be shared with a customer at all, and optionally who else should confirm it. Capture writes the file and appends an audit event without calling an agent. I store the human’s words verbatim, not paraphrased.

A captured answer starts as a draft with an empty verification list, a partial posture and a ninety-day review horizon. It is not born stale, and it cannot render as a confident Yes. Promotion into the permanent bank requires a different second human, valid frontmatter, a freshness horizon, a catalogue row and a log entry. I do not let one person both write and bless a security claim.

“Every human answer is captured as a durable concept.”

I made escalations state, not suspended processes

I keep everything waiting on disk as state in a manifest. A reconciler walks those files on a loop, derives what is actionable now, and dispatches it. Nothing is held in memory waiting for a callback, so the queue is a derived value.

I do not send buyers to a dashboard when they need their file filled in. The exporter reopens the original workbook and writes answers back using the row-and-sheet provenance captured during parsing, preserving structure, row order and merged cells. Text is protected from being interpreted as an Excel formula. A PDF is produced alongside it, with a cover page of currently disclosable certifications.

I put two guards in front of that export. Every row must carry one of five explicit dispositions, answered, answered with caveat, declined, not applicable, or deferred to a call, or the export refuses and names the offending rows. Both exporters run through the same sanitiser, which strips Slack URLs, employee identifiers, local file paths and ticket ids, and drops every citation that is not public, NDA-gated or available on request.

I wanted an unanswered question to leave useful evidence about the product. A concept with a partial or absent posture becomes a standing gap-register entry, ordered by blocked revenue, then by how often buyers ask, then by path.

My offline run produces 48 distinct questionnaire gaps and an eleven-entry standing register. I can use that register in a roadmap conversation: a recurring security question with revenue attached stops being a support burden and becomes a prioritised input. It is the same loop the second assignment is about, seen from the demand side.

Everything above was true of the offline version. Then I connected it to a real Slack workspace, Socket Mode, an installed bot, a live channel, and asked it real questions. That is a different kind of testing than a fixture file, and it found problems the offline suite never would have.

• ✓The Answer button on an escalation card did nothing. Slack puts button clicks inside body.actions[0], not at the top level of the payload the way I had assumed, so the handler was silently returning early on every click.
• ✓AES-256 kept coming back as [redacted]. My ticket-id redaction regex matched any short run of capital letters and digits, which is also the shape of a standard cipher name, so I had to teach it the difference.
• ✓A bulk upload of nine questions was fully answered, but every single row landed as a draft instead of auto-approving, even though the same questions had auto-approved cleanly in another run minutes earlier.

That last one is the one I actually care about, because it is not a UI bug, it is the gate lying to me. I trace it below rather than just listing it, because the trace is the point.

Tracing the bug that mattered: a false zero, not a real one

My auto-approve rule requires every atomic claim in an answer to come back entailed, verified === total, and total has to be greater than zero. A verifier that says nothing at all, 0/0, can never pass that check on purpose, because I never want “I checked nothing” to look the same as “I checked it and it’s fine.”

When I pulled the actual answer artifacts for both runs side by side, the buyer text, the citations and the resolved concept paths were identical. The only difference was that in the drafted run, the Claude Agent SDK verifier call had come back with an empty concept-path list for those rows, and my merge code trusted that empty list over the concept paths I already knew were resolved. Six correct answers got zeroed out because the model dropped a field on that particular pass, not because the evidence was actually thin.

“When the verifier drops the paths it was asked to check but still says it verified them, fall back to what I already resolved instead of trusting a blank field.”

I fixed the merge so a row with real resolved concepts but an empty echoed list re-derives its total from the concepts I know it actually checked, while a genuinely partial verification, a real 1 out of 2, stays exactly as reported. Then I went back into the run that had drafted six answers, confirmed each one against the correctly-verified version, and approved them the same way a human reviewer would from the dashboard, through the Approve action, not by editing a file directly.

The features I only found the shape of by using them

None of that shows up in a spec. It shows up when you send the bot a real questionnaire from a real channel and watch what actually happens to your screen.

The live run gave me real confidence in the wiring. It did not give me an accuracy benchmark, and I want to be exact about the difference.

• ✗My offline verification confirms a selected concept has a body and at least one source. My online verification is a live entailment check by the Claude Agent SDK, and I now know it can drop its own output on a given pass, which is exactly why I added the fallback above.
• ✗I have no accuracy evaluation set. A 26-question hand audit came out around 90% top-1 precision at roughly 63% row coverage; that is a hand count, not a measured benchmark.
• ✗Owners, Linear issue ids, account names and blocked-revenue figures are illustrative placeholders. The questionnaire is synthetic, structured on public CAIQ v4, SIG Lite and VSA patterns.
• ✗I simulate dispositions in the offline run so the export gate can be demonstrated end to end. That is recorded as an audit event and is explicitly not human review.

“Answers are selected, not generated.”