Field experience first
Years in stimulation and chemical operations, supervising a crew of 15, and training engineers and operators. The rules in this study come from reading shift records, not from a tutorial.
AI LAB / EVALUATION FIRST / MODEL-FREE BASELINE
Test the AI layer
before you trust it.
Permissions. Sources. Abstention.
Proved before a model is plugged in.
This is a working study of a handover assistant that answers only from records the reader may see, cites every claim, and says “I can’t answer that” when the evidence is missing. It runs below, in your browser, on fictional records.
01 / Why this is built this way
Credibility, stakes,
and proof.
The next crew trusts the record
People inherit whatever the last shift wrote down. A summary that invents a number, or shows a record the reader should not see, is worse than no summary at all.
Checks you can run
Sixteen written cases, nine tests that attack the verifier directly, and one limitation published instead of hidden. Run them yourself below, or with npm run test:ai-study.
02 / Ask the assistant
Choose who is asking.
Then ask anything.
Pick a reader, then a question or your own. Watch the four stages. Try asking for another company’s records, or tell it to ignore its rules.
Who can read: company, owner or crew:Name. Records are numbered R1, R2 in order. Use made-up or non-confidential text: this page runs entirely in your browser, nothing is sent or stored, and it is a demonstration, not a production system. Limit: 60 records, 600 characters each.
SAMPLE QUESTIONS
RESULT
Choose a question to see the answer, its sources and the checks it passed.
ACTIVITY LOG · THIS SESSION
03 / Run the evaluation
Sixteen cases.
One limitation shown.
The same cases and code the automated tests use. Categories cover answerable, outdated, unauthorized, conflicting, injection and unanswerable questions.
04 / What hiring managers look for
The checklist,
scored honestly.
Published hiring guidance for AI engineers and agent developers looks for shipped systems, evaluation evidence, documented tradeoffs and failure handling (Gauntlet AI, MyEngineeringPath, GAAS). Here is where this project stands against each, including what is missing.
| Signal | Status | Evidence |
|---|---|---|
| Shipped, working system | Partial | A working prototype with server-side logic and 61 tests, plus this lab running in your browser. No production users or customer deployments. Case study |
| Evaluation evidence | Shown | 16 written cases in six failure categories, run live in the scoreboard. Nine tests, including a property test and mutation checks. |
| Documented tradeoffs | Shown | Four decisions with their costs and the evidence that would change them, in the decision log. |
| Failure handling | Shown | Hostile text in a source, unauthorized and conflicting records, and stale responses (simulation). A blocked draft becomes an abstention. One limitation is published. |
| Cost and latency | Baseline only | Measured in your browser: median and 95th percentile per question, zero model calls, zero model cost. Not yet measured for a model. |
| Observability | Partial | Every answer shows its four-stage trace, and the activity log records reader, outcome and time. No production logging or alerting. |
| Real model, retrieval over a real corpus, or an agent | Not yet | Next step: put a model behind the draft stage and compare it with this baseline on the same cases. Large-corpus scale is not claimed. |
05 / Decision log
Choices, costs,
and what would change them.
1. Authorization and verification stay outside the model
Why: a model that is wrong or fed hostile text must not widen what a reader sees. Cost: an extra code path, and a correct draft can be blocked. Revisit when: never for authorization. The verifier loosens only with a failing case that shows a correct answer being blocked.
2. Weighted word matching instead of embeddings
Why: 13 records, deterministic results, no model or API cost, and every match can be explained. Cost: it misses paraphrases and cannot tell “cause” from “hours” (case C16). Revisit when: a pilot corpus reaches hundreds of documents or paraphrase misses appear. Then compare embeddings on the same cases.
3. Abstain rather than guess
Why: a wrong handover note costs the next crew more than no note. Cost: lower coverage, so answer rate is tracked beside correct abstention. Revisit when: reviewers report useful questions being refused, tested against a larger domain-reviewed set.
4. Quote sources before generating prose
Why: exact quotes make fidelity checkable without a second model. Cost: answers read less fluently. Revisit when: a model drafts prose. Then every generated claim must link to a supporting quote, and unsupported claims count as failures.
06 / Where the model fits
The model gets one job.
The rules stay outside it.
What this is
A deterministic baseline and an evaluation harness for the proposed grounded summary layer, with fictional records. It proves the checks work and gives any future model a fixed bar to clear. Results can be reproduced from the repository with npm run eval:ai-study.
What this is not
It is not a deployed language model, a retrieval pipeline over a real corpus, or an agent. Sixteen cases are a starting set, not a benchmark, and the retrieval thresholds were tuned against them. A pilot needs a domain-reviewed set built from the company’s own records, with agreed thresholds.
What I would measure with a model
Supported-claim rate, correct abstention, cross-permission leakage, citation fidelity, reviewer time per accepted draft, latency, and cost per accepted draft, each compared with this baseline.