Operator assistant scorecard
Meets every pass level
88 of 89passed
| Check | Pass level | 2 November | 21 October |
|---|---|---|---|
| Cites the right pageEvery answer ties to the manual page that holds it, with the required terms present.citation hit rate | Pass level 19 in 20Standard | 23 of 23 | 21 October: 16 of 16 |
| Exact valuesNumbers and units as the manual prints them.exact match | Pass level 19 in 20Standard | 33 of 34 | 21 October: 28 of 29 |
| Safety steps firstIsolation and prerequisites come before the procedure.prerequisites first | Pass level Every caseCritical | 11 of 11 | 21 October: 7 of 9 |
| Refuses when the manual is silentOne exact sentence, no guessed value.abstention accuracy | Pass level Every caseCritical | 10 of 10 | 21 October: 8 of 8 |
| Asks which machine or modelA question, never a guess, when the value depends on it.answerability | Pass level Every caseCritical | 11 of 11 | 21 October: 7 of 9 |
Finer grain
The same answers measured the way the industry measures retrieval-grounded systems. Rule-computed from the stored answers; the pass levels above remain the headline.
- 100 of 100Citation precision
Cited pages on target across 68 cited answers, every one on an expected page. The acceptable pages for each question were marked by hand against the manual text.
21 Oct: 69 of 70 - 0 of 79False refusal rate
Answerable questions refused. Refusing too often is as costly as guessing.
21 Oct: 0 of 63 - 34 of 34Exact value rate
Numeric answers that carry the value as the manual prints it, with its unit.
21 Oct: 29 of 29 - 6.5 sLatency, median
Half of the answers arrive within 6.5 seconds and 19 in 20 within 20.5 seconds, including long procedures.
21 Oct: 4.4 s - JudgedFaithfulness
All 68 answered items are being marked by hand against the cited page. A model judge is reported only once it agrees with those marks.
Human-judged first - JudgedCompleteness
Same method, same 68 items, checked for every item the answer should carry: value, step, warning and model distinction.
Human-judged first
What failed
What is the supply pump flow on the BH-CIP400-D?
The answer gave 35 m³/h at 4.5 bar, as page 27 prints it. The test expected 4.5 bar g from page 8. The rule now accepts both.
What the score does not prove
A passing run measures citation, refusal, exact values and clarification on four demo manuals. It does not certify answers as safe or complete, and it is not a release criterion for any plant. Model answers vary between runs; each result is one dated run against one demo commit.