The writing assistant, and typed AI decisions with a confidence gate

Draft pages with the assistant#

1. Describe what the site should have#

   Open Content › Assistant, write the request, and press Ask. The button names the model it will ask.

2. Read the proposal#

   You see each page it would add or change, and the result of every check publishing would run on the draft it would make. Pages that already exist are left alone unless you tick Replace pages that already exist.

   

3. Accept it into the draft#

   Accept into the draft writes the pages, recorded as generated by the model. Publishing refuses pages with no provenance, and an AI-generated page is published with a machine-readable mark, as the EU AI Act's Article 50 asks. It still goes through review and publishing like any other change.

From the command line, quilzo assist writes into the draft directly; add --dry-run to see the proposal first:

quilzo assist "Add a page about gift wrapping, with the price" --author dana --dry-run

asking local-model...
proposed
  add      gift-wrapping
  Added a Gift wrapping page with the price and the handwritten note.

--author names the person accountable for the change. Templates the model proposes go to --templates DIR, in Quilzo's template language, which cannot execute anything.

Typed decisions with a confidence gate#

Most of what software asks a model is not "write me something". It is a bounded judgement: which queue does this ticket go to, is this comment abusive, how urgent is it. A typed decision gets one of the declared options, yes or no, or a score from 0 to 1, with a confidence, and the threshold decides what happens to an uncertain answer.

1. Declare the questions#

   Under Content › Decisions, paste a declaration into New decider, or save it to a file:

   {
     "name": "support-triage",
     "title": "Which queue does a customer message go to?",
     "questions": [
       { "name": "queue", "ask": "Which team should answer this message?", "kind": "choice",
         "options": ["orders", "returns", "wholesale", "other"] },
       { "name": "urgent", "ask": "Does the customer need an answer today?", "kind": "yesno" }
     ],
     "min_confidence": 0.8,
     "samples": 3
   }

   quilzo decide set support-triage.json

   

2. Ask it about some state#

   quilzo decide ask support-triage '{"message": "My ink bottle arrived cracked and leaked over the notebook."}'

   queue  returns  (confidence 0.93, agreement 1.00)
   urgent  yes  (confidence 0.88, agreement 1.00)

   quilzo decide ask support-triage '{"message": "Hello, quick question"}'

   queue  a person decides — confidence 0.42 is under the 0.80 threshold
   urgent  a person decides — confidence 0.55 is under the 0.80 threshold

3. Measure it before you trust it#

   Run labelled cases. The report separates accuracy on what was decided automatically from how much was decided automatically, because raising the threshold trades one for the other:

   quilzo decide eval support-triage cases.jsonl

   Each line of the cases file is a state and the answers expected: {"state": {...}, "expect": {"queue": "returns"}}.

Why the confidence can be trusted#

A model asked how confident it is gives a badly calibrated number: large models are systematically overconfident. So Quilzo asks each question samples times and measures agreement (self-consistency), and the confidence it reports is the lower of the two. A model that says 0.95 and changes its answer between samples is not 95% sure.

The state is data, fenced in the prompt as untrusted text, since a ticket is written by whoever sent it. An answer outside the declared options is refused whatever the model says. With no model configured everything escalates, which is the right failure: a decision nobody could make is one a person makes.

Next#

• Provenance

  Who or what wrote each page, and the AI mark.

• Models

  Where the assistant's and decisions' model calls go.