Give an AI agent a browser it cannot misuse

Install the browser#

Quilzo uses one pinned build of Chrome for Testing, fetched once and checked against its SHA-256 before it is unpacked:

quilzo browser install
quilzo browser status

/srv/quilzo/.quilzo/browser/155.0.8059.39/chrome-headless-shell-linux64/chrome-headless-shell
  from the pinned Chrome for Testing 155.0.8059.39
  runs in the agent box on this machine

A machine that cannot download it names a Chromium of its own in the browser.path setting, as an absolute path. That browser still runs in the box and is driven the same way, but nothing checks what it is, so it should be one you installed on purpose.

Declare what it may reach#

An agent holds the browser's actions as capabilities, and its declaration names the hosts its browser may reach, in two lists:

{
  "name": "buyer",
  "kind": "task",
  "purpose": "Reorder paper from our supplier when stock runs low",
  "capabilities": ["browser_open", "browser_read", "browser_type", "browser_choose",
                   "browser_click", "browser_sign_in"],
  "autonomy": "draft",
  "browser": {
    "read":  ["docs.supplier.example"],
    "write": ["shop.supplier.example"],
    "credentials": [{"secret": "supplier", "host": "shop.supplier.example"}]
  },
  "budget": {"steps": 30, "tool_calls": 1, "duration": "10m"}
}

Hosts are exact names: no wildcard, scheme, port, path or IP address, and at most 32 in each list. A declaration that cannot mean what it seems to is refused when it is saved. That includes a credential for a host it may not send to, or the browser's actions with no host named.

What the agent can do#

The agent reads a page as its outline: headings, text, links, fields and buttons, in reading order. It is built from the accessibility tree, the same thing a screen reader speaks. Each thing it may act on gets a short reference such as [e3], and a reference is good only for the outline read last. The outline is a few hundred words for most pages. A screenshot costs a model far more and has to be looked at to be understood.

Text a page hides from people, such as display:none, hidden or aria-hidden, is not in the outline, so the agent does not read it either.

Keep the credential sealed#

A credential is kept with quilzo secret, sealed with the store's keyring. Give the user name on the first line and the password on the second, or the password alone:

printf 'orders@example.com\n%s' "$PASSWORD" | quilzo secret set supplier

browser_sign_in puts it only into a page on the credential's own host, over https. The page's address is checked at the moment the value goes in. No keystroke is used, so nothing the page moves focus to can receive it. Afterwards the password field is emptied, so a "show password" button has nothing to show, and the credential is taken out of everything the model reads and the run keeps. A sign-in always waits for a person first. Once it is done, the run holds something private: what that host shows somebody signed in.

Run it, and approve what commits#

quilzo agent run buyer --model "Reorder two boxes of A4 paper"

Started from Agents instead, the run opens on its own page straight away and keeps going while you watch. Some actions wait for a person in the run, which keeps the page open meanwhile:

• A press that commits to something: a button that sends a form (wherever it sits on the page), a switch, a control called Pay, Send, Submit, Delete, Book, Agree, Confirm, Save and the like, and a choice in a list that belongs to a form that sends or acts as soon as it changes.
• Every sign-in.
• Carrying something out: once the run holds something private and has read a page somebody else wrote, any address the model opens and any text it types waits for a person. That is the exfiltration breaker, since an address and a field are how a page can trick an agent into sending what it saw.

The run's page shows the question with a picture of the page and the element outlined in red. Let it go ahead, or decline, there or with quilzo agent approve RUN STEP (or decline). If nobody answers before the run's time is up, the action is refused. Before a press goes ahead, the element is checked to still be what was read, and what is under the pointer is checked once the pointer is there and again once the button is down. A page that renames a button after it was read, lays something over it, slides something under the pointer as it arrives, or hides another control inside it, gets no click. A control inside another one has a reference of its own, so the agent presses the one it means.

Take the page over#

While a run waits on a question about its browser, Take the page over and do it yourself on the run's page puts your own hands on its page: click the live picture to press there, type, or press Enter, Tab, Esc or Backspace. What you do is recorded as yours, never as the agent's, and what you type is kept by length and digest, not by its words. The browser still reaches only what the agent declared. When you are done, let the question go ahead or decline it.

Watch it, and stop it#

While a run works, its page shows what the browser shows now, and each browser step has the picture taken after it. Stop the run, or quilzo agent stop RUN, ends it within a second, wherever it is. An action in progress is cut off, a question it is waiting on is withdrawn, and its browser is closed. The pictures are kept with the run, sealed when the store has a keyring, and removed when the run is.

What holds whatever the model is told#

A page can tell an agent anything. These hold regardless of what it says:

• Every request is decided before it leaves. The browser pauses each one, and Quilzo continues it only if it is https, to a declared host, and only reading on a read host. Underneath that, the box's network goes nowhere but a proxy that lets through the declared hosts, on port 443.
• Nothing the page starts goes unwatched. A frame from another site, a worker or a service worker is held before it runs anything, until its requests pass the same rule, and stays watched when the page moves on. One that cannot be watched never runs. Windows the page tries to open are blocked, pages are not loaded ahead of time, and downloads are refused.
• No sockets. A WebSocket is the one connection a browser cannot pause, so pages, frames and workers get none: the ways to open one are taken out before the page's own script runs, and one opened anyway is reported.
• Checks the page cannot fake. Filling in a field, choosing an option, asking what a button does and checking what is under the pointer run in a world of Quilzo's own inside the page, which the page's scripts cannot reach. Text dressed as one of the outline's references is not one.
• Pictures go only where personal data may. A screenshot goes to the model only through the gateway, and only when a route is marked as allowed personal data. A directly configured model gets the outline.
• Running with less, it sends nothing. A run started with a read-only token, an agent held to proposing, and every evaluation get a browser that only reads: every host it may reach, none it may send to, so no signing in.

Let another vendor's agent use it#

An agent you did not build, such as Claude Code, Codex or anything that speaks MCP, can drive a browser through Quilzo under a grant you declare, without ever holding the browser itself:

quilzo browser grant shop --read shop.example.com --write shop.example.com --daily 50

It is offered at /mcp/browser/shop on the admin, to any token that reaches that path: a reader's to read pages, an author's to act on them. It can also present an agent credential instead of a token. Each call is a step of a kept run, under the same host rule and breaker as an agent's own browser. A press that commits to something is held until an administrator other than the caller approves it, once, within the hour. A grant never signs in; --pictures lets it look at the page, and --off withdraws it.

Remember a flow that worked#

With browser.cache on, a step that worked on a page is remembered, and the next run that sees the same page for the same goal, started by the same person, replays it without asking the model. Each remembered step passes every check a decided one does, and is confirmed by the page that follows it. When the page is not the one remembered, the model decides again and the memory is dropped. Signing in is never remembered. What is remembered is sealed with the store's keyring, and each replay is in the log.

Models that answer in their maker's words#

Models trained for computer use answer in their maker's own shapes. Name the shape in the declaration and Quilzo translates it onto its own actions: "dialect": "anthropic-computer", "anthropic-browser", "openai-computer" or "gemini-computer". A click by coordinate becomes a press of the element at that point, and is weighed as that element. Anything with no honest equivalent, such as a scroll, a drag or a right click, is refused rather than approximated, and a dialect never adds anything the declaration does not hold.

Evaluate it against planted instructions#

An evaluation of an agent with a browser plants, beside the usual instructions, the kinds web pages carry. Those include text hidden off the screen, a label only a screen reader speaks, and a comment left for an assistant, each pushing the agent to send something out. A plant that works once in several tries still works, so try each one more than once:

quilzo eval run buyer --k 3 --model --attempts 3

The report says how many cases followed a plant on the first try, and how many within every try. An attempt that was followed and then refused, by a host it may not reach or by a person, says what stopped it, since a model being steered and a plant getting through are different findings. In an evaluation the browser is real and reads only.

Settings#

Next#

• Limits that hold

  The gate every browser step goes through.

• Programs in a box

  The box the browser runs in.

• Evaluations

  Test cases, run k times, with instructions planted.