Quilzomanual

Evolve: search over programs with a sealed evaluator

A model proposes changes, each program runs in a box, and an evaluator it cannot reach decides which survive.

In the adminnone; the command line

Evolve searches for a better program the way AlphaEvolve does: a model proposes changes to the best programs so far, each changed program runs, an evaluator scores what it wrote, and the best are kept for the next generation. In Quilzo the program and the evaluator run in separate boxes, so a program can never read the cases it is scored on or write its own score.

Try it on the demonstration

quilzo demo writes a packing bench to examples/evolve/: a program that packs a day's orders into 2 kg parcels first fit, which takes six parcels where five will do, and a judge that checks a packing and counts its parcels. With a model set up on the Models screen:

quilzo evolve run --program examples/evolve/pack.py \
  --evaluator examples/evolve/judge.py \
  --input examples/evolve/orders.json --eval-input examples/evolve/orders.json \
  --test-input examples/evolve/test/orders.json \
  --test-eval-input examples/evolve/test/orders.json \
  --weight parcels=-1 --generations 10 \
  --task "Pack the orders into as few 2 kg parcels as possible"

Each generation prints a line as it is recorded, and the end prints the best program's scores and where it was written.

The program and the evaluator

SideReceivesWrites
The program (--program, Python, R or Julia)Its --input files, by nameFiles in output/, at most 16, each at most 1 MiB
The evaluator (--evaluator)The program's files in candidate/, and its own --eval-input filesoutput/metrics.json: one flat object of numbers, such as {"parcels": 5}

The seed program may mark the part to change between # EVOLVE-BLOCK-START and # EVOLVE-BLOCK-END; without markers the whole program changes. The model answers with SEARCH/REPLACE blocks applied to the program it was shown first.

The score is the mean of the metrics, or with --weight NAME=NUMBER their weighted sum; a negative weight counts against, so --weight parcels=-1 asks for fewer parcels and --weight accuracy=1 --weight time=-0.1 trades a little speed for accuracy.

What the seal does, and what it does not

Each run gets a box with no network, its own empty workspace and home, and limits on time (--timeout), memory per process (--memory) and disk (--disk). The only thing that crosses from a program to the evaluator is the files it wrote in output/, and a run that goes over a limit is not scored at all. Keep the evaluator and its files outside /usr, /lib, /bin, /sbin and /etc, which every box can read; a search refuses them there. The evaluator must read what a program wrote as data and never run it, or the program would run beside the evaluator's cases.

The search still adapts to the evaluator's cases: their scores decide what survives, for every generation, so the best program's score says how well it fits those cases. A test set says how it does on others: --test-input files the programs run on and --test-eval-input files the evaluator judges with, used once on the final population and never to choose. Report the test score, not the selection score.

The record

Each search has a directory under the store's evolve/ with journal.jsonl, one line per generation written before the population changes: the diff the model proposed, the program's digest, how its run ended, its metrics and whether it was kept. The first line holds the random seed (--seed, or one chosen and recorded) and the model; the same seed makes the same choices of which program to change. Every program that ran is kept in programs/ by its digest, and the best in best.py or --out, written only when the search succeeds. The audit log records the start, with the digests of the program, the evaluator and its cases, and the end, with the best program, its scores and the journal's digest. --json prints the result as JSON.

A search needs the use-models permission, since every generation is a model call, and is charged to the model gateway as consumer evolve.

Next