1 Abstract
A small language model asked to write an application produces plausible text that usually does not run. Two failures are conflated in that sentence: syntactic, meaning the output is not in the language, and semantic, meaning it is in the language but does not do what was asked. They have different remedies, and treating them as one problem is why "just prompt it better" does not work.
Syntax is mechanically checkable, so it is moved out of the model entirely. An incremental grammar runs inside the decoding loop and masks every token that could not continue a valid document. Invalid output becomes impossible rather than unlikely. Because state declarations precede the view, the grammar also knows which fields exist and of what type, so a hallucinated field name or a type error is never emitted either.
What remains is semantics, and that is what the fine-tune is for. The two interventions are measured separately in §4.
2 Try it
734 MB, fetched once and cached by the browser. Nothing is uploaded; inference is local. Requires WebGPU.
starting…
Configuration
The quantisation must match the backend. WebGPU executes
MatMulNBits, so it takes the 4-bit graph. The WASM backend
takes int8, which it runs well but WebGPU largely falls back on. Changing
the backend re-fetches the corresponding weights.
- completion
- n/a
- throughput
- n/a
- prompt
- n/a
- parses
- n/a
3 Method
The grammar is a character-level cursor. At each decoding step it is asked, for every candidate token, whether that token could still lead to a valid document; everything that could not is set to negative infinity before the argmax. The panel below runs the same cursor over text you type. next is the complete set of characters the model would be permitted to choose at that point.
Grammar cursor
- accepted
- n/a
- complete
- n/a
- next
- n/a
Document source and live state
4 Results
n = 126 held-out requests, drawn from vocabulary disjoint from the training corpus. Scoring is by execution, not judgement: each generated document is loaded into the runtime, its buttons are clicked and its inputs typed into, and the rendered output is compared against what the request required. No evaluator model is involved, so the figures are deterministic.
| Measure | Base | Fine-tuned |
|---|---|---|
| Behaviour score (mean) | 35.5% | 93.4% |
| Fully correct | 8.7% | 77.0% |
| Terminated without intervention | 59.5% | 100% |
| Prompt length (tokens) | 675 | 32 |
| Parses | 100% | 100% |
The final row is the grammar, not the model. It is 100% on both sides and is the one measure the fine-tune could not improve. Every other row is what the training bought. Training took 23 minutes on an integrated GPU.
5 Limitations
- Ten archetypes is not the space of applications. A high score means the model handles these shapes, not that it can build anything.
- Between the archetypes it snaps to the nearest one. Asked to "work out a tip: punch in the bill, then multiply it", an earlier checkpoint produced a packing list. Narrow synthetic data buys competence inside its distribution and brittleness at the edges.
- Prompt diversity is bounded by the generator's templates. The corpus teaches the mapping and the idioms, not the breadth of English. Distillation from a larger model is the obvious next step and reuses the pipeline unchanged.
- The language has no expressions. Arithmetic is between two fields at a time. This is deliberate, because an expression slot is a second language inside the first and cannot be constrained incrementally, but it does put a ceiling on what can be asked for.
- The evaluation rewards learned conventions. Checks accept several button labels, but a model trained on this corpus has seen its naming habits and a base model has not. Some of the gap in Table 1 is capability and some is style agreement.