Notes · 2026-10-01
A 421M model routes my assistant's requests in 42 ms
Eliza is a voice assistant that runs on my PC. Every request first needs one decision: is this small talk, a simple action she can do herself, a question back to me, or a hard task to hand to a stronger model? Until October a 9-billion-parameter model made that decision. It took about half a second and got 178 of 196 test requests right.
A model 20 times smaller now does it in 42 milliseconds and gets 193.
The model
Laya is a 421-million-parameter decision model (ModernBERT-large, open weights). It doesn't write text. You give it a question and named options, and it scores the options in one pass. That is exactly the shape of a routing decision.
Out of the box it was worse than what I had: 63 of 97. So I fine-tuned it.
The test came first
Before any training I wrote the test sets, and kept adding sets that the training data never saw:
- the assistant's existing routing cases;
- 43 new requests;
- follow-ups that only make sense with context ("do it again", "no, the other one");
- 22 of my own real requests, labelled by hand;
- two fresh sets written before later rounds of training data, never trained on.
A weakness found on a set the model had seen was checked on a set written afterwards. That is the only way I trust a number like "193 of 196".
Four rounds
| Round | Training examples | Right, of 196 |
|---|---|---|
| The 9B model | none | 178 |
| Round 1 | 792 | 115 of 128 (the sets that existed then) |
| Round 2 | 1,765 | 187 |
| Round 4 | about 5,000 | 193 |
The training data is generated by category from templates, never copied from test lines. Each round took 15 to 40 minutes per pass on an RTX 2080 Ti and used 11.1 GB of its memory, so the big model had to be unloaded while the small one trained.
A fifth round taught the same model quick decisions as well (is this reply a yes or a no: 49 of 52 right, never a false yes) and kept the routing (223 of 226 across seven sets). That is the one Eliza runs.
What it bought
- Routing: 42 ms against 432 to 470 ms.
- Instant commands. "Next song", "volume to 30", "close this tab" go from request to action in 43 to 47 ms. The big model needed 1.5 to 1.8 seconds just to choose its first tool.
- Memory. In half precision it takes about 0.8 GB, next to speech recognition, speech and the 9B model on one 11 GB card.
What it didn't
- It is sure even when it is wrong. Its probabilities sit near 1 on its misses, so "ask the big model when unsure" doesn't work as a safety net. The threshold had to be set from measured misses.
- Alone it made two bad hand-offs: it sent "open notepad and do it again but type 22222" and a plain "no" to the expensive model. Now Laya decides, and the 9B model double-checks only the hand-offs. Together: 224 of 225 on the full routing eval, at a median 1.2 seconds per request.
- In real use it agreed with the 9B model on about 90% of 355 requests. The test sets are cleaner than I am.
The general point
A large model is a fine first version of any decision. Once the decision has a fixed shape and you can write its test, a small model you trained yourself is faster, cheaper and, with honest test sets, more often right.