Case study
Eliza: a voice assistant that drives a Windows PC
I say what I want; she does it on my computer, asks before anything risky, and tells me when it is done. Sixteen days from the first commit to what is described here.
The ripple follows the voice it hears.
The sound stays in your browser. Nothing is recorded or sent.
- lines of Python
- 34,000
- automated tests
- 1,370+
- commits in 16 days
- 270+
- graphics card, 11 GB
- 1
What she is made of
- Ears and voice
- Push-to-talk or the phrase "wake up Eliza". Speech recognition and speech both run on the PC; nothing spoken leaves it.
- Brain
- A small model I fine-tuned sorts each request in 42 ms. Everyday requests go to a local 9B model; hard ones go to Claude in the background while she keeps answering.
- Hands
- She opens apps, presses buttons by name, reads the screen and the terminal, and checks the screen before saying "done".
- The safety gate
- Every action passes a gate written in code, not in a prompt. Commands and deleting files ask first. Erasing disks, reading keys or passwords and a few more are refused whatever anyone says.
- Phone
- A Telegram bot paired by a code shown on the PC: requests, a screenshot, Allow and Deny buttons, voice notes both ways.
- Memory
- An Obsidian vault she writes to and searches: what she learned, what went wrong, and how she did each task the last time.
- Face
- A black pill at the top of the screen that grows with her answer, and a full window with the orb below. React, Three.js, Electron.
- Dashboard
- A dense live screen for her paper-trading research, with charts written by hand.
Measured, not estimated
Nothing in this project counts until a test or a measurement says so. Test sets are written before the change they judge, and some are never trained on.
| What | Result | Against |
|---|---|---|
| Routing a request (which kind of work is it?) | 193 of 196 right, 42 ms | The 9B model it replaced: 178 of 196, 432 to 470 ms |
| Instant commands ("next song", "volume to 30") | 43 to 47 ms to the action | Before: 1.5 to 1.8 s just to choose a tool |
| The wake phrase | 30 of 30 caught, 0 of 72 false starts | Synthetic voices with noise and music; my own room is still to be measured |
| Speech to text | 4.7% of words wrong, 109 ms | Was 6.7% before a vocabulary prompt and beam search |
| Yes or no to a confirmation | 49 of 52 right, never a false yes | Word rules before: 35 of 52, two false yeses |
| Everything on one graphics card | 10.8 of 11.3 GB | An 11 GB RTX 2080 Ti from 2018 |
The routing work has its own note: A 421M model routes my assistant's requests in 42 ms.
What didn't work
- She is not yet as fast as I want in real use. In test conditions a spoken reply starts in 0.8 seconds. In my own logs the first audio of a voice request comes after a median of 6 seconds. The target was under one.
- Georgian. I tried to let her take requests in my own language. The translation step got 2 of 7 real requests right, so it was dropped. She works in English.
- Her trading research found no strategy that beats its costs. Over a hundred ideas were tested on data they hadn't seen, with fake money. None passed. Two write-ups came out of it: what a model was really picking and who does make money in those pools.
- The graphics card is full. 10.8 of 11.3 GB with everything loaded. A browser using the card can push the main model onto the CPU.
How it was built
With an AI coding agent. I decide what gets built and in what order, read every change, and run it on my own machine before it is kept. The agent does most of the typing. That is how one person ships this much in two weeks, and it is how I work for clients.
- Small steps, each one runnable and tested before the next.
- Rules that matter are enforced in code. A prompt can ask; only the gate decides.
- Honest numbers, including the ones above that I would rather were better.
Want something like a part of her?
The phone bot, the orb and the dashboard are each a week of work for your product. See the three packages, or join the list if you would want Eliza herself on your PC.