02 — LOCAL AI · 2026
Orvyn
A voice assistant that never sends audio to the cloud: wake word, speech-to-text, a model on my own GPU, a synthesized voice, and Home Assistant to act.
- Built
- A voice assistant for my house with no cloud service in the loop.
- Stack
- Wake word, local speech-to-text, a language model on my own GPU, local text-to-speech, Home Assistant.
- State
- Runs on my own hardware; push-to-talk from Cleddi since 29 Sep 2026.
What it is
Orvyn is the assistant layer over my house. You talk to it and it turns on lights, answers questions and controls what Home Assistant controls. What matters is what is not in the loop: no cloud API, no account, no subscription, no audio leaving my network.
It has more than one body: a satellite in the lab (a small mic-and-speaker unit), a wrist puck with an animated face, and since 29 Sep 2026 a row at the top of Cleddi, my handheld: hold to talk, read the answer. They all speak to the same brain.
How it works
- Signal path
- Microphone, wake word, Whisper speech-to-text, a local model served by Ollama on an RTX GPU, Home Assistant tool calls, Piper text-to-speech, speaker.
- Wake word
- A trained 'hey Orvyn' on each satellite (openWakeWord in the lab, microWakeWord on the puck), so audio only flows after the word fires.
- Tools
- Home Assistant exposes the house as tools the model can call.
- Router
- The lab agent sorts each sentence before the model sees a tool list: a question gets 4 of its 23 tools, and a command with small talk gets somewhere to put the small talk.
- Gym
- A regression harness of scripted conversations I run against the model, its prompt and its Home Assistant tools, to see whether a prompt change made things better or worse.
By the numbers
- 112 cases
- scripted test conversations that prompt changes are checked against
- 7 of 50
- test conversations gave different results across three identical runs
What I learned
- 01
One run is not evidence
I built the gym to prove prompt changes worked, then found 7 of its first 50 cases (it now has 112) were flaky across three runs of the same suite. Every 'this fixed it' before that rested on a single run, which proves nothing. Nothing counts unless it holds across repeats.
- 02
A tool description must claim the request in its first sentence
A new tool would never get picked, however correct, until its description claimed the request up front. The model reads the first line and decides.
- 03
Negative examples are quotable
I wrote 'do not say X' into a system prompt and the assistant started saying X. If a phrase is in the prompt, it is in the model's mouth.
- 04
The measurement rig is part of the house
Gym runs drive real light switches. The daily scheduled run kept flipping lights around the house mid-morning, so the gym is manual-only now, on purpose.
What went wrong
- 01
It said it had done things it hadn't
Asked to act on something with no matching device, it received a raw error object and described it as success; shaping that reply into a sentence fixed it. The same defect had a bigger shape: in 'turn off the lights, and what's the score?', the chat half spent a tool call and starved the half that did the work. The sentence router took that case to 5 of 5.
- 02
Whisper hallucinates on silence
Ungated, it transcribed room noise as 'Wow.' over and over. A junk-phrase filter and a signal-to-noise gate fixed it.
