06 — AI AGENTS · 2026
Cleddi
A pocket remote for my AI coding agents: every running session, each question in full, approve or deny, and push-to-talk replies.
- Built
- A handheld remote for my AI coding sessions: the firmware, the case and the PC bridge.
- Stack
- ESP32-S3 touch board, C++ firmware, Bluetooth LE, Wi-Fi with HMAC authentication, a Python bridge, local speech-to-text.
- State
- Carried daily; desk dock designed.
What it is
A handheld for running AI coding sessions from across the room. It lists every running agent session, jumps to whichever one finishes or asks something, shows the question in full, approves or denies each action the agent asks to take, and takes push-to-talk replies that the PC types into the right session.
The firmware and the PC bridge are my own. The top row of the list is Orvyn, the house assistant, so the same handheld runs the coding agents and the lab. An Agents tab opens any of my agent desks on the PC: tap one, then confirm. It hangs on the Agent Wall when it's not in my hand.
How it works
- Board
- A 2.8-inch ESP32-S3 touch board with a microphone and a codec.
- Approvals
- Over Bluetooth LE to the coding agent's desktop app.
- Sessions and chat
- Over Wi-Fi to a Python bridge on the PC, with mutual HMAC authentication on the link.
- Speech
- Push-to-talk audio streamed to the PC and transcribed locally with Parakeet via sherpa-onnx, about 0.2 s for a 4-second clip on the CPU.
- Sound
- Synthesized bell cues for approvals, questions and finished sessions, with a quiet night mode.
- Enclosure
- A printed case with two capacitive touch pads under white caps: left is back, right is talk and confirm. Short and long presses, no buttons to hunt for.
By the numbers
- 0.2 s
- to transcribe a 4-second push-to-talk clip on the PC's CPU
- 0.5 s
- to jump to a session that just asked a question
- ~6 s
- from wake to linked, down from 20–45
What I learned
- 01
Measure the clip before theorizing
When replies started coming back garbled, the bridge began saving every push-to-talk clip with its levels. The microphone was fine; the bad clips lined up with Wi-Fi dropping out. One task was reading the mic and sending the audio, so a stalled send starved the mic. It is now a recorder with a 10-second buffer and a separate sender. Every clip of my voice since then has arrived whole.
- 02
Follow the work, let me pin it
The handheld jumps to whichever session finishes or asks something; a pin stops it. That one rule made it usable while walking around.
- 03
Count pushed frames, not drawn ones
Bench screenshots read the frame buffer, so they looked right while the real screen was frozen. I now check that a frame was actually pushed (loop time, a push counter), not just that it was drawn.
What went wrong
- 01
One shadowed variable froze the screen
After the settings page went in, the glass stuck on the boot splash. I chased deep sleep, the display reset, DMA, memory, sound and haptics. The cause was a new brightness variable named 'full' that hid the flag meaning 'a frame was drawn, push it', so no full frame reached the glass. Big UI edits now get a build with shadow warnings on.
- 02
The amplifier enable was backwards
The amp's enable pin on this board is active low. My earlier orb firmware on the same board drove it high, which shuts the amp off, and I had written up the result as a speaker 24 dB under the room. The board's own mic settled it: a test tone at -34 dBFS with the pin low, nothing with it high.
- 03
The dock's '5V' pin was not 5 V
The labeled pin on the UART header is the system rail, not USB power, so the pogo dock has to feed the USB input through a diode. Caught in design, before anything was printed.
