A one-day experiment with local AI, a gaming PC, and a lot of "wait, why did it do that?"
The idea
I play story games, the slow, cinematic kind. While I play, I often want to ask Claude something or give it a small job: "note that down", "fix that bug when you get a minute", "what's on my screen?" But the moment I alt-tab out of a game, I lose the mood.
So I wondered: what if I could just talk? No keyboard, no switching windows. A little program that sits on my PC, listens all the time, and when I say something, it either handles it itself or passes my exact words to Claude.
I'm not an AI engineer. I had a gaming PC with a 16 GB graphics card, a Bluetooth headset, and a lot of curiosity. This is the story of one day of building it with Claude's help, including the parts that went wrong.
First, how a voice assistant even works
If you've never looked inside one, it's less magic than it sounds. It's a chain of small helpers, each doing one job:
- 1.
The ears (voice detection). A tiny program that only answers: "is someone talking right now?" It cuts the endless stream of microphone sound into sentences.
- 2.
The transcriber (speech to text). Turns each sentence of audio into written words. I used Whisper, a free model from OpenAI that runs on your own computer.
- 3.
The brain (a language model). Reads the words and decides what to do. I ran a model called Qwen 9B through a tool named Ollama. "9B" means 9 billion adjustable numbers inside it, which sounds huge, but the big commercial models are many times bigger.
- 4.
The mouth (text to speech). Turns the reply back into a voice.
- 5.
The memory. A small database for notes and facts about me.
Everything runs locally: no audio leaves my PC. That was a deliberate choice. It also meant I had to live with whatever my own graphics card could handle.
Chapter 1: A working skeleton, and a graphics card that wouldn't cooperate
The first version came together fast: microphone, transcriber, a few rules, a web page on my second monitor showing a live transcript. It could pause and resume, take notes, and remember things.
Then I tried to measure how fast it ran. The numbers were terrible. The culprit: I was playing Spider-Man at the same time, and the game was using about 95% of the graphics card. My benchmark was fighting my game for the same chip.
That was the first hint of the central problem of the whole project: a game and an AI model both want the same graphics card.
Chapter 2: The day it heard itself
This was the scariest moment of the day.
I tested the assistant with its voice coming out of my speakers. The microphone picked up the assistant's own voice. The brain read its own words as my commands and started a real Claude session on its own, which then tried to search my email and files. (It was blocked, because background Claude sessions don't have those permissions. Lucky.)
It's a perfect example of why voice assistants are tricky: a microphone can't tell whose voice it's hearing.
I fixed it in layers, because no single fix is enough:
The agent now remembers what it just said, and ignores speech that matches it.
It never guesses where to send something. I choose the Claude session out loud, or say "new session".
If the brain only thinks I wanted something sent, it asks "send this to Claude?" and waits for my yes.
A message waits two seconds before leaving, so I can cancel it.
The lesson I took away: if a program can do real things, assume it will sometimes hear garbage, and design for that.
Chapter 3: Teaching it to hear me
An important detail: I mix Hindi and English in the same sentence, the way many people in India do. Most speech tools struggle with that.
Instead of guessing which settings were best, I recorded 25 short clips of myself, then replayed them through different configurations and counted how many words came out wrong. The results surprised me:
A medium-sized Whisper model beat the largest ones. Bigger wasn't better. The best setup got about 21% of words wrong, against 52% for my first attempt.
Giving Whisper a short cheat sheet of words I use ("Claude", "VS Code", "game mode"...) helped noticeably.
Whisper sometimes invents text out of silence or noise. "Thank you." and "thank you for watching" appeared from nowhere, because it was trained on lots of video subtitles. I had to add filters for those.
Whisper often heard my command "stop" as "off", "top" or "stuff". So I made those count as "stop" too, following a rule I set for the project: when in doubt, stop.
Chapter 4: Finding a voice that doesn't sound like a robot
The built-in Windows voice sounded awful. I tried Kokoro, a small free voice, which was much better but still flat. Along the way I learned something new: the Hindi voices could only read Devanagari (the Hindi script). Hindi typed in English letters, "kya haal hai", came out as nonsense. So now the assistant writes Hindi in Devanagari and English in English, and picks the voice for each sentence.
Then I told Claude to use as much of my graphics card as it wanted (more on that in a moment), and I moved to a bigger voice model called Chatterbox. It sounded much smoother, but it took real work to tame:
It needed a newer version of its software to run on my new graphics card.
With its default settings, it rambled in Hindi.
It crashed on tiny scraps of text like "1.".
After a settings search and some guard rails, it became stable. I wanted a female voice, so I generated clean reference clips from synthetic female voices. Chatterbox can imitate the voice in a short clip, which also hints at how you could give an assistant a custom voice. (Cloning a voice is something I only want to do with a person's clear consent, and for personal use.)
Chapter 5: A rule I set on purpose
At one point I told Claude: "Don't give it delete access, for anything, even if I ask."
An assistant that listens through a microphone, makes mistakes, and can run commands is exactly the kind of tool that should never be able to delete things. I enforced it in several places: no code path deletes anything, memory only ever adds, spoken delete requests are politely refused, and every Claude run gets a list of blocked delete commands. A test even scans the source code looking for any way to delete.
If you build something like this, I'd copy this idea: decide what it must never do, and make that impossible instead of unlikely.
Chapter 6: Changing the goal halfway
For the first half of the day, I wanted the assistant to run alongside a game without stuttering it. That was hard: a game can use most of the graphics card, and every model I wanted needed some. I tried a "game mode" that moved things to the processor, but the processor was too slow (16 seconds to make one decision) and the audio got distorted.
So I changed the goal. I said: forget the game, build the best assistant possible. That one decision unlocked everything: the bigger transcriber, the smoother voice, and a model that stays loaded, ready to answer. Sometimes the best fix for a hard constraint is to question the constraint.
Chapter 7: "Why does it feel unintelligent?"
By evening it could talk, answer simple questions, and sound pleasant. But when I really tried to chat with it, it felt dumb. Looking at the logs, I found real reasons:
It was throwing away my questions. The anti-echo logic ignored anything I said while it was still talking, or half a second after. So "what's the capital of France?" simply vanished.
"Stop" didn't stop it. While it talked, its own voice and the room noise meant the microphone never found a quiet gap, so my "stop" waited in line. It stopped at ten when I wanted it to stop at four.
Noise stretched everything. Background noise kept the voice detector "on" for up to 14 seconds, so one sentence reached the brain 14 seconds late.
One misheard word became a wrong answer. I said "capital", it heard "total", and the brain obediently tried to add France to 17. Nothing asked "did you mean...?"
The brain is small. A 9-billion-number model on my PC isn't Claude or Gemini. It has no internet and can't reason as well.
I fixed what I could: stop words now work over its voice, the things I say over it get answered, and it looks back at earlier questions. In my noisy room, interrupting is still imperfect. My decision: treat spoken replies like tasks in a queue, and make sure that "stop" reliably ends a running Claude session. That part works.
Did I reinvent the wheel?
Partly, yes. I asked Claude to check for open-source projects, and a framework called Pipecat does much of the hard "listening and turn-taking" work, and RealtimeVoiceChat does something similar locally. I should have looked sooner. What those projects don't give you are the parts specific to me: choosing Claude sessions myself, the no-delete rule, Hindi-English tuning for my voice and headset, and the chosen voice.
What I learned
- 1.
Measure, don't guess. Recording my own voice and counting wrong words changed my choices more than any advice did.
- 2.
A bigger model isn't always better. For listening, the medium one won.
- 3.
Every layer makes mistakes, and they pile up. A mishearing becomes a wrong answer, which becomes an odd reply. Fix the early layers first.
- 4.
Turn-taking is most of what feels "smart". Timing, interrupting, and not dropping what you said matter as much as the answers.
- 5.
Decide what the tool must never do, and enforce it.
- 6.
Shared hardware is a real constraint. A game and an AI model fight over the same chip.
- 7.
Your headset matters. A Bluetooth headset mic is a poor microphone for this.
Where it stands, and what's next
It listens, pauses and resumes, takes notes, remembers things, reads the screen, chats, speaks in a natural voice, and can pass my words to Claude and stop a running session. It's not yet smart, and it can't yet control the Claude desktop app, VS Code or the browser.
Next up: a smarter brain (probably a hybrid, where the local model handles quick chat and Claude handles hard questions), a noise-aware voice detector, asking "did you mean...?" when it isn't sure, and a better microphone.
It took about eight hours to go from "wouldn't it be cool" to "okay, now I understand why this is hard." That's the part I enjoyed most.
I built this with Claude, which wrote most of the code while I made the decisions and did the live testing, speaking into a microphone and reading logs together.
Don't trust the marketing on Instagram, don't trust small fry who say they have made Jarvis from Iron Man, things are not there yet. Try it yourself.