I wrote an article a brace of months ago describing how I rebuilt my old sound aide pipeline in Platypush.
And I wrote before articles describing my journey through self-hosted sound assistants - in 2024 (with Picovoice), in 2020 (diving into the bequest assistant.google integration, the now defunct assistant.alexa integration, the now defunct Snowboy integration, the now defunct DeepSpeech integration and Picovoice) and the original part in 2019 describing the Google Assistant integration.
For a brace of months I used the sound pipeline described in my latest part as my chief driver, but following getting used to the robustness of business sound assistants I was unhappy alongside the elevated rates of false positives/negatives of my self-hosted pipeline, as fine as its deficiency of robustness in detecting start/end of speech, particularly in noisy environments and using cheap microphones.
After a bit of experimentation, I decided to toggle and fine tune a few parts, and following a brace of days of trying the results are satisfying adequate for me to compose a abbreviated part concerning it.
The resulting pipeline is:
Wake term detection: a new assistant.microwakeword plugin
Until lately I used assistant.openwakeword, which wraps openWakeWord.
It works, but the project has been quiescent for a during and all of its pre-trained models are v0.1 records from 2023, mostly trained on synthetic speech.
In custom I was getting the two false positives (the hotword triggering on backdrop noise) and false negatives (the hotword not detected unless I was near to the microphone).
I added a new plugin, assistant.microwakeword, on top of microWakeWord through the pymicro-wakeword Python bindings.
microWakeWord is the motor used by ESPHome and Home Assistant Voice, its models are actively maintained, and the built-in models (okay_nabu, hey_jarvis, hey_mycroft, alexa) container alongside the bundle — no download required.
The current authoritative models are provided under the ESPHome micro-wake-word-models repository.
Custom models are supported too: autumn a microWakeWord JSON manifest (with its .tflite model) in the plugin's models directory and citation it by name.
The plugin exposes the identical API as assistant.openwakeword (same models, detection_threshold, frame_duration, patience_frames, input_device, input_volume parameters, identical list_models action, identical HotwordDetectedEvent), so switching is a matter of fair changing the configuration to item to the new plugin:
assistant.microwakeword: models: - alexa
A few scheme notes:
- Without an definitive detection_threshold, all example uses the probability_cutoff from its own manifest (0.9 for alexa), which is the value the example was validated against.
- Models already gentle their output complete a sliding window, so dissimilar openwakeword, patience_frames defaults to 0 here.
- The audio thread never touches the TFLite models directly: it queues frames (and a reset marker) to the chief thread, since MicroWakeWord.reset() reloads the translator and isn't harmless to call concurrently.
- Like assistant.openwakeword, the plugin releases the audio equipment whenever another aide plugin starts a conversation, and re-acquires it whenever the conversation ends.
Speech-to-text: assistant.openai, alongside a much improved ear
assistant.openai records audio whenever a conversation starts and transcribes it through Whisper.
The tricky part has continually been end-of-speech detection: knowing whenever the person has completed talking. The former implementation used a per-frame boolean from WebRTC VAD, and any sole example classified as address would reset the silence timer. With equal a bit of backdrop noise, phrases would lone end whenever the maximum conversation duration was hit.
The audio preprocessing tier (AudioPreprocessor, shared by the STT integrations) has been reworked:
- Silero VAD as the chief backend (through onnxruntime, no PyTorch required), alongside WebRTC VAD and a plain RMS-based detector as fallbacks. To get an idea of Silero VAD's achievement compared against another solutions for address action detection, obtain a appearance at the insightful part One sound detector to regulation them all. The Silero example is downloaded on archetypal use into the Platypush workdir and checksum-verified:
assistant.openai: # car = silero -> webrtc -> energy, depending on what's installed vad_backend: auto
-
Hysteresis + sliding window: a example becomes address complete the VAD threshold and stays address until its mark drops by a margin, and the per-frame decisions are smoothed by bulk ballot complete a 0.5s window. A sole noise burst can no longer keep a conversation alive.
-
Adaptive noise floor: the backdrop noise flat is tracked complete time, and frames that aren't louder than noise_floor_factor (default 2.0, ~6 dB) times the flat are never considered as speech.
On a synthetic benchmark (Piper-generated phrases blended alongside babble and burst noise), the old settings kept conversations open for 6–10 seconds following the address had ended; the new pipeline ends them 1.0–1.6 seconds following the final word, which is the configured conversation_end_timeout affirmative the smoothing lag.
Since the changes on Platypush have been performed immediately in the AudioPreprocessor additionally used by another native speech-to-text engines, another speech-to-text plugins additionally advantage from them - e.g. if you use assistant.vosk for complete on-device detection.
Responses and context: the openai plugin
The transcribed expression is processed by the openai plugin, which handles the two the API key and the conversation context. By default the aide renders the reply through the configured tts_plugin; if the reply ends alongside a inquiry mark, the conversation continues alongside a follow-up.
Custom hooks can obstruct phrases before the default reply kicks in — for example, to grip residence automation commands locally without an LLM round-trip:
from platypush import run, when from platypush.message.event.assistant import SpeechRecognizedEvent @when(SpeechRecognizedEvent, phrase='play (the)? music') def play_music(): run('music.mpd.play')
And the hotword event is wired to the conversation commencement alongside a akin hook:
from platypush import run, when from platypush.message.event.assistant import HotwordDetectedEvent @when(HotwordDetectedEvent) def on_hotword(**_): run('assistant.openai.start_conversation')
Text-to-speech: tts.openai
Responses are rendered by tts.openai, alongside quantity and sound configurable. In my tests nevertheless tts.piper additionally yielded fine results, and it runs entirely locally. In my tests on a RaspberryPi 5 nevertheless it added ~3-4s of latency to procedure the reply to audio compared to tts.openai.
Since the TTS output goes to the speakers and the awaken term detector listens on the microphone, the aide plugins intermission hotword finding during a conversation is energetic — alternatively the aide would comprehend itself.
The complete configuration
openai: api_key: YOUR_API_KEY # Or, for a tradition setup using a example provider compatible alongside the OpenAI API, # e.g. a local Ollama instance: # base_url: http://localhost:8000/v1 # model: gpt-5 # Optional environment that can be passed to initialize the assistant # context: # role: system # content: You are a helpful assistant. assistant.microwakeword: models: - alexa assistant.openai: tts_plugin: tts.openai conversation_start_sound: ~/sounds/wake_word_detected.wav input_volume: 100 # assistant.vosk: # model: /path/to/vosk/model # conversation_start_sound: ~/sounds/wake_word_detected.wav # input_volume: 100 tts.openai: output_volume: 80.0 # Or, for a completely local setup that uses tts.piper: # tts.piper: # output_volume: 80.0
What changed on the code side
- New assistant.microwakeword plugin: microWakeWord hotword finding through pymicro-wakeword, identical API as assistant.openwakeword, built-in models bundled alongside the package, tradition models via JSON manifests.
- assistant.openwakeword fixes: frames are now 80ms (1280 samples, the size openWakeWord recommends) alternatively of 500ms — longer chunks lone come back the highest mark complete their sub-frames and can young female hotwords — affirmative a patience_frames indicator (consecutive frames complete threshold) and an definitive detection_threshold indicator (default 0.5).
- Shared AudioPreprocessor rework: Silero/WebRTC/energy VAD backends alongside vad_backend: car selection, threshold hysteresis, sliding-window bulk vote, and an adaptive noise floor. Used by the two assistant.openai and assistant.vosk.
- New Silero VAD wrapper: ONNX conclusion through onnxruntime, recurrent province carried throughout frames, checksum-verified example download into the workdir.
Caveats
The microphone on my RPi doesn't assistance 16 kHz natively, so audio is captured at 44.1 kHz and software-resampled — it works, but if finding feels sluggish application resampling is normally the archetypal suspect.
The VAD and threshold defaults were tuned on synthetic audio affirmative a brace of my audio recordings.
Different rooms, microphones and acoustics may output distinct results. You may desire to modify detection_threshold on the awaken term plugin and noise_floor_factor on the STT plugins if you don't awareness satisfied alongside your mileage.