Contents

MonsterHouse shows how to use NVIDIA Nemotron models to power talking Halloween characters on a single workstation. The practical approach is to build voice interaction first, manage conversation state in application code, cache common responses, and then add animation and cameras.

Key Takeaways#

  • MonsterHouse combines speech recognition, a small language model, speech synthesis, and facial animation to turn a garage door into an interactive character.
  • According to its creator, median response latency was 1.70 seconds for generated speech and 0.72 seconds for cached responses.
  • The system uses one workstation with two professional GPUs, separating conversation processing from graphics and animation.
  • As of the October 30, 2026 article, the system had not been tested with real visitors, and the MonsterHouse code had not been packaged for release.

How the System Works#

MonsterHouse listens to visitors and responds as a grumpy building made from pirate ship timbers. Conversations follow a set progression. It initially turns visitors away, softens when they persist or say “please,” and offers a warm farewell when they say “thank you.” The application manages state to prevent restarting negotiations with someone it has already warmed to.

The speech pipeline consists of four main stages.

  • NVIDIA Nemotron 3 ASR converts visitors’ speech into text.
  • NVIDIA Nemotron 3.5 Bolt E4B 4GB generates replies.
  • NVIDIA Magpie TTS Zeroshot generates the character’s voice.
  • NVIDIA Audio2Face-3D drives mouth animation in Unreal Engine 5.

NVIDIA Nemotron Voice Agent orchestrates the conversation pipeline. Optional camera processing uses Nemotron 3 Nano Omni to add costume descriptions, and YOLO and ByteTrack to detect and track visitors.

Build the Conversation First#

Start with the Generic Assistant sample in Nemotron Voice Agent. Get microphone input, speech recognition, and spoken replies working before connecting a projector or rigging a character. This keeps the system you need to troubleshoot small.

Keep replies short and ready to read aloud, and clearly define conversation states such as “first-time visitor,” “negotiating,” “welcomed,” and “finished.” Store that state in the application. Prompts can describe personality, but code should control whether a visitor has already met the conditions for being welcomed.

The creator tested more than a dozen models before choosing the 4-bit version of Bolt E4B, then trained a personality LoRA using more than 5,000 reviewed multi-turn conversations. This is a later optimization, not a prerequisite for getting started. Begin with a system prompt and example conversations, and consider training only when repeated testing reveals problems that prompting cannot resolve.

Reduce Response Latency#

According to the creator, median latency from the end of a visitor’s speech to audible output was 1.70 seconds with Bolt Q4 and TTS, and 0.72 seconds with cached responses. End-of-speech detection currently waits for 300 milliseconds of silence. These measurements apply to this system and do not guarantee performance on other hardware.

There are two shortcuts to reduce processing. Prepared text skips fresh generation by the language model, while saved audio also skips speech synthesis. Streaming text that is ready to read aloud lets the character start speaking before the entire reply is complete.

If hardware is limited, use the generation pipeline offline to prepare a response library. During the event, speech recognition and a Python conversation router can select cached recordings. This reduces conversational flexibility, but it is a practical option when predictable response times matter more than open-ended dialogue.

Add Voice and Animation#

Magpie TTS Zeroshot generates speech from a short reference recording. MonsterHouse uses seven emotions, each with a 3–10-second audio sample. Use recordings you have permission to use, and check the latest access requirements before planning a local deployment. The original article does not clearly explain the steps for getting access.

Send unprocessed audio to Audio2Face-3D, and apply theatrical effects to the audio played through the speakers. This separation lets MonsterHouse preserve suitable input for animation while delivering a deep, reverberant voice to the audience. According to the creator, a 100-millisecond timing adjustment compensates for animation buffering.

The workstation assigns language and speech processing to an RTX PRO 6000 Blackwell Max-Q, and Unreal Engine and Audio2Face-3D to an RTX PRO 4000 Blackwell. This separation maintains the animation frame rate while conversation processing runs. “One workstation” should therefore not be taken to mean modest hardware requirements.

Check What Remains Unfinished#

The October 30 article presents a working prototype, but does not establish reliable unattended operation. Visitors gathering in the driveway, nighttime lighting, and costume recognition still needed testing. The five costumed sheep in the demo were synthetic test subjects, not evidence from a real Halloween event.

Automatic candy dispensing was also unfinished, and the custom MonsterHouse application had not been packaged for release. Treat the article as an implementation reference, rather than a complete installation guide.

Keep a manual fallback available, too. MonsterHouse has a microphone mode that bypasses recognition, dialogue generation, and speech synthesis. This lets the operator’s unprocessed voice drive the mouth while the audience hears processed audio.

Next Steps#

Run the Generic Assistant sample in Nemotron Voice Agent, conduct 20 short conversations on your intended hardware, and measure latency from the end of speech to audio output.

Source: https://huggingface.co/blog/nvidia/nemotron-halloween

Contents