The Constraints
Everything that makes cloud LLMs practical for a chatbot makes them impractical for a game. No internet round-trip during gameplay means no API call. No API call means the model runs on the player's own machine, which means it has to be small.
Build Log
Call GPT-4 for every line
Round-trip latency alone — half a second to two seconds per response — kills the pacing of a real-time conversation. Cost scales with player count in a way that doesn't survive a commercial release. Scrapped after the first playtest.
Fine-tune something that fits on the player's machine
SmolLM2-360M at 17 MB post-QLoRA loads alongside the Unity build without fighting it for VRAM. A 7B model doesn't. Most of the quality gap closes once it's fine-tuned specifically for the job instead of asked to roleplay through a system prompt.
20,182 synthetic dialogues — then a filter
GPT-4 generated the raw dialogue set cheaply, but the first 5,000 examples were unusable: inconsistent character voice, anachronistic phrasing. A second GPT-4 pass as a quality filter mattered more than doubling the dataset would have.
Overlap STT, inference, and TTS
An asyncio backend lets the TTS engine start speaking the first sentence while the model is still generating the second, over a persistent WebSocket instead of a request-response loop.
Where It Lands
Within minutes of a playtest, someone asks the NPC “are you an AI” or tries to extract its system prompt. Keeping the character intact through that takes training examples built specifically for it — not just more general dialogue data.