Twitch AI Streams
A suite of fully autonomous, 24/7 AI-powered Twitch livestreams that collectively attracted over 200,000 followers and received international media coverage from NBC News, Vice, Kotaku, PC Gamer, The Wrap, and Quartz. Several streams went viral and became some of the most-watched AI-generated content on Twitch at the time.
The Streams
My Contributions
- Personality prompt engineering — Wrote the character prompts for roughly a third of the streamer personas, crafting the instructions and example dialogue that shaped each AI's tone, vocabulary, and mannerisms
- Voice cloning — Cloned character voices using ElevenLabs during the early phase of the project, before the team migrated to Play.ht
- Moderation infrastructure — Led the moderation effort end-to-end, managing a team of ~5 moderators running 24/7 coverage to catch anything the AI said that could get the channels banned. Over time I built out an automated flagging system that progressively took over the manual workload — surfacing borderline outputs for quick human review — until the pipeline was reliable enough to run without constant oversight
Media Coverage
Technical Architecture
- Text generation — OpenAI GPT-3 (early streams) and GPT-4 (later streams), with custom prompt engineering and per-character personality datasets to capture the voice and mannerisms of each persona
- Voice synthesis — Play.ht for AI text-to-speech combined with custom audio processing to match character voices
- Visual generation — Looping video footage of each character with real-time lip-sync driven by the audio output, built on open-source components
- Chat integration — Automated Twitch chat reading, question filtering, and moderation to select appropriate viewer questions and route them to the correct AI persona
- Memory system — Custom short-term and long-term memory modules that allowed AI personas to remember individual viewers and reference past interactions weeks later — a key differentiator from other AI streams at the time
The entire pipeline ran autonomously 24/7, with human moderators standing by primarily for content safety rather than content generation.
Reflections
This project was made not long after ChatGPT came out. One of the motivations was to show people the dangers of what could be done with this technology. The Trump vs Biden stream for example was an obvious parody with excessive use of profanity, with the goal of showing what is possible with the technology while making it obvious that it's not real. Many were entertained, but saw the potential for abuse of the technology for misinformation, while some actually thought it was real despite our best efforts to make it an obvious parody.