Improving Access to Voice-based Prototyping
Created a comprehensive toolkit enabling Google designers to prototype Assistant experiences without coding, reaching 400+ cross-functional users and powering 8000+ voice prototypes.
Background
Before voice-driven consumer products were ubiquitous, Google needed to rapidly scale the capabilities of its Assistant.
The core challenge: Designers couldn't prototype voice interactions and user flows without extensive engineering support, creating a bottleneck that slowed innovation and limited exploration.
Key challenges
- ā¢Limited prototyping capabilities for voice interactions
- ā¢Engineering support requirements creating bottlenecks
- ā¢Need for rapid iteration and testing of voice experiences
- ā¢Lack of tools for quick validation during design reviews
Outcomes
- ā¢Used by 400+ designers, engineers, and product managers across Google to synthesize 90K+ utterances in 8K+ voice prototypes
- ā¢Powered the speech recognition and synthesis in 300+ high-fidelity voice prototypes
- ā¢Enabled 5 critical user studies leading to 4 patents (USD879815S1, USD879816S1, USD881915S1, USD882612S1)
- ā¢Open-sourced tool for external partners
- ā¢Supported launch of 12 new features at I/O, including new languages and surfaces
Business Impact
Faster Innovation
- ā¢Teams could test ideas in hours instead of weeks
- ā¢By using familiar tools, designers could collaborate on prototypes with engineers and PMs
Quality Improvements
- ā¢Created shared language between design and engineering
- ā¢Early testing caught issues before engineering investment
Scale
- ā¢Enabled Assistant's rapid expansion to new languages and platforms
- ā¢Open-sourced to extend impact beyond Google and establish industry standards
Voice in Slides extension demo
I've recently rebuilt the Chrome Extension from scratch while experimenting with Claude.ai. If you install it, you can use these Slides to test. If you're interested in the code, you can find it on Github.
Solution
Created a comprehensive prototyping toolkit that let designers build functional voice interfaces without writing code. The core innovation was making complex voice technology accessible through familiar design tools.
Chrome Extension
Bridged gap between VUI and GUI work, accessible to non-technical users
- ā¢Built 'Voice in Slides' enabling designers to quickly prototype voice experiences using familiar tools
- ā¢Enabled voice prototyping in Google Slides with sharing capabilities
Unified Speech Service
Simplified complex voice processing for designers, enabling rapid prototyping of voice interactions
- ā¢Architected and built a speech server handling recognition (speech-to-text) and synthesis (text-to-speech)
- ā¢Managed bidirectional real-time audio streaming
- ā¢Coordinated microphone/speaker events
Component Library
Accelerated development of new Assistant experiences across platforms
- ā¢Built shared React components for Assistant's visual elements
- ā¢Enabled rapid prototyping across mobile, Google Home, and smart displays like the Lenovo Smart Clock
- ā¢Provided simple API and client library for designers without exposing complex voice processing
Assistant library demo
This Google I/O demo was creating using the libraries I built that manage voice input and output. It was featured at Google's largest developer conference.
Process
Research
- ā¢Shadowed designers to understand their workflows
- ā¢Identified need for quick validation during design reviews
- ā¢Analyzed existing tools and identified unfulfilled needs
Design Choices
- ā¢Real-Time Preview: Enabled designers to hear changes immediately, reducing back-and-forth
- ā¢Integrated with existing tools (Google Slides, Chrome Extensions, and Framer)
- ā¢Handled complex voice processing behind the scenes
Role
UX Engineer
Tools
Visual Studio Code, Google Cloud, Google Slides, Git, Google Chrome
Tech
Web Speech API, Google Text-to-Speech API, HTML, CSS, JavaScript