Improving Access to Voice-based Prototyping

Created a comprehensive toolkit enabling Google designers to prototype Assistant experiences without coding, reaching 250+ cross-functional users and powering 1000+ prototypes.

Background

Before voice-driven consumer products were ubiquitous, Google needed to rapidly scale the capabilities of its Assistant.

The core challenge: Designers couldn't prototype voice interactions and user flows without extensive engineering support, creating a bottleneck that slowed innovation and limited exploration.

Key challenges

  • add_circle
    Limited prototyping capabilities for voice interactions
  • add_circle
    Engineering support requirements creating bottlenecks
  • add_circle
    Need for rapid iteration and testing of voice experiences
  • add_circle
    Lack of tools for quick validation during design reviews

Outcomes

  • add_circle
    Used by >250 designers, engineers, and product managers across Google to prototype Assistant experiences
  • add_circle
    Powered the speech recognition and synthesis in >1,000 prototypes
  • add_circle
    Synthesized >90,000 utterances in >7,500 presentations
  • add_circle
    Enabled critical user studies leading to 4 patents
  • add_circle
    Open-sourced tool for external partners
  • add_circle
    Supported launch of 12 new features at I/O, including new languages and surfaces

Business Impact

Faster Innovation
  • add_circle
    Teams could test ideas in hours instead of weeks
Better Collaboration
  • add_circle
    Created shared language between design and engineering
Quality Improvements
  • add_circle
    Early testing caught issues before engineering investment
Scale
  • add_circle
    Enabled Assistant's rapid expansion to new languages and platforms
  • add_circle
    Open-sourced to extend impact beyond Google and establish industry standards

Demo of the "Voice in Slides" extension

I've recently rebuilt the Chrome extension Chrome Extensionfrom scratch while experimenting with Claude.ai. If you install it, you can use these Slides to test. If you're interested in the code, you can find it on Github.

Solution

Created a comprehensive prototyping toolkit that let designers build functional voice interfaces without writing code. The core innovation was making complex voice technology accessible through familiar design tools

Chrome Extension for rapid concept validation

Bridged gap between VUI and GUI work, accessible to non-technical users

  • add_circle
    Built 'Voice in Slides' enabling designers to quickly prototype voice experiences using familiar tools
  • add_circle
    Enabled voice prototyping in Google Slides with sharing capabilities
Unified Speech Service

Simplified complex voice processing for designers

  • add_circle
    Architected and built a speech server handling recognition (speech-to-text) and synthesis (text-to-speech)
  • add_circle
    Managed real-time audio streaming
  • add_circle
    Coordinated microphone/speaker events
  • add_circle
    Provided simple API for designers without exposing complex voice processing
Component Library

Consistent experience across platforms

  • add_circle
    Built shared React components for Assistant's visual elements
  • add_circle
    Enabled rapid prototyping across mobile, Google Home, and smart displays like the Lenovo Smart Clock

I am a demo title

I am a demo description (maybe I belong after the demo?)

I am a demo description (maybe I belong before the demo?)

Process

Research
  • add_circle
    Shadowed designers to understand workflow
  • add_circle
    Identified need for quick validation during design reviews
  • add_circle
    Analyzed existing tools and gaps
Design Choices
  • add_circle
    Real-Time Preview: Enabled designers to hear changes immediately, reducing back-and-forth
  • add_circle
    Integrated with existing tools (Google Slides, Chrome Extensions, and Framer)
  • add_circle
    Handled complex voice processing behind the scenes