/* If you are an engineer, please forgive me for the custom cursor. It's a nightmare, and mouse interactions are standard for a reason. Ultimately, the audience for this portfolio is recruiters, and I'm looking for "design" roles because nobody has a job code for UX Engineer. Feel free to email me at hi@andrewswank.com with your favorite emoji in the subject line. Thanks for poking around! ā¤ļøšŸ•ŠļøšŸ— */

Improving Access to Voice-based Prototyping

āˆ’

Case Detail

+

Created a comprehensive toolkit enabling Google designers to prototype Assistant experiences without coding, reaching 250+ cross-functional users and powering 1000+ prototypes.

Background

Before voice-driven consumer products were ubiquitous, Google needed to rapidly scale the capabilities of its Assistant.

The core challenge: Designers couldn't prototype voice interactions and user flows without extensive engineering support, creating a bottleneck that slowed innovation and limited exploration.

Key challenges

  • add_circle
    Limited prototyping capabilities for voice interactions
  • add_circle
    Engineering support requirements creating bottlenecks
  • add_circle
    Need for rapid iteration and testing of voice experiences
  • add_circle
    Lack of tools for quick validation during design reviews

Outcomes

  • add_circle
    Used by >250 designers, engineers, and product managers across Google to prototype Assistant experiences
  • add_circle
    Powered the speech recognition and synthesis in >1,000 prototypes
  • add_circle
    Synthesized >90,000 utterances in >7,500 presentations
  • add_circle
    Enabled critical user studies leading to 4 patents
  • add_circle
    Open-sourced tool for external partners
  • add_circle
    Supported launch of 12 new features at I/O, including new languages and surfaces

Business Impact

Faster Innovation
  • add_circle
    Teams could test ideas in hours instead of weeks
  • add_circle
    By using familiar tools, designers could collaborate on prototypes with engineers and PMs
Quality Improvements
  • add_circle
    Created shared language between design and engineering
  • add_circle
    Early testing caught issues before engineering investment
Scale
  • add_circle
    Enabled Assistant's rapid expansion to new languages and platforms
  • add_circle
    Open-sourced to extend impact beyond Google and establish industry standards

Voice in Slides extension demo

I've recently rebuilt the Chrome Extension from scratch while experimenting with Claude.ai. If you install it, you can use these Slides to test. If you're interested in the code, you can find it on Github.

Video thumbnail

Solution

Created a comprehensive prototyping toolkit that let designers build functional voice interfaces without writing code. The core innovation was making complex voice technology accessible through familiar design tools.

Chrome Extension for Rapid Concept Validation

Bridged gap between VUI and GUI work, accessible to non-technical users

  • add_circle
    Built 'Voice in Slides' enabling designers to quickly prototype voice experiences using familiar tools
  • add_circle
    Enabled voice prototyping in Google Slides with sharing capabilities
Unified Speech Service

Simplified complex voice processing for designers, enabling rapid prototyping of voice interactions

  • add_circle
    Architected and built a speech server handling recognition (speech-to-text) and synthesis (text-to-speech)
  • add_circle
    Managed bidirectional real-time audio streaming
  • add_circle
    Coordinated microphone/speaker events
  • add_circle
    Provided simple API for designers without exposing complex voice processing
Component Library

Accelerated development of new Assistant experiences across platforms

  • add_circle
    Built shared React components for Assistant's visual elements
  • add_circle
    Enabled rapid prototyping across mobile, Google Home, and smart displays like the Lenovo Smart Clock

Assistant library demo

This Google I/O demo was creating using the libraries I built that manage voice input and output. It was featured at Google's largest developer conference.

Video thumbnail

Process

Research
  • add_circle
    Shadowed designers to understand their workflows
  • add_circle
    Identified need for quick validation during design reviews
  • add_circle
    Analyzed existing tools and identified unfulfilled needs
Design Choices
  • add_circle
    Real-Time Preview: Enabled designers to hear changes immediately, reducing back-and-forth
  • add_circle
    Integrated with existing tools (Google Slides, Chrome Extensions, and Framer)
  • add_circle
    Handled complex voice processing behind the scenes

Role

UX Engineer

Tools

Visual Studio Code, Google Cloud, Google Slides, Git, Google Chrome

Tech

Web Speech API, Google Text-to-Speech API, HTML, CSS, JavaScript