/* If you are an engineer, please forgive me for the custom cursor. It's a nightmare, and mouse interactions are standard for a reason. Ultimately, the audience for this portfolio is recruiters, and I'm looking for "design" roles because nobody has a job code for UX Engineer. Feel free to email me at [email protected] with your favorite emoji in the subject line. Thanks for poking around! ā¤ļøšŸ•ŠļøšŸ— */

Improving Access to Voice-based Prototyping

Created a comprehensive toolkit enabling Google designers to prototype Assistant experiences without coding, reaching 400+ cross-functional users and powering 8000+ voice prototypes.

Background

Before voice-driven consumer products were ubiquitous, Google needed to rapidly scale the capabilities of its Assistant.

The core challenge: Designers couldn't prototype voice interactions and user flows without extensive engineering support, creating a bottleneck that slowed innovation and limited exploration.

Key challenges

  • •Limited prototyping capabilities for voice interactions
  • •Engineering support requirements creating bottlenecks
  • •Need for rapid iteration and testing of voice experiences
  • •Lack of tools for quick validation during design reviews

Outcomes

  • •Used by 400+ designers, engineers, and product managers across Google to synthesize 90K+ utterances in 8K+ voice prototypes
  • •Powered the speech recognition and synthesis in 300+ high-fidelity voice prototypes
  • •Enabled 5 critical user studies leading to 4 patents (USD879815S1, USD879816S1, USD881915S1, USD882612S1)
  • •Open-sourced tool for external partners
  • •Supported launch of 12 new features at I/O, including new languages and surfaces

Business Impact

Faster Innovation
  • •Teams could test ideas in hours instead of weeks
  • •By using familiar tools, designers could collaborate on prototypes with engineers and PMs
Quality Improvements
  • •Created shared language between design and engineering
  • •Early testing caught issues before engineering investment
Scale
  • •Enabled Assistant's rapid expansion to new languages and platforms
  • •Open-sourced to extend impact beyond Google and establish industry standards

Voice in Slides extension demo

I've recently rebuilt the Chrome Extension from scratch while experimenting with Claude.ai. If you install it, you can use these Slides to test. If you're interested in the code, you can find it on Github.

Video thumbnail

Solution

Created a comprehensive prototyping toolkit that let designers build functional voice interfaces without writing code. The core innovation was making complex voice technology accessible through familiar design tools.

Chrome Extension

Bridged gap between VUI and GUI work, accessible to non-technical users

  • •Built 'Voice in Slides' enabling designers to quickly prototype voice experiences using familiar tools
  • •Enabled voice prototyping in Google Slides with sharing capabilities
Unified Speech Service

Simplified complex voice processing for designers, enabling rapid prototyping of voice interactions

  • •Architected and built a speech server handling recognition (speech-to-text) and synthesis (text-to-speech)
  • •Managed bidirectional real-time audio streaming
  • •Coordinated microphone/speaker events
Component Library

Accelerated development of new Assistant experiences across platforms

  • •Built shared React components for Assistant's visual elements
  • •Enabled rapid prototyping across mobile, Google Home, and smart displays like the Lenovo Smart Clock
  • •Provided simple API and client library for designers without exposing complex voice processing

Assistant library demo

This Google I/O demo was creating using the libraries I built that manage voice input and output. It was featured at Google's largest developer conference.

Video thumbnail

Process

Research
  • •Shadowed designers to understand their workflows
  • •Identified need for quick validation during design reviews
  • •Analyzed existing tools and identified unfulfilled needs
Design Choices
  • •Real-Time Preview: Enabled designers to hear changes immediately, reducing back-and-forth
  • •Integrated with existing tools (Google Slides, Chrome Extensions, and Framer)
  • •Handled complex voice processing behind the scenes

Role

UX Engineer

Tools

Visual Studio Code, Google Cloud, Google Slides, Git, Google Chrome

Tech

Web Speech API, Google Text-to-Speech API, HTML, CSS, JavaScript