VoCo: Text-based Insertion and Replacement in Audio Narration

Zeyu Jin

Princeton University

Gautham Mysore

Adobe Research

Stephen DiVerdi

Adobe Research

Jingwan (Cynthia) Lu

Adobe Research

Adam Finkelstein

Princeton University

Editing audio narration using conventional software typically involves many painstaking low-level manipulations. Some state of the art systems allow the editor to work in a text transcript of the narration, and perform select, cut, copy and paste operations directly in the transcript; these operations are then automatically applied to the waveform in a straightforward manner. However, an obvious gap in the text-based interface is the ability to type new words not appearing in the transcript, for example inserting a new word for emphasis or replacing a misspoken word. While high-quality voice synthesizers exist today, the challenge is to synthesize the new word in a voice that matches the rest of the narration. This paper presents a system that can synthesize a new word or short phrase such that it blends seamlessly in the context of the existing narration. Our approach is to use a text to speech synthesizer to say the word in a generic voice, and then use voice conversion to convert it into a voice that matches the narration. Offering a range of degrees of control to the editor, our interface supports fully automatic synthesis, selection among a candidate set of alternative pronunciations, fine control over edit placements and pitch profiles, and even guidance by the editors own voice. The paper presents studies showing that the output of our method is preferred over baseline methods and often indistinguishable from the original voice.

VoCo in Adobe Max 2016:

VoCo in the news:

Digitaljournal “New Software edits voices like text”

Sciencealert “Adobe’s New ‘Photoshop For Voice’ App Lets You Put Words in People’s Mouths”

Peek Behind the Sneaks: Controversy and Opportunity in Innovation

Project Publications

CUTE: a Concatenative Method for Voice Conversion Using Exemplar-based Unit Selection

Jin, Z., Finkelstein, A., DiVerdi, S., Lu, J., Mysore, G. (Mar. 1, 2016)
The 41st IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

VoCo: Text-based Insertion and Replacement in Audio Narration

Jin, Z., Mysore, G., DiVerdi, S., Lu, J., Finkelstein, A. (Aug. 1, 2017)
ACM Transactions on Graphics (Proc. of SIGGRAPH 2017)