Zeyu Jin

Senior Research Scientist

San Francisco

Zeyu is a senior research scientist at Adobe Research in San Francisco. His research area is in deep generative models for speech, on topics such as studio-quality speech enhancement, speech quality assessment and personalized voice generation. He is also interested in HCI for audio applications and music generation.

He received a Ph.D. degree in computer science from Princeton University adviced by Adam Finkelstein and M.S in music technology in Carnegie Mellon University. Between 2015 and 2017, he interned at Adobe for three times and presented his primary research project – VoCo – at Adobe MAX Sneaks (link to video) in 2016.

Publications

SpeakEasy: Enhancing Text-to-Speech Interactions for Expressive Content Creation

Brade, Stephen., Anderson, Sam., Kumar, Rithesh., Jin, Zeyu., Truong, Anh. (Apr. 26, 2025)

CHI 2025

Visual Description Grounding Reduces Hallucinations and Boosts Reasoning in LVLMs

Ghosh, Sreyan., Evuru, Chandra., Kumar, Sonal., Tyagi, Utkarsh., Nieto, Oriol., Jin, Zeyu., Manocha, Dinesh. (Apr. 21, 2025)

International Conference on Learning Representations (ICLR)

Efficient Spoken Language Recognition Via Multilabel Classification

Nieto, Oriol., Jin, Zeyu., Dernoncourt, Franck., Salamon, Justin. (Aug. 24, 2023)

Interspeech 2023

Audio Similarity is Unreliable as a Proxy for Audio Quality

Manocha, Pranay., Jin, Zeyu., Finkelstein, Adam. (Sep. 18, 2022)

Interspeech 2022

HEAR: Holistic Evaluation of Audio Representations

Turian, Joseph., Shier, Jordie., Khan, Humair., Raj, Bhiksha., Schuller, Björn., Steinmetz, Christian., Malloy, Colin., Tzanetakis, George., Velarde, Gissel., McNally, Kirk., Henry, Max., Pinto, Nicolas., Noufi, Camille., Clough, Christian., Herremans, Dorien., Fonseca, Eduardo., Engel, Jesse., Salamon, Justin., Esling, Philippe., Manocha, Pranay., Watanabe, Shinji., Jin, Zeyu., Bisk, Yonatan. (Jul. 20, 2022)

NeurIPS 2021

SQAPP: No-Reference Speech Quality Assessment Via Pairwise Preference

Manocha, Pranay., Jin, Zeyu., Finkelstein, Adam. (May. 22, 2022)

ICASSP 2022

Controllable Speech Representation Learning Via Voice Conversion and AIC Loss

Wang, Yunyun., Su, Jiaqi., Finkelstein, Adam., Jin, Zeyu. (May. 22, 2022)

ICASSP 2022

Music Enhancement via Image Translation and Vocoding

Kandpal, Nikhil., Nieto, Oriol., Jin, Zeyu. (May. 8, 2022)

International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

Controllable deep melody generation via hierarchical music representation

Dai, Shuqi., Jin, Zeyu., Gomes, Celso., Dannenberg, Roger. (Nov. 8, 2021)

International Society for Music Information Retrieval Conference

HiFi-GAN-2: Studio-quality speech enhancement via generative adversarial networks conditioned on acoustic features

Su, Jiaqi., Jin, Zeyu., Finkelstein, Adam. (Oct. 17, 2021)

IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA)

Neural Pitch-Shifting and Time-Stretching with Controllable LPCNet

Morrison, Max., Jin, Zeyu., Bryan, Nicholas., Caceres, Juan-Pablo., Pardo, Bryan. (Oct. 5, 2021)

Arxiv

CDPAM: Contrastive learning for perceptual audio similarity

Manocha, Pranay., Jin, Zeyu., Zhang, Richard., Finkelstein, Adam. (Jun. 9, 2021)

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

Bandwidth Extension is All You Need

Su, Jiaqi., Wang, Yunyun., Finkelstein, Adam., Jin, Zeyu. (Jun. 9, 2021)

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

Context-Aware Prosody Correction for Text-Based Speech Editing

Morrison, Max., Rencker, Lucas., Jin, Zeyu., Bryan, Nicholas., Caceres, Juan-Pablo., Pardo, Bryan. (Jun. 6, 2021)

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

HiFi-GAN: High-fidelity denoising and dereverberation based on speech deep features in adversarial networks

Su, Jiaqi., Jin, Zeyu., Finkelstein, Adam. (Oct. 26, 2020)

Interspeech 2020

Controllable Neural Prosody Synthesis

Morrison, Maxwell., Jin, Zeyu., Salamon, Justin., Bryan, Nicholas., Mysore, Gautham. (Oct. 26, 2020)

Interspeech 2020

A Differentiable Perceptual Audio Metric Learned from Just Noticeable Differences

Manocha, Pranay., Finkelstein, Adam., Zhang, Richard., Bryan, Nicholas., Mysore, Gautham., Jin, Zeyu. (Oct. 26, 2020)

Interspeech 2020

Metric Learning vs Classification for Disentangled Music Representation Learning

Lee, Jongpil., Bryan, Nicholas., Salamon, Justin., Jin, Zeyu., Nam, Juhan. (Oct. 11, 2020)

International Society for Music Information Retrieval Conference (ISMIR)

Disentangled Multidimensional Metric Learning For Music Similarity

Lee, Jongpil., Bryan, Nicholas., Salamon, Justin., Jin, Zeyu., Nam, Juhan. (May. 4, 2020)

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

Text-based Editing of Talking-head Video

Fried, Ohad., Tewari, Ayush., Zollhofer, Michael., Finkelstein, Adam., Shechtman, Eli., Goldman, Dan., Genova, Kyle., Jin, Zeyu., Theobalt, Christian., Agarwala, Maneesh. (Aug. 1, 2019)

ACM Transactions on Graphics (Proc. SIGGRAPH'19)

FFTNet: a Real-Time Speaker-Dependent Neural Vocoder

Jin, Zeyu., Finkelstein, Adam., Mysore, Gautham., Lu, Jingwan. (Apr. 15, 2018)

IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)

VoCo: text-based insertion and replacement in audio narration

Jin, Zeyu., Mysore, Gautham., DiVerdi, Stephen., Lu, Jingwan., Finkelstein, Adam. (Jul. 31, 2017)

ACM Transactions on Graphics (SIGGRAPH)

CUTE: a Concatenative Method for Voice Conversion Using Exemplar-based Unit Selection

Jin, Zeyu., Finkelstein, Adam., DiVerdi, Stephen., Lu, Jingwan., Mysore, Gautham. (Mar. 1, 2016)

The 41st IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

News