Audio

We invent audio AI technology for human-centered creativity. Our goal is to empower people to bring their creative ideas to life through high quality audio content. This ranges from music and podcasts to video and immersive experiences, as well as emerging media types. We are dramatically simplifying the creation process with AI so that people can quickly go from idea to produced content regardless of skill level, iterating on the creative aspects rather than the technical ones. We do this through our research in the analysis, processing, and generation of speech, music, everyday sounds, and more. We also work on problems at the intersection of audio with video, augmented reality, and natural language processing. To advance all of these research areas, we develop new machine learning models, novel signal processing algorithms, and new human computer interaction paradigms. 

Meet our Audio Researchers

All Researchers
Dongjun Kim Zhepei Wang Johannes Imort Ke Chen Gautham Mysore Ge Zhu Dingzeyu Li Prem Seetharaman Nicholas J. Bryan Justin Salamon Zeyu Jin Hugo Flores García

Publications

All Publications
July 20, 2022 NeurIPS 2021

HEAR: Holistic Evaluation of Audio Representations

Joseph Turian, Jordie Shier, Humair Raj Khan, Bhiksha Raj, Björn W Schuller, Christian J Steinmetz, Colin Malloy, George Tzanetakis, Gissel Velarde, Kirk McNally, Max Henry, Nicolas Pinto, Camille Noufi, Christian Clough, Dorien Herremans, Eduardo Fonseca, Jesse Engel, Justin Salamon, Philippe Esling, Pranay Manocha, Shinji Watanabe, Zeyu Jin, Yonatan Bisk

AI & Machine Learning Audio
August 1, 2019 ACM Transactions on Graphics (Proc. SIGGRAPH'19)

Text-based Editing of Talking-head Video

Ohad Fried, Ayush Tewari, Michael Zollhofer, Adam Finkelstein, Eli Shechtman, Dan B Goldman, Kyle Genova, Zeyu Jin, Christian Theobalt, Maneesh Agarwala

AI & Machine Learning Audio Computer Vision, Imaging & Video
January 1, 2017 Communications of the ACM

Eulerian Video Magnification and Analysis

Neal Wadhwa, Hao-Yu Wu, Abe Davis, Michael Rubinstein, Eugene Shih, Gautham Mysore, Justin G. Chen, Oral Buyukozturk, John V. Guttag, William T. Freeman, Frédo Durand

Audio Computer Vision, Imaging & Video
December 2, 2016 ACM Transactions on Graphics (SIGGRAPH Asia)

Crumpling Sound Synthesis

Gabriel Cirio, Dingzeyu Li, Eitan Grinspun, Miguel A. Otaduy, Changxi Zheng

Audio Graphics (2D & 3D)
March 25, 2012 37th IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

UMD-JHU 2011 Speaker Recognition System

D. Garcia-Romero, X. Zhou, D. Zotkin, Balaji Vasan Srinivasan, Y. Luo, S. Ganapathy, S. Thomas, S. Nemala, G. Sivaram, M. Mirbagheri, S. Mallidi, T. Janu, P. Rajan, N. Mesgarani, M. Elhilali, H. Hermansy, S. Shamma, R. Duraiswami

AI & Machine Learning Audio
October 30, 2007 In Proceedings of IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA) . New Paltz, NY. October 2007

Example-driven Bandwidth Expansion

Paris Smaragdis, B. Raj.

Audio