By Meredith Alexander Kunz, Adobe Research
The day began inauspiciously with the wrong train ticket. It ended with the idea for a research project that would later go viral across continents.
Richard Zhang, a research scientist at Adobe Research’s San Francisco office, had just left a major conference in Vancouver. As the first Adobe Research fellowship winner to become a full-time hire at the lab just a few months prior, Zhang was excited to pursue new projects in computer vision. He was heading towards Seattle to touch base with colleagues.
Due to a mix-up between two remarkably similar destinations—Vancouver, Canada, and Vancouver, Washington—and an incorrect train ticket, Zhang came very close to missing his connection to Seattle. Eventually, he made it to the office and met with a handful of researchers, including Senior Research Scientist Oliver Wang.
“Oliver and I were talking about how AI is quickly democratizing content creation and making better creative tools for users. On the other hand, we wanted to see if AI could similarly help the other side—image forensics and content authenticity,” that is, the ability to tell original from doctored images, says Zhang.
“We were looking at the Face-Aware Liquify tool in Photoshop for another project and thought, could we possibly identify its use in the wild? Maybe there’s some signal it leaves that we could detect.”
The work’s outcome was uncertain from the beginning. Their goal: To be able to discern which photos were manipulated to make people appear to smile or frown or look distorted in other ways. More ambitiously, they hoped to find a way of reversing the manipulation, creating an “undo” button.
“Scientifically, we weren’t sure this was even possible. After all, a slightly warped face still looks quite natural to the naked eye,” says Zhang.
Advancing image forensics
They did manage to do it, with a close partnership with UC Berkeley researchers Sheng-Yu Wang, an undergraduate student, Andrew Owens, a postdoctoral researcher, and Professor Alexei A. Efros, who had been Zhang’s PhD advisor and is now an Adobe consultant. They addressed their initial lack of data first by automatically generating a large set of warped faces, and then by hiring an artist, who created manipulated images by hand.
The researchers then trained deep neural networks to identify with great accuracy which images were “fake.” Zhang says that the networks seemed to be able to identify warping artifacts, such as traces left from stretching and compressing pixels or perturbed block patterns from JPG compression, but the mechanisms of a deep network aren’t completely transparent.
“Deep networks are really powerful because they can learn what cues they need, under the hood,” says Zhang. “Unfortunately, you don’t know exactly how they solved the problem.”
In the spring of 2019, the group released a well-received paper on detecting photographs manipulated using the Photoshop Face Liquify feature, getting the attention of more than 100 million people all over the world after being picked up by the press, social media, and TV news (and our blog).
It’s a step forward in the emerging field of image forensics and was recently featured as an Adobe MAX Sneak, Project About Face. Zhang took to the main stage to present the work to 15,000 people in Los Angeles (with over 60,000 more watching the video on YouTube later).
Circuitous path—and fellowship win
The chance discussion between Zhang and Wang was a good example of the circuitous path that brought Zhang to where he is today at Adobe Research. It’s far from his early work.
His undergraduate degree focused on integrated circuits and signal processing, and he worked a brief stint in the defense industry on radar systems. But Zhang quickly realized that the defense field wasn’t the best fit. He became captivated by the promise of machine learning in 2012.
“I was fortunate to apply to grad school right before things really exploded in the field,” he recalls. Zhang loved computer vision and graphics work simply because it has “fun data to play with.”
Winning an Adobe Research Fellowship and internship as a graduate student at UC Berkeley helped to supercharge Zhang’s work. During his internship, he collaborated on a conference paper that showed how deep learning could help computers perceive images more like humans do. He also won a Thesis Fast Forward Award for Best Presentation at SIGGRAPH 2018.
“Getting the fellowship was very exciting for me—it felt like validation. And then being the first fellowship winner to join full-time was really fun,” says Zhang. Since then, other fellowship winners—Zoya Bylinskii, Dingzeyu Li, Yijun Li, and Deepali Aneja—have joined the team.
From interning at Adobe to mentoring his own interns
His first summer as an intern and fellow, in 2017, introduced Zhang to Adobe Research. He interned in the Seattle office with Oliver Wang and Principal Scientist Eli Shechtman. Now, he’s returning that favor by hiring his own interns. This past summer he mentored four of them.
For each intern he works with, Zhang learns a new working style and tries to adapt to the individual’s needs. He remembers well what it was like interning. “Being just one year removed helps,” he says.
It’s also one of the greatest rewards of his role. “Working with interns is the best part of the research scientist position. Having students cycle in during the summer reinvigorates the environment and brings in diverse perspectives,” Zhang reports. This year, one of his interns presented a paper at ICCV (the International Conference on Computer Vision), and three others will likely submit papers to conferences. Up to three will submit patents, too.
“Adobe Research has an environment similar to a university setting over the summer. Working with interns allows me to explore ideas much faster than I can do by myself.”
And those ideas are free flowing, creating new pathways to discovery. As Zhang puts it: “Successful research requires openness to serendipity.”
