By Meredith Alexander Kunz, Adobe Research
If you’ve ever wondered what you might look like as a painting by a famous artist, Puppetron has the answer. This experimental technology by Adobe Research and collaborators turns your image into a work of art. And when the tool unites with Adobe’s Character Animator, your face can become a moving, talking, laughing character.
In a demo by research engineer Jakub Fišer at the Adobe MAX event in October 2017, Puppetron impressed the audience with its interactive ability to create a visually-rich portrait based on a specific artistic style.
The technology preserves the recognizable aspects of a person’s face as it quickly synthesizes a stylized artistic version. As the demo shows, the technology can apply styles as varied as watercolor paint, colored pencil, ink, graffiti markers, and wood and bronze sculptures, with remarkable results. The tool can also tune the stylization effect, allowing the user to add just a touch of artistry to a portrait, or to merge the face more fully with the artistic style.
Adding icing to this visual cake, combining Puppetron with Adobe’s Character Animator—co-created by Adobe Research—allows these artistic portraits to be animated interactively. Fišer showed himself as a moving Renaissance painting in the demo, and much more is possible.
How does it work? Fišer describes the technology as a “patch-based, guided texture synthesis algorithm.” The tool matches patches of an original image—a face—to an artistic style input image—a painting, drawing, sculpture, or other artwork.
The system generates “behind-the-scenes” guiding maps that direct a transfer of corresponding facial parts—from the style’s eyes to the user’s eyes, from the style’s hair to the user’s hair, and so on. It works fast because many of these maps can be precomputed. When used with Character Animator, a set of stylized still frames can come to life based on live tracking of a moving person’s face.
In partnership with colleagues from the Czech Technical University in Prague, the research was published in a conference paper at SIGGRAPH 2017 and has since garnered widespread media attention. It continues to hold promise as an ongoing research pathway, Fišer reports, into the brave new world of artistically stylized portraits and videos.
Jakub Fišer made himself into a line drawing, a painting, and a sculpture using the experimental tool Puppetron.
Want to try out Puppetron and Character Animator for yourself? Check out this story on the Adobe Blog to learn how to apply to test the beta version.
Contributors:
Jakub Fišer, Eli Shechtman, Jingwan (Cynthia) Lu, Paul Asente, Mike Lukáč (Adobe Research)
David Simons (Adobe Fellow, Advanced Product Development)
Ondřej Jamriška, Daniel Sýkora (Czech Technical University in Prague, Department of Computer Graphics and Interaction, Faculty of Electrical Engineering)
