Generative Visual Manipulation on the Natural Image Manifold

Jun-Yan Zhu

UC Berkeley

Philipp Krähenbühl

University of Texas at Austin

Eli Shechtman

Adobe Research

Alexei A. Efros

UC Berkeley

 

Interactive GAN

Interactive GAN

Realistic image manipulation is challenging because it requires modifying the image appearance in a user-controlled way, while preserving the realism of the result. Unless the user has considerable artistic skill, it is easy to “fall off” the manifold of natural images while editing. In this paper, we propose to learn the natural image manifold directly from data using a generative adversarial neural network. We then define a class of image editing operations, and constrain their output to lie on that learned manifold at all times. The model automatically adjusts the output keeping all edits as realistic as possible. All our manipulations are expressed in terms of constrained optimization and are applied in near-real time. We evaluate our algorithm on the task of realistic photo manipulation of shape and color. The presented method can further be used for changing one image to look like the other, as well as generating novel imagery from scratch based on user’s scribbles.

Project Page

Demo Code

Project Publications

Generative Visual Manipulation on the Natural Image Manifold

Zhu, J., Krähenbühl, P., Shechtman, E., Efros, A. (Oct. 11, 2016)
European Conference on Computer Vision (ECCV'16)