Scene Designer: a Unified Model for Scene Search and Synthesis from Sketch

ICCV Workshop on Sketching and Human Expressivity (ICCVW)

Published October 17, 2021

Leo Sampaio Ferraz Ribeiro, Tu Bui, John Collomosse, Moacir Ponti

Scene Designer is a novel method for searching and generating images using free-hand sketches of scene compositions; i.e. drawings that describe both the appearance and relative positions of objects. Our core contribution is a single unified model to learn both a cross-modal search embedding for matching sketched compositions to images, and an object embedding for layout synthesis. We show that a graph neural network (GNN) followed by Transformer under our novel contrastive learning setting is required to allow learning correlations between object type, appearance and arrangement, driving a mask generation module that synthesizes coherent scene layouts, whilst also delivering state of the art sketch based visual search of scenes.

Learn More