Marketers and creatives collaborate to bring new products to market—and it helps to share the same vision from the start. So Adobe Researchers collaborated with the Adobe InDesign team, providing the core technology behind  Project Vision Cast. The experimental prototype allows users to combine data insights and explore brand imagery, fostering creative brainstorming backed by numbers. One of the key elements of Project Vision Cast is its innovative generative AI, which can seamlessly place a product image into any catalog spread. 

Attendees at this spring’s Adobe Summit were treated to an early look at Project Vision Cast in action.  

Using cutting-edge AI techniques for swapping a new object into a marketing spread with just the right angle, lighting, and shadows 

When the InDesign product team came to Adobe Research, they had an idea: they wanted to make it easy for designers to swap one product for another in a catalog image.  

“So, if someone was originally holding a bag, and the marketer wanted to change that to another product in their brand line, the InDesign team wanted a feature that would make that very simple to do,” explains Soo Ye Kim, Research Scientist with Adobe Research and one of the collaborators behind Project Vision Cast.  

Adobe Researchers already had promising technology in the works that could make this possible. The research began as Adobe Research Intern Yizhi Song’s project in the summer of 2022 and was published the following summer. It offered the industry’s first method for compositing objects into new backgrounds using the latest generative AI technology. The method could help with lots of things designers need to do, from inserting an object into an existing background (imagine adding a sofa into an image of a living room) to replacing an object (perhaps you want a different sofa in that room) and trying on new outfits virtually (perhaps the model on your new sofa wants different sweater).

But the Adobe Research team wanted to go far beyond just pasting a new object into a layout. They wanted the technology to automatically adapt the object to its new setting, taking into account the direction and type of lighting, the shadows, and even the orientation of the object. For example, if a user adds an image of a mug shown from above onto a new background, but the view in the layout should show the side of the mug, they wanted that view to update automatically, too.  

“If you just do a copy and paste, a new item is going to look weird and out of place,” explains Kim. “But if you use our technology, it automatically adjusts the geometry and adapts to the new background.”  

Researchers also wanted to preserve the fundamental qualities of the object, even as the feature modified it to fit with a new background. “Basically, if you want to insert your own object, you want it to continue to be that exact object. That can be a challenge because generative AI models tend to do a lot of generation. So we needed to make sure the technology could preserve the object’s identity,” says Kim. 

According to He Zhang, Adobe Research Senior Research Scientist and Engineer, “Adding these abilities to position the angle of the object without changing it was completely new, based on the newest generative AI technology.” 

Providing this new level of generation posed a big challenge: gathering enough data to train the model. “The ideal data needed to have the same objects in different views and under different lighting and different backgrounds, but we didn’t have that at all, and it wasn’t available in public datasets either. So we worked with a vendor to collect the exact pairs of data we needed for this model,” says Kim.  

The arduous process of collecting and refining the data and improving the model took the team two years. “Throughout the work, we always backed each other up, from the initial prototyping to the huge effort to collect the correct dataset, to the data filtering, and then train the model and update the inference pipeline. All along the way, our goal was to make sure the model would be robust for all different kinds of use cases,” says Zhang. 

The work benefited from several additional intern project innovations, including more published work on object identity preservation, along with intern Gemma Canet Tarrés’ research on unconstrained generative object compositing.  

From Research to Summit Sneak 

With years of research already in place, the Adobe Research team provided the technology as an API to the InDesign product team, who then led the system design by combining the API with additional segmentation techniques to extract key insights from the data. From there, the team developed a compelling demo and submitted it as a potential Summit Sneak.

“When Project Vision Cast was selected as a Summit Sneak, we were so excited,” says Kim. “This was the first time it was shared with the public, and that’s something we’re really proud of.” 

The team is already talking to several other Adobe product teams about additional use cases for the technology. “Like a lot of Adobe Research projects, we’re providing a core technology that product teams can leverage. They can come to us when they want something new, and we’re ready for them,” says Zhang. 

Key contributors  

Presenter: Sanyam Jain 
Adobe Research collaborators: Scott Cohen, Soo Ye Kim, Zhe Lin, Yizhi Song (intern), Gemma Canet Tarrés (intern), He Zhang, and Zhifei Zhang 
Additional collaborators: Pragya Kandari, Manit Singhal (tech intern), Gaurav Ahlawat (product intern), Wes Hopkins, and Eric Matisoff 

Wondering what else is happening inside Adobe Research? Check out our latest news here.