After all the work a designer pours into creating the perfect logo and decals for a product, the next step is to visualize everything in the real world. But that’s easier said than done. Mocking up a label on a 3D surface, like a mug, t-shirt, billboard, or bottle, has always been an arduous process that requires 3D content-creation experience and lots of time.
That’s why Adobe Researchers, in collaboration with the Illustrator team, invented the new Mockup feature in Illustrator. Mockup uses AI to automatically understand the 3D shape of a 2D image, and with just a couple of clicks it can wrap a label or logo around the shape’s curves and edges. Designers can easily move their decals anywhere on and around the object to achieve just the right placement.
“Mockup is such an exciting example of the power of modern machine learning,” says Nathan Carr, one of the Research Scientists who helped create the new feature. “The ability of our trained network to accurately predict the shape of the world from a single photograph can feel like magic.”
The public first glimpsed Mockup during a Sneak at Adobe MAX in 2022—back when the technology was called Project Vector Edge. It’s now available as a beta feature in Adobe Illustrator.
Mocking up product design—before and after Mockup
Before the Mockup feature, designers had two primary—and cumbersome—options for visualizing 2D designs in a 3D world. They could find or create a 3D model of their target object (a t-shirt, bottle, tube, etc.), export their design into raster format, and then map the image onto their 3D model with a separate tool. Another option was to use 2D warping tools in Illustrator to painstakingly mimic the look of a design on a curved surface—a slow process that has to be re-done with every small change to the object, decal, or camera position.
Mockup, on the other hand, automatically infers the 3D qualities of an input image (that t-shirt, bottle, tube, etc.), and then preserves the quality of a decal as users place it on the 3D surface. Repositioning is simple, and users don’t need to learn any additional tools or techniques—they can drag, rotate, and scale, all within Illustrator.
“Visualizing design on a 3D surface has always been hard work, but it’s also extremely important for designers—it’s a necessary step for validating designs and showcasing them to clients,” says Siddhartha Chaudhuri, Senior Research Scientist at Adobe Research and Mockup contributor. “So, the goal with Mockup was to take the pain out of this important part of the design process.” Beta users for Mockup have already shared glowing feedback with the product team and on social media. “The positive response indicates just how important the Mockup solution is for design workflows,” adds Chaudhuri.
Developing the technology behind Mockup
Before they helped create Mockup, members of Adobe Research had already collaborated with the Illustrator tech transfer team to develop one of the key elements of the tool, the vector decals feature that ensures that a 3D preview works well, regardless of its resolution.
While that work was happening, the Adobe Research team was also developing and training a neural network that could predict depth values from images, making it possible to automatically create 3D maps of 2D color images. Adobe Researchers worked with the Illustrator product team to fine-tune this technology for Mockup’s scenarios.
With the ability to get a 3D map from any color image, the challenge remained how to place these decals onto the 3D mapped images. For this, researchers turned to state-of-the art algorithms in meshing and geometry processing to complete the final piece of the puzzle.
“Working with the Illustrator product team is always fantastic,” says Chaudhuri. “They have such a strong sense of user needs and they’re extremely hands-on, so we’re able to go rapidly from ideas to execution,” he adds.
In the development process, the team hit a few challenges along the road. “As with any AI inference algorithm there are failure cases,” says Carr. “This happened when we were working on decaling objects like coffee mugs. Humans recognize that mugs are perfectly round, but our algorithm predicted squished oval shapes, and this resulted in less appealing, flattened decals To overcome these issues we needed to build a larger training set specifically targeted at the classes of objects we anticipate our customers wanting to decal.” With the first batch of hurdles behind them, Carr’s excited about what comes next. “Making this algorithm robust enough across a wide range of content was a marvel. The goal to perfectly decal any photograph still leaves many open challenges including the ability to match lighting and operate on highly complex shapes.”
Mockup was developed with contributions from many members of the Adobe Research team, including Jianming Zhang, Kevin Blackburn-Matzen, Matheus Gadelha, Nathan Carr, Oliver Wang, Pierre Gueth, Qingnan Zhou, Siddhartha Chaudhuri, Simon Niklaus, and Yannick Hold-Geoffroy.
Wondering what else is happening inside Adobe Research? Check out our latest news here.
