By Meredith Alexander Kunz, Adobe Research
Project Cloak, an experimental tool by Adobe Research, offers an innovative way to remove unwanted objects from a video. Object removal is a huge pain point for visual effects artists, who currently spend countless hours manually erasing boom-microphones, special effect wires, distracting objects, and more.
A user begins by identifying what she would like taken out of the video. Cloak then estimates the motion of the corresponding scene, looking both forwards and backwards in time to replace missing pixels with the background as seen in other frames. This approach is known as “dense tracking.”
For videos where the region behind the object is never seen, the system employs Adobe’s content-aware-fill to guess what is missing. Ultimately, the algorithm figures out how to replace the unwanted pixels with new ones that best match the appearance of the scene.
In the Adobe MAX Sneak presented by Adobe Research’s Geoffrey Oxholm, a tall lamppost was effectively removed, then a backpack strap, and then whole people as they walked across the screen, eliciting an awed response from the audience.
“Cloak marries dense tracking with pixel replacement,” explains Oxholm, a research engineer at Adobe Research. This work originated with the groundbreaking 2004 PhD thesis of Eli Shechtman, now a principal scientist with the group.
For this new application to make it into the hands of artists, there’s more to be done. “It’s late breaking research—we are gathering more data now,” Oxholm says. The team hopes that someday, this approach could help “democratize” elements of video editing that are currently extremely time- and money-consuming, opening up new avenues for creativity outside of high-end studios.
Contributors:
Geoffrey Oxholm, research engineer; Oliver Wang, senior research scientist; Eli Shechtman, principal scientist; Mike Lukáč, research scientist (Adobe Research)
Xin Wang (UCSB)
