By Meredith Alexander Kunz, Adobe Research
The art of painting pre-dates recorded history. Early humans’ cave paintings reveal that they were already eager to express themselves this way. And until recently, the tools of watercolor and oil painting were ancient, too: brush bristles and colored pigments.
Now that’s changing. A partnership of Adobe Research scientists and artists set out to create a revolutionary digital painting technology, one that would form the backbone of a new painting tool from Adobe. Their goal? To satisfy a most discerning audience: traditional analog painters who want their digital “paint” to seamlessly simulate the real thing.
The powerful results of this union of technology and creativity can now be found at the heart of Adobe Fresco, the mobile painting app built and released by the Adobe Fresco team. Its fall 2019 debut was met by the acclaim of artists, who are now able to use realistic watercolor and oil paints on a digital platform. (The app is currently available for free for iOS and Windows mobile devices.)
Fresco was born of an interweaving of science and art, a collaboration that made this new technology shine even for traditionalists.
“This technology was built with deep interplay between artists and scientists,” says Nathan Carr, senior principal scientist at Adobe Research. “It wasn’t just about pure mathematics. There were aesthetic components going into it from the start.”
The rocky road to natural media simulation
At the center of this story is a San Jose-based senior research scientist who was inspired by the long history of painting: Byungmoon Kim.
“I’m very far from artistic, but I thought that digital painting would be a meaningful problem for me to take on,” says Kim. Painting distinguished homo sapiens from other hominids in our early days, Kim points out.
“People have been doing it for a long time, and we will continue doing it,” Kim explains. “And now that we have the chance to do digital painting, I thought that if we could invent something new, it could have an impact on the future.”
Digital painting is not completely new, but making it feel “real” for artists has had a rocky road. Adobe Research scientists had been exploring “natural media simulation” in digital painting for over a decade before the most recent efforts began. The team was interested in creating software for a mobile tablet device. But they found that on the early version of the iPad, for example, re-creating the physical properties of working with paint was challenging.
“We had a lot of interns who tried to make it work. They had strong math skills. But we couldn’t get it to run fast enough for deployment,” Carr says.
Then an intern arrived who was an art student. He posed the problem differently: Why not forget the real-world physics of paint, and focus on making a watercolor tool that “feels right” to an artist? That student’s name was Daichi Ito, and he leveraged those ideas to create a polygon-based watercolor painting algorithm. Adobe later used that technology to create Adobe Eazel, an early painting mobile app.
Today, Ito is a technical research artist with Adobe Research and a winner of the 2019 Adobe Founder’s Award. He has partnered with Kim in his quest to create an ideal watercolor and oil painting engine for tablets.
“Squeezing” digital paint onto a mobile device
Kim asked himself: Could his team design a digital painting technology that would feel realistic, genuine, and satisfying not just for one-time users, but robust enough even for analog painters? Especially for artists who loved to use media such as watercolor and oil paints, both of which have unique properties?
After years of development, one successful result came with WetBrush, a real-time oil painting simulation designed to run on a desktop machine because of its intense computing needs. This experimental technology was featured at the Adobe MAX creativity conference.
WetBrush worked so well that the Tech Interactive, a technology and science museum in San Jose, created a technology installation out of it, allowing users to try it themselves. One major limitation, though, was that the system required high-end hardware and graphics processors, and it could not be used on a tablet.
Soon, Kim began to tune a new version that could run on an iPad. “Squeezing” the tool onto a mobile device required a different approach.
Kim developed a new custom collision algorithm, one that was faster than past versions. He also created a tile-based dynamic simulation that could support multiple kinds of devices. “It’s about the communication among tiles, to allow them to flow,” he says.
Kim also began experimenting with futuristic styles of brushes that pushed the envelope of the medium of painting. He created a painting simulation that caused paint to appear to boil and percolate, which he termed a “boiling brush.”
“Competing with nature can be hard, but with digital painting, we can also invent something different,” Kim says.
Artists give input: “Painting with toothpaste”
As Kim labored over the Iris engine, which simulated watercolor and oils, Ito and other artists played a crucial role in helping to craft a better painting experience. Testers included Kyle Webster, an Adobe senior design evangelist who shared his Fresco art publicly, other artists outside Adobe, and interns.
Ito didn’t hold back when giving his honest impressions. At one point, when Carr asked him to try out a version of the system, Ito said it was too slow. Carr asked him, “Why don’t you just paint slower?” Ito explained that that would ruin the quality of his lines, which require a quick movement. The team kept working.
Later, when Ito first tried out the oil painting tool, it wasn’t quite what he expected. “I told them, ‘I feel as if I’m painting with toothpaste.’”
Ito tried to help researchers understand what an artist would anticipate in an oil painting tool. “We talked about how soft or how hard the pigment should be. We tweaked and adjusted it. It’s hard to explain, because there are no words for what an artist experiences.”
Kim was eager to gather feedback. “He listened to me so carefully, and to other artists,” Ito says. The team worked day and night in partnership with the Fresco team to prep the simulation engine for the product’s release date. For Fresco, Paul George, the architect of Fresco’s foundational painting engine, SketchLib, along with Ankit Jain, Shubham Gupta, and others, married Iris with SketchLib, creating an exceptionally artistic product.
Opening up new avenues for art
Ultimately, when Fresco was released in late 2019, both Kim and Ito were happy to see the positive response. The application was quickly recognized by Popular Science magazine with a spot on its list of the top 100 innovations of 2019.
“Fresco satisfied many artists. Japanese users were so excited,” says Ito, who has presented at Adobe MAX Japan. (The Fresco team eagerly delivers new features to users in monthly updates, matching the enthusiasm of artists who use the product.)
Fresco also opened up new avenues for art creation. “You can bring the iPad for painting anywhere, anytime. I can do painting in new places, like cafes. I don’t have to worry about taking a lot of time to set up, prep, and clean,” Ito explains.
He also notes that you can now combine media in ways that would never be possible in analog painting, where watercolor and oils don’t mix.
The best of all? Fresco is a tool so realistic that you can’t tell that artists used it to create their masterpieces, says Ito. It’s digital and new, but it produces old-school art seamlessly. “People use it, and they just make great art.”
