Data has enormous value. But understanding charts and graphs filled with data—and finding the story behind that raw information—is far from simple. It’s even more difficult to get insights from data in a format that’s accessible for all.
Now, a new AI-based technology created by Adobe Research is offering users a way to do all of this. It’s called Intelligent Captions, and it has been released in Adobe Experience Platform and Adobe Analytics Technology Preview.
If you’ve ever wanted your data to speak to you, now it can—in accessible captions that are automatically generated to show key trends and important events. The captions sit on top of your charts, interpreting the data shifts in clear language.
“Our technology uncovers valuable hidden data patterns and makes complex data and charts much more interpretable and accessible,” explains Fan Du, Senior Research Scientist at Adobe Research and one of the key minds behind Intelligent Captions.
“Automatically detecting and explaining insights is very useful to customers who may not have the knowledge or time to manually slice-and-dice the data,” adds Eunyee Koh, Principal Research Scientist.
From intern project, to sneaks, to product
Like many other Adobe Research efforts, this work started with an intern project. It was initiated by University of Maryland PhD candidate Xin Qian in 2019.
Before it was a fully-fledged technology, two Adobe Research-led SUMMIT sneaks—tech previews at the company’s annual conference for digital marketers—gave a sense of the value of the research breakthroughs that would become Intelligent Captions.
One such sneak was Data Unbound, presented at Adobe Summit 2019. More recently, at Adobe Summit 2022, another sneak called KPI Pop gave a window into the caption technology.
Under the hood
How does it work? Under the hood, the new system leverages cutting-edge machine learning models that detect important information in the data. The models look for signals or patterns that could help marketers and other data analysts understand the meaning behind the information displayed in the chart.
It took more than just a simple statistical model to make this happen, Du reports. “Multiple Adobe researchers teamed up and collectively invented a ‘treasure chest’ of insight detection models,” he says. These models earned five US patents along the way.
The system goes beyond just surfacing insights—it translates them into easy-to-read caption text. To do that, the team behind this technology worked on a second layer for their system: A natural language model that could convey these insights clearly and effectively to human readers. They wanted an approach that would read well and naturally, so the researchers included humans in the mix.
“We were able to curate high-quality captions written by human professionals, train natural language generation models, and again have human professionals verify quality,” Du says. The researchers published two peer-reviewed research papers describing this approach at CHI 2020 and WWW 2021.
An important aspect of Intelligent Captions is that it provides an accessible way for those who use screen readers to get insights from data. A user study conducted by Adobe’s accessibility team confirmed the benefits of this feature for this group. “Rather than relying on visuals alone, users have the option to read text or hear it read aloud,” says Koh.
Building collaboratively – to help customers
To make their experimental idea a reality in two products, Adobe Researchers partnered closely with product teams to test and iterate, ultimately integrating their work into software that real-world users can apply to their data. Design, engineering, machine learning, user experience, and accessibility experts also collaborated with the team.
Interesting challenges arose in the process. For example, the team needed to ensure that non-English speakers could effectively use this technology. Listening to customers helped shape the tool, and also helped the team understand its power for users.
Customers have already let the team behind this tech know how helpful it is. One customer reported that this technology helped their company address accessibility needs, enabling more people to work effectively with data visualizations and meeting compliance requirements. Another customer said that Intelligent Captions made storytelling with data much easier, saving a lot of time—always welcome for busy marketers.
Contributors from Adobe Research: Fan Du, Eunyee Koh, Shiv Kumar Saini, Sunav Choudhary, Ryan Rossi, Gromit Chan, Shunan Guo, Jane Hoffswell, Sana Malik, Sungchul Kim, Xin Qian (intern)
Other contributors: Product Management, Design, Engineering, and Accessibility and Usability Teams
