For people to truly co-create with AI, we’ll need tools that go beyond simple AI prompts and outputs. So, Adobe Researchers, in a paper presented at this year’s prestigious ACM CHI conference on human-computer interaction, outlined a new approach for designing environments where humans and AI can collaborate and iterate together across an entire creative process. The approach infuses AI into the key components of a creative project—the various steps, apps, and procedures that are often siloed—and then knits those components together with AI. The result is a new human-AI collaboration paradigm. To evaluate their approach, the team built a prototype that reimagines the video creation workflow.
The work behind the new paper, Compositional Structures as Substrates for Human-AI Co-creation Environment: A Design Approach and A Case Study, was a collaboration between Adobe Researchers and researchers at the University of California San Diego. The team began their work in 2023, just as applications of LLMs (large language models) were beginning to take off.
“At that time, we were thinking about how AI could help bring together the different elements of the video editing process,” explains Valentina Shin, Adobe Research Scientist and one of the authors of the paper. “There are many steps to creating a video, from writing a script to organizing the footage and finding the narrative. We wanted to see if we could use AI to make it easier for people to go back and forth between processes—because that’s the organic way people work. Along the way, we wanted to re-think human-AI creative collaboration.”
Envisioning a new way to build AI into the creative process
The team began their research with a comprehensive literature review to determine the critical elements, called compositional structures, that people use to organize the pieces of a creative project: spatial, temporal, narrative, and congruent (the integration of multimodal elements). From there, they developed a blueprint for designing tools that include AI in each compositional structure, and then link those structures together with AI. The goal was to allow users to co-create with AI at each step, move easily between the steps, and collaborate with AI in a controlled and fluid way.
“At the core of the paper is the idea that embedding AI within and between different structures and creative workflows will make it a lot easier for people to iterate,” explains Anh Truong, Adobe Research Engineer and one of the paper’s contributors.
To evaluate their approach, the team built a new tool for video creation.
An environment for planning, exploring, and iterating videos in partnership with AI
When the team first began planning their research, they envisioned a tool that could help with video storyboarding. But former Adobe Research intern Yining ‘Rima’ Cao, who contributed much of the work behind the paper and presented it at CHI, urged the team to dream bigger.
“Rima kept pushing our thinking. Instead of just video planning, she asked how we could apply our findings to the whole process of video creation, while also building a framework that could help with all types of creative tasks,” remembers Truong.
“As a creator myself,” says Cao, “I know that creation is not just about producing an outcome—it is a process through which our thought takes form and evolves. I believe that an effective human-AI co-creation environment should honor this iterative nature, supporting the way we think and reflect, rather than merely accelerating the path to a final output. So, in this research, we wanted to investigate the core components and design approaches necessary to build such environments.”
Once the team developed its framework for AI-infused creative environments, they conducted interviews with video creators to understand their processes. From there, they designed a human-AI co-creation environment which includes a freeform canvas for spatial organization of assets, a narrative editor, a grid-based scene planner, and a timeline editor. Each element offers AI generation, which users can easily inspect and manage, and it’s simple to move between the elements to explore new ideas.
The team conducted initial testing of their prototype with ten video creators and found that it helped them stay in their creative flow, try new ideas, and take control of AI generation. “All of the testers appreciated having full context and the ability to jump into any part of the work to make edits and take control. This flexibility aligned naturally with their creative process. Users with less experience with video creation appreciated the structures as a form of guidance while experts especially appreciated the controllability and transparency,” says Cao.
According to the Research team, video was an especially good test case for their new paradigm because it relies on all four of the key compositional structures—and it’s a popular medium that’s difficult for people to master. “There’s a huge demand for video and the video landscape is consistently changing,” says Truong. “Creators feel a lot of pressure to keep up with that, but the process of ideation to creation for video can take a really long time. This is a real opportunity to help alleviate some of that burden for creators.”
“We’re also lowering that barrier for people who don’t have video expertise, and that’s exciting to me,” adds Shin. “And we’re helping build new capabilities for AI—more ways to help people turn their thoughts into something real that they can see on the screen. New tools can help you write the script, generate the footage, and organize the footage. It’s no longer just using AI to do the menial tasks—AI can help you direct as well.”
The AI-infused future of creative tools
The research behind the new framework, and the work that went into the video tool, are part of a bigger mission at Adobe Research, says Truong. “One thing that our group focuses on is using AI to help people tell stories that are meaningful to them. So I’m especially excited about the new ways that AI can help people figure out how to form a good story, and even give them insights and feedback about how other people might view their stories.”
The CHI paper also reflects a shift in the way researchers are thinking about the creative environments they build. “We used to define tools first, and then people had to stay with that tool to do their work or else go to another tool and just use that one,” explains Shin. “But I see more and more of a trend where the tool doesn’t define a person’s workflow—instead, the workflow defines which tools come in, and they aren’t siloed. This vision, and the work in our paper, doesn’t fit into just one application. It’s about bridging the different tools people are using.”
Wondering what else is happening inside Adobe Research? Check out our latest news here.
