34th ACM International Conference on Multimedia (MM 2026)

LoopEdit: Learning Data-driven Hierarchical Planners for Short-Video Assembly

Saumya Gupta, Ishita Dasgupta, Stefano Petrangeli, Jayakumar Subramanian

Read Research Paper

Creating a compelling social media video from raw clips requires decisions at multiple levels, from story structure and pacing to clip selection and local editing effects. Existing language-based editing systems typically respond to explicit edit commands, retrieve clips, or generate videos directly, but they do not produce an inspectable plan that can be optimized for engagement. We present LoopEdit, a hierarchical multi-LLM framework that converts a user directive and a set of input assets into an editable short-video plan and a rendered video. LoopEdit first generates a high-level plan that defines the video arc, scene structure, pacing, and style, and then generates a low-level plan that maps scenes to specific clips, durations, effects, text, and narration. To train this system, we reformulate existing short-videos into asset-plan pairs through reverse-engineering. We also learn evaluator models from human annotations to predict how well a candidate video plan will perform, capturing both overall engagement and fine-grained attributes that drive it. These learned evaluators also serve as reward models to align the hierarchical planners with human preferences through reinforcement learning (GRPO). In our experiments, the learned evaluators correlate strongly with human judgments, and the optimized planners outperform strong zero-shot Multimodal LLM baseline in automatic evaluation and user studies. These results show that data-driven hierarchical planning with preference alignment can produce more engaging and controllable short-form videos.