Existing approaches to interaction with digital video are complex, and some operations lack the immediacy of interactive feedback. Our research in video annotation, visualization, and interaction harnesses computer vision to aid users in understanding and communicating with digital video. We are developing new approaches for computing the motion of points and objects in a video clip, and interactive systems that utilize this data to visually annotate independently moving objects in the video. We have applied this interface to construct visualizations of a short video clip in a single static image, using the visual language of storyboards. The principal advantage of the storyboard representation over standard representations of video is that it requires only a moment to observe and comprehend but at the same time retains much of the detail of the source video. The layout of the storyboard can be optimized to place the elements in a configuration that maximizes the clarity of presentation. We also explore novel interaction techniques for random video frame access using the natural spatial dimensions of a storyboard representation or an individual video frame.
A layman’s introductory video to these technologies can be found here, and the official supplementary video for the paper is embedded below: