Layout-induced Video Representation for Recognizing Agent-in-Place Actions

International Conference on Computer Vision (ICCV)

Published October 29, 2019

Ruichi Yu, Hongcheng Wang, Ang Li, Jingxiao Zheng, Vlad Morariu, Larry S. Davis

We address scene layout modeling for recognizing agentin-place actions, which are actions associated with agents who perform them and the places where they occur, in the context of outdoor home surveillance. We introduce a novel representation to model the geometry and topology of scene layouts so that a network can generalize from the layouts observed in the training scenes to unseen scenes in the test set. This Layout-Induced Video Representation (LIVR) abstracts away low-level appearance variance and encodes geometric and topological relationships of places to explicitly model scene layout. LIVR partitions the semantic features of a scene into different places to force the network to learn generic place-based feature descriptions which are independent of specific scene layouts; then, LIVR dynamically aggregates features based on connectivities of places in each specific scene to model its layout. We introduce a new Agent-in Place Action (APA) dataset to show that our method allows neural network models to generalize significantly better to unseen scenes.

Learn More