Edinburgh Napier University

  Deep convolutional networks are widely used in video
action recognition. 3D convolutions are one prominent approach to deal with the additional time dimension. While
3D convolutions typically lead to higher accuracies, the inner workings of the trained models are more difficult to interpret. We focus on creating human-understandable visual
explanations that represent the hierarchical parts of spatiotemporal networks. We introduce Class Feature Pyramids,
a method that traverses the entire network structure and
incrementally discovers kernels at different network depths
that are informative for a specific class. Our method does
not depend on the network’s architecture or the type of 3D
convolutions, supporting grouped and depth-wise convolutions, convolutions in fibers, and convolutions in branches.
We demonstrate the method on six state-of-the-art 3D convolution neural networks (CNNs) on three action recognition (Kinetics-400, UCF-101, and HMDB-51) and two
egocentric action recognition datasets (EPIC-Kitchens and
EGTEA Gaze+).