We call this new family of hybrid models conditional networks. We present a systematic analysis of how to fuse conditional computation with representation learning and achieve a continuum of hybrid models with different ratios of accuracy vs. CNNs achieve state of the art accuracy, thanks to their representation learning capabilities.

Decision forests are computationally efficient thanks to their conditional computation property (computation is confined to only a small region of the tree, the nodes along a single branch). This paper investigates the connections between two state of the art classifiers: decision forests (DFs, including decision jungles) and convolutional neural networks (CNNs). Applying our method to a near state-of-the-art network for CIFAR, we achieved comparable accuracy with 46% less compute and 55% fewer parameters. Applying our method to the GoogLeNet architecture for ILSVRC, we achieved comparable accuracy with 26% less compute and 41% fewer model parameters. Applying our method to an improved version of VGG-11 network using global max-pooling, we achieve comparable validation accuracy using 41% less compute and only 24% of the original VGG-11 model parameters another variant of our method gives a 1 percentage point increase in accuracy over our improved VGG-11 model, giving a top-5 center-crop validation accuracy of 89.7% while reducing computation by 16% relative to the original VGG-11 model. Our results show similar or higher accuracy than conventional CNNs with much less compute. We validate our approach by applying it to several existing CNN architectures and training these networks from scratch using the CIFAR, ILSVRC and MIT Places datasets. This allows effective initialization of connection weights in convolutional layers composed of groups of differently-shaped filters.

To train such networks, a novel weight initialization scheme is used. Rather than approximating filters in previously-trained networks with more efficient versions, we learn a set of small basis filters from scratch during training, the network learns to combine these basis filters into more complex filters that are discriminative for image classification. We propose a new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of convolutional filters.

Free downloads Zip Express 2.18.2.1