A Not-So-Daily Paper Review: An Analysis of Single-Layer Networks in Unsupervised Feature Learning

Quick review before I go and grab some AYCE sushi with my babe. And I apologize I have not been reviewing as I once was, I was dealing with some things IRL. Note: even though I complain and have opposing views with some in this subreddit, I still love you all, and a- happy holidays to all as well.

Link to paper: [Analysis of Single-Layered Unsupervised Networks], A. Coates, A. Ng, H. Lee, Stanford University & UMichigan. (http://www.google.ca/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&ved=0CDMQFjAA&url=http%3A%2F%2Fweb.eecs.umich.edu%2F~honglak%2Fnipsdlufl10-AnalysisSingleLayerUnsupervisedFeatureLearning.pdf&ei=Gr2aUvu9CJDXoASPoYGwDw&usg=AFQjCNH4eLwXs6RD_dfgXZIM3ydM9_Vsjg&sig2=fss3O7Gib68rqEAAK28OHg&bvm=bv.57155469,d.cGU)

Abstract: A great deal of research has focused on al- gorithms for learning features from unla- beled data. Indeed, much progress has been made on benchmark datasets like NORB and CIFAR by employing increasingly complex unsupervised learning algorithms and deep models. In this paper, however, we show that several simple factors, such as the number of hidden nodes in the model, may be more im- portant to achieving high performance than the learning algorithm or the depth of the model. Specifically, we will apply several off- the-shelf feature learning algorithms (sparse auto-encoders, sparse RBMs, K-means clus- tering, and Gaussian mixtures) to CIFAR, NORB, and STL datasets using only single- layer networks. We then present a detailed analysis of the effect of changes in the model setup: the receptive field size, number of hid- den nodes (features), the step-size (“stride”) between extracted features, and the effect of whitening. Our results show that large numbers of hidden nodes and dense fea- ture extraction are critical to achieving high performance—so critical, in fact, that when these parameters are pushed to their limits, we achieve state-of-the-art performance on both CIFAR-10 and NORB using only a sin- gle layer of features. More surprisingly, our best performance is based on K-means clus- tering, which is extremely fast, has no hyper- parameters to tune beyond the model struc- ture itself, and is very easy to implement. De- spite the simplicity of our system, we achieve accuracy beyond all previously published re- sults on the CIFAR-10 and NORB datasets (79.6% and 97.2% respectively).