I'd like propose some research directions in large scale deep learning (so that those of you in academia can publish papers and code and I can get the fruits of that labor).
The basic premise - stacking autoencoders in various ways, where an autoencoder is simple a 3 layer neural network whose in and out layers are the same size and the hidden layer is smaller. Training consists of minimizing the difference between data going into and coming out of the network. Afterwards the hidden-to-out layer is thrown away (the decoder stage) and the in-to-hidden is kept (the encoder stage). Training can be sgd or simulated annealing or ga etc.
Multiple autoencoders can be trained over the same data (from random starting spaces) and they will land on different "hills" of the optimization landscape (so that ensembles will tend work better).
The output of one encoder stage can be fed as data into 1 or more other autoencoders, so that you have a directed acyclic graph of encoder stages. Further, this encoder dag will tend to be too large to fit in the memory of one computer and will be distributed across a network.
Research+code:
Determining whether two encoders have hit the same "hill". This can be done wither be directly examining the connection weights or examining the data output. If the same hill has been reached, methods for caching/memoizing the data input so that encoder dag is more efficient.
Building a "library" as in genomics/connectomics, the "artificial connectome", of the encoders and encoder dags, over various datasets such as imagenet, speech dataset, wikipedia, human genome etc.. Analysis on those artificial connectomes, and between connectomes (e.g. have the same encoder dags been found in the wikipedia connectome and the imagenet connectome).
performance improvements, such as creating fpga/asic for autoencoder training.
[link] [4 comments]