Hello,
I'm in the middle of modifying some neural network code to build Stacked Autoencoders (SAEs), and I'm trying to put together some unit tests. I'm hoping some people can help me by verifying some assumptions that I'm making about SAE behavior:
A very basic autoencoder (for my purposes, a feed-forward neural network with a single hidden layer having a number of units that is smaller than the input and output layers), should act something like a sparse coding algorithm, where the weights between a hidden layer unit and the input layer define a code. To test, I've generated a data set made up of random linear superpositions of a couple target codes that I made up. Since this is only a test and I want an answer to be generated as reliably as possible, the codes are highly independent (i.e. dot at the center of an image, non-overlapping dot at the corner of an image, etc), and the hidden layer size is equal to the number of codes backing the dataset (three). I should expect the weights of a trained network to resolve the codes. Does this sound like a reasonable test?
The second test is solely for the gradient descent algorithms. The classic FFNN example you always see in textbooks is the XOR problem. You can postulate some weights for a 2-2-1 FFNN (2-2-1 denotes input layer size, hidden layer size, and output layer size, respectively) that will solve XOR. What I don't know is if simple backpropagation will reliably find this solution. My FFNN implementation seems to need at least a 2-2-2-1 network to solve XOR, and a 2-5-2-1 network to find a correct solution reliably. I'm not sure if this is a flaw in my algorithm or a just a property of backprop. Thoughts?
Thanks!
[link][2 comments]