I've written code for an RBM to make sure I understand how they work. Unfortunately, it seems to work for some things (MNIST features look great) and not for others and I hate the feeling that I don't quite understand the choices when setting meta-parameters. I've read Hinton's paper on this which was very helpful but it's a few years old now and only has a few comments about training with real-valued data.
The main information I feel like I'm missing is how the number of input layers, number of hidden layers, learning rate and regularization methods interact. As far as I can tell, the proper number of hidden units is "lots + drop(Out/Connect)" and the learning rate for real valued neurons should be 10e-3 or 10e-4 or thereabouts.
It also seems like there are some standard weird things (lots of dead neurons, everything turning into the average value of the data, etc) that keep cropping up that I don't quite know how to interpret and I just fiddle with parameters until things improve. Probably I should have recorded the changes that fixed those things, in hindsight.
In any case, I've tried to model some real-valued time series data and it has a lot of difficulty finding anything at all. It seems like the bias units just take over and report the average value of the data and the reconstructions just report the bias unit + constant. I've tried a lot of different parameters and this seems much more resistant to my efforts. I hesitate to go the recurrent route even though that's obviously more suited to the data because I just want to learn to use this tool in a few scenarios first.
Any experts out there have some more advice?
[link][10 comments]