I was wondering what the preferred way to handle (often large-dimensional) parameter sets in Python is, as you'd find in neural networks and Bayesian variational methods (particularly the latter). I am especially wondering how you make your code interface well with gradient methods.
Numpy functionality seems a must, but storing parameters in separate named arrays has two weaknesses as I see it.
- Operating on the entire parameter set, e.g. adding a gradient, is impossible, so you have to write the operation for each array of parameters explicitly.
- If you have variations of the model with only a subset of the parameters inheritance becomes difficult to work with.
Storing everything in a single array/vector seems like a terrible solution as the code would become unintelligible. So what is a better way to do it? A dictionary so you can loop over the parameters? A Pandas Panel frame? A parameter class with its own update method?
Would there be any significant overhead on using a dictionary of a few large arrays and then looping over the keys, as opposed to writing each operation explicitly?
[link][5 comments]