Hi ML, I've been competing in several Kaggle competitions. I can get in the top 10-20%, but after that my rank stagnates. My template workflow is to process the data(derive every variable I can come up with), partition the data into k-folds or k-stratified-folds, (optionally or separately) perform feature selection, throw every algorithm appropriate from sklearn/scipy/theano-scripts into my model class then bag the models. I do grid-searching on the hyper-parameters for algorithms that perform poorly. I try to keep my individual model fitting time to under 12 hours on my 8-core workstation. I can usually eek out a little more if I run add models ran with more of the features to my bag.
I have a good understanding of many of the algorithms that sklearn implements, but I don't have the foggiest clue how this knowledge could help me except in something like diagnosing stochastic gradient descent. I've implemented a couple(slower and less pretty) versions of them. I've worked through Elements of Statistical Learning. I've well-versed in hypothesis testing and visualizing data(my job). I don't know where to go from here. I thought I was pretty good at building models, but I am humbled by the leaderboard.
Where do I go from here? What books do I read? What software/languages do I learn? I'm fine with logarithmic returns on my effort, but I don't know where to find them.
[link] [20 comments]