A while back I needed a decision tree learning implementation in Java and wasn't satisfied with any of the options out there, so I decided to roll my own, you can find the result here. My goals were to make it efficient enough to run on Google App Engine, and have a clean and fluent API consistent with modern Java API design approaches.
I tried a variety of different split scoring algorithms and eventually settled on this one which is of my own design, but seemed to yield better results on various datasets I tested with. The code is of reasonable quality, but could use a lot more unit testing, and there are some complex methods in there that should be broken down.
I came up with an approach to handling nominal attributes (eg. gender, city, zipcode) which seemed smart at the time, but now I'm having second thoughts and wanted some feedback.
So let's say I'm trying to find an optimal split for a "city" attribute in a dataset. Nominal branches in the decision tree are of the form "is the value in the following set of values?".
I start by picking the first city, placing it in this "in" set, and scoring this split. I do this for each successive city, in each case creating a set of size 1 and testing the split. Once I've tested all cities I pick the best one, and it becomes the first city in my "in" set. I then repeat this process, adding each second city to the set, and testing the split.
I keep adding cities to the "in" set until adding a new city results in a worse split score, at this point I'm done, I have my nominal decision node.
You can see an example of the type of decision tree this builds here.
My concern is that the trees built using this approach rapidly approach 100% recall on the training set, even with a relatively limited depth. My concern is that I'm giving too much flexibility to the decision tree builder to construct any set whatsoever at each nominal branch.
I'd appreciate any thoughts on this.
[link][3 comments]