I've been tasked with coming up with a way of ranking the difficulty of elements in a problem set. We have a set of about 30 "problems", which we will get a binary correct/incorrect outcome from when an individual goes through the problem set. We also have data now for about 300 individuals going through this set.
The end result is we want a weighting for each of these problems, based on its difficulty, so that we can decide which did best. In earlier versions we have simply manually assigned a point value, and tallied up the points, but we are concerned about mis-valuing problems, since we have in the past. These problems have a large range of difficulty, the easiest of which nearly everyone got correct, while some of the most difficult on had a single solve (by different individuals).
One of the proposed solutions is just to assign a point value for all problems, then simply decrease the point value by some scheme based on the number of correct solves. While that would certainly work and be fairly accurate, it seems like there should be a more elegant solution than that.
I've spent some time looking for sources on how to do this, but I think a lot of the problem is I just don't know the right Machine Learning terms to find what I'm looking for. I'm hoping that you all will know what I'm talking about enough to point me in the right direction.
[link][3 comments]