First post, effects of cluster size on recognition rate

Hi, it's my first post here. I am working on implementing an object recognition algorithm in OpenCV. It goes like this:

1. build vocabulary from data (300 images)
2. generate response histogram from vocabulary given descriptors of image N
3. Response histograms and class labels fed into training matrix
4. training matrix fed to SVM

With 300 images, I get 70% accuracy with the algorithm (Clusters 186) With ~500 images, it goes down to 52% (Clusters 186)

So, with the 300 images and 186 clusters, im at 70%. Then I can chance cluster amount to 180 and it'll go down to 50%. Or I can change it to 500 and it'll be at 66%

How does the number of clusters effect my results?

Not sure if this is a stupid question, i'm a newb with machine learning/computer vision