One thing I have used this for was to create codebooks for SIFT features in images. It is a common, though fairly naïve, method for converting high dimensional features into a simple word-like features. Thus, if you have 200 SIFT features for an image, you can reduce that to 200 ‘words’ that can be directly compared across your entire image set. The drawback is that usually some parts of the feature space are much more dense than other parts and distinctive features could be lost. You can try to minimize that by increasing K, but there are diminishing returns. If you measure the quality of your clusters in such situations, you will find that the quality levels off between 1000 and 4000 clusters (at least it did for my SIFT feature set, YMMV on other data sets).
Dave From: Chester Chen [mailto:[email protected]] Sent: Monday, April 28, 2014 9:31 AM To: [email protected] Cc: [email protected] Subject: Re: K-means with large K David, Just curious to know what kind of use cases demand such large k clusters Chester Sent from my iPhone On Apr 28, 2014, at 9:19 AM, "Buttler, David" <[email protected]<mailto:[email protected]>> wrote: Hi, I am trying to run the K-means code in mllib, and it works very nicely with small K (less than 1000). However, when I try for a larger K (I am looking for 2000-4000 clusters), it seems like the code gets part way through (perhaps just the initialization step) and freezes. The compute nodes stop doing any CPU / network / IO and nothing happens for hours. I had done something similar back in the days of Spark 0.6, and I didn’t have any trouble going up to 4000 clusters with similar data. This happens with both a standalone cluster, and in local multi-core mode (with the node given 200GB of heap), but eventually completes in local single-core mode. Data statistics: Rows: 166248 Columns: 108 This is a test run before trying it out on much larger data Any ideas on what might be the cause of this? Thanks, Dave
