[
https://issues.apache.org/jira/browse/MAHOUT-588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990317#comment-12990317
]
Ted Dunning commented on MAHOUT-588:
------------------------------------
{quote}
Your suggestion to implement v2 (centroid vector) by means of a hashmap would
definitelly improve the speed of calculating the distance between points and
centroids and as a result the kMeans itself.
{quote}
Uh.... my suggestion was to use a hashed feature encoding of the feature
vector, not to use a hashmap. Hashed feature encoding assigns multiple
features to each position in the vector and multiple locations to each feature.
It says nothing about the matrix representation.
For speed considerations alone, if the centroid vector has more than about
10-20% of the elements non-zero, then you should avoid sparse representations
and just use the dense form. If space is critical then you may want to use a
sparse representation up to about 30-40% fill.
So how sparse is your centroid vector?
> Benchmark Mahout's clustering performance on EC2 and publish the results
> ------------------------------------------------------------------------
>
> Key: MAHOUT-588
> URL: https://issues.apache.org/jira/browse/MAHOUT-588
> Project: Mahout
> Issue Type: Task
> Reporter: Grant Ingersoll
> Attachments: SequenceFilesFromMailArchives.java,
> SequenceFilesFromMailArchives2.java, TamingAnalyzer.java,
> TamingCollocDriver.java, TamingCollocMapper.java, TamingDictVect.java,
> TamingDictionaryVectorizer.java, TamingGramKeyGroupComparator.java,
> TamingTFIDF.java, TamingTokenizer.java, Top1000Tokens_maybe_stopWords,
> Uncompress.java, clusters1.txt, clusters_kMeans.txt,
> distcp_large_to_s3_failed.log, seq2sparse_small_failed.log,
> seq2sparse_xlarge_ok.log
>
>
> For Taming Text, I've commissioned some benchmarking work on Mahout's
> clustering algorithms. I've asked the two doing the project to do all the
> work in the open here. The goal is to use a publicly reusable dataset (for
> now, the ASF mail archives, assuming it is big enough) and run on EC2 and
> make all resources available so others can reproduce/improve.
> I'd like to add the setup code to utils (although it could possibly be done
> as a Vectorizer) and the publication of the results will be put up on the
> Wiki as well as in the book. This issue is to track the patches, etc.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
