[
https://issues.apache.org/jira/browse/MAHOUT-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489762#comment-13489762
]
Paritosh Ranjan edited comment on MAHOUT-1103 at 11/2/12 9:23 PM:
------------------------------------------------------------------
I tried to test cluster post processing with Junit but the problem is while
running through Junits ( standalone mode ), the numpartitions is always 1. So,
the partitioner does not behave in the way it should.
I will try running it on a cluster to test the results.
I think the current approach to post process is highly dependent on cluster ids
+ partitioner which is somehow not correct. Any change in clustering approach
will have an impact on cluster post processing. I will also think of another
ways to solve it.
was (Author: paritoshranjan):
I tried test cluster post processing with Junit but the problem is while
running through Junits ( standalone mode ), the numpartitions is always 1. So,
the partitioner does not behave in the way it should.
I will try running it on a cluster to test the results.
I think the current approach to post process is highly dependent on cluster ids
+ partitioner which is somehow not correct. Any change in clustering approach
will have an impact on cluster post processing. I will also think of another
ways to solve it.
> clusterpp is not writing directories for all clusters
> -----------------------------------------------------
>
> Key: MAHOUT-1103
> URL: https://issues.apache.org/jira/browse/MAHOUT-1103
> Project: Mahout
> Issue Type: Bug
> Components: Clustering
> Affects Versions: 0.8
> Reporter: Matt Molek
> Assignee: Paritosh Ranjan
> Labels: clusterpp
> Attachments: MAHOUT-1103.patch
>
>
> After running kmeans clustering on a set of ~3M points, clusterpp fails to
> populate directories for some clusters, no matter what k is.
> I've tested this on my data with k = 300, 250, 150, 100, 50, 25, 10, 5, 2
> Even with k=2 only one cluster directory was created. For each reducer that
> fails to produce directories there is an empty part-r-* file in the output
> directory.
> Here is my command sequence for the k=2 run:
> {noformat}bin/mahout kmeans -i ssvd2/USigma -c 2clusters/init-clusters -o
> 2clusters/pca-clusters -dm
> org.apache.mahout.common.distance.TanimotoDistanceMeasure -cd 0.05 -k 2 -x 15
> -cl
> bin/mahout clusterdump -i 2clusters/pca-clusters/clusters-*-final -o
> 2clusters.txt
> bin/mahout clusterpp -i 2clusters/pca-clusters -o 2clusters/bottom{noformat}
> The output of clusterdump shows two clusters: VL-3742464 and VL-3742466
> containing 2585843 and 1156624 points respectively.
> Discussion on the user mailing list suggested that this might be caused by
> the default hadoop hash partitioner. The hashes of these two clusters aren't
> identical, but they are close. Putting both cluster names into a Text and
> caling hashCode() gives:
> VL-3742464 -> -685560454
> VL-3742466 -> -685560452
> Finally, when running with "-xm sequential", everything performs as expected.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
