[
https://issues.apache.org/jira/browse/MAHOUT-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147210#comment-13147210
]
Jeff Eastman commented on MAHOUT-843:
-------------------------------------
I've downloaded and installed your latest patch and it mostly passed (1 hunk
failed in src/conf/driver/classes.props). I tried running the
ClusteredOutputPostProcessorTest and it failed with an IOException: wrong value
class at ClusterOutputPostProcessor line 94.
Looking at your unit test, I'd suggest simplifying it a lot:
- Use the sequential version of Canopy to create your top clusteredPoints
directory. It writes the same files as the mapreduce version and runs a lot
faster during a build.
- Skip the k-means step as it adds no value when testing the postprocessor. The
canopy clusteredPoints are all you need.
- Get your sequential version of postProcessor working and verify that the
points output to the respective input directories for the bottom clustering are
correct.
- Run a bottom clustering canopy job if you want to prove you got the input
file directories right in the previous step, but make it sequential too
- Delete the SpectralKMeans stuff. It uses an affinity matrix as input and not
a list of input vectors. It also won't produce clusteredPoints like the other
algos. I'd concentrate on Canopy, KMeans, FuzzyK, MeanShift and Dirichlet which
all behave similarly.
- Make a new small patch with just the postprocessor stuff in it.
- Write a small shell script to invoke the canopy top, the postprocessor and
the canopy bottom using the CLIs for both. Maybe have a couple of flavors using
different top/bottom combinations.
>From a minimalist point of view, this would make a reasonable Mahout
>submission to enable hierarchical clustering
> Top Down Clustering
> -------------------
>
> Key: MAHOUT-843
> URL: https://issues.apache.org/jira/browse/MAHOUT-843
> Project: Mahout
> Issue Type: New Feature
> Components: Clustering
> Affects Versions: 0.6
> Reporter: Paritosh Ranjan
> Labels: clustering, patch
> Fix For: 0.6
>
> Attachments: MAHOUT-843-patch, MAHOUT-843-patch-v1,
> Top-Down-Clustering-patch
>
>
> Top Down Clustering works in multiple steps. The first step is to find
> comparative bigger clusters. The second step is to cluster the bigger chunks
> into meaningful clusters. This can performance while clustering big amount of
> data. And, it also removes the dependency of providing input clusters/numbers
> to the clustering algorithm.
> The "big" is a relative term, as well as the smaller "meaningful" terms. So,
> the control of this "bigger" and "smaller/meaningful" clusters will be
> controlled by the user.
> Which clustering algorithm to be used in the top level and which to use in
> the bottom level can also be selected by the user. Initially, it can be done
> for only one/few clustering algorithms, and later, option can be provided to
> use all the algorithms ( which suits the case ).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
