[
https://issues.apache.org/jira/browse/MAHOUT-843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13145680#comment-13145680
]
Jeff Eastman commented on MAHOUT-843:
-------------------------------------
Paritosh,
Top-down Clustering involves running a top driver A which produces clustered
points as output. Then a postprocessing job moves each of the k cluster's
points into a separate folder which is given as input to each of the k bottom
clustering drivers B. Given the existence of the postprocessing job then a user
can elect to code it entirely in Java or write a shell script to use the CLI
for each of the three steps.
Though your postprocessor is still sequential and will not scale to large
datasets, I see creating a CLI for a M/R version of this as being the smallest
incremental change to Mahout which will facilitate top-down clustering. Since
each choice of A and B clustering algorithms carries its own set of parameters,
I see this as making an overall CLI to bundle the entire top-down process as
problematic.
I can see you approach in the pure Java implementation is creating config and
executor classes which bundle up the top and bottom cluster driver parameters
and then orchestrate the top-down clustering process. Then, in the middle, the
postprocessor is run to set up the bottom clustering folders. This is not a
complicated pattern for users to do manually: configure A and run it; run the
postprocessor; then configure B and run it against each of the bottom level
input dictionaries.
>From a minimalize perspective, all we really need is a scalable postprocessor
>with Java driver & CLI and an example shell script that shows how to do
>top-down with one particular set of A and B.
> Top Down Clustering
> -------------------
>
> Key: MAHOUT-843
> URL: https://issues.apache.org/jira/browse/MAHOUT-843
> Project: Mahout
> Issue Type: New Feature
> Components: Clustering
> Affects Versions: 0.6
> Reporter: Paritosh Ranjan
> Labels: clustering, patch
> Fix For: 0.6
>
> Attachments: MAHOUT-843-patch, Top-Down-Clustering-patch
>
>
> Top Down Clustering works in multiple steps. The first step is to find
> comparative bigger clusters. The second step is to cluster the bigger chunks
> into meaningful clusters. This can performance while clustering big amount of
> data. And, it also removes the dependency of providing input clusters/numbers
> to the clustering algorithm.
> The "big" is a relative term, as well as the smaller "meaningful" terms. So,
> the control of this "bigger" and "smaller/meaningful" clusters will be
> controlled by the user.
> Which clustering algorithm to be used in the top level and which to use in
> the bottom level can also be selected by the user. Initially, it can be done
> for only one/few clustering algorithms, and later, option can be provided to
> use all the algorithms ( which suits the case ).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
