[
https://issues.apache.org/jira/browse/SPARK-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345575#comment-14345575
]
Denis Dus commented on SPARK-6137:
----------------------------------
As I see, splitting rules are different.
1) GMeans using Anderson-Darling test in original paper (but, in general, any
other test of data normality can be used).The main idea is: If cluster's data
looks gaussian, then no splitting is needed, otherwise GMeans willl split
cluster's centroid into new two.
2) I looked in the code, and it seems, that Streaming KMeans using some custom
empirical rule.
I think, that the main advantage of GMeans is that it has a reasonable idea
(understandable to human) for subsplitting clusters (if data in cluster doesn't
look as gaussian -> split data into new two parts) and the process of splitting
has a rather good statistical reasons (based on well-known non-parametric
statistical test and etc.).
> G-Means clustering algorithm implementation
> -------------------------------------------
>
> Key: SPARK-6137
> URL: https://issues.apache.org/jira/browse/SPARK-6137
> Project: Spark
> Issue Type: New Feature
> Components: MLlib
> Reporter: Denis Dus
> Priority: Minor
>
> Will it be useful to implement G-Means clustering algorithm based on K-Means?
> G-means is a powerful extension of k-means, which uses test of cluster data
> normality to decide if it necessary to split current cluster into new two.
> It's relative complexity (compared to k-Means) is O(K), where K is maximum
> number of clusters.
> The original paper is by Greg Hamerly and Charles Elkan from University of
> California:
> [http://papers.nips.cc/paper/2526-learning-the-k-in-k-means.pdf]
> I also have a small prototype of this algorithm written in R (if anyone is
> interested in it).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
