[ https://issues.apache.org/jira/browse/SPARK-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341326#comment-14341326 ]
Sean Owen commented on SPARK-6068: ---------------------------------- Is this something for which a PR can easily be created then? it sounds like you're saying you have fixed it in your copy and that bit still resembles the original code here. Or if you'll point me at it I can try to extract the change. On a broader question, IMHO: I think you are ultimately creating a fairly different implementation in order to get in your improvements, and it's quite hard to propose a radical change to the implementation here. Especially if it changes the API or behaviors people are using. It's a shame that any library can only reasonably contain one implementation of a thing, but of course, nobody said MLlib is supposed to have everything or every bell and whistle, and we can and should be able to drop in other implementations in a Spark program as we like. Overhauls are possible at inflection points in a project lifecycle, and there is of course the 'pipelines' API rewrite going on now, note. I don't have any view on how realistic it is to drop in your work as the new implementation there. Failing that, I wonder if a lot of the improvements you've suggested here require a substantial rewrite, and won't realistically happen on the current impl? For those, I might suggested withdrawing the existing PRs / JIRAs and instead leave one placeholder JIRA summarizing the key features you'd like to see in a future rewrite, and track it as a feature request. It's up to your judgment but I suggest it since you say you abandoned the open PR. It would still be good to get in any smaller clear-win changes that can be 'back-ported'. > KMeans Parallel test may fail > ----------------------------- > > Key: SPARK-6068 > URL: https://issues.apache.org/jira/browse/SPARK-6068 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.2.1 > Reporter: Derrick Burns > Labels: clustering > Original Estimate: 24h > Remaining Estimate: 24h > > The test "k-means|| initialization in KMeansSuite can fail when the random > number generator is truly random. > The test is predicated on the assumption that each round of K-Means || will > add at least one new cluster center. The current implementation of K-Means > || adds 2*k cluster centers with high probability. However, there is no > deterministic lower bound on the number of cluster centers added. > Choices are: > 1) change the KMeans || implementation to iterate on selecting points until > it has satisfied a lower bound on the number of points chosen. > 2) eliminate the test > 3) ignore the problem and depend on the random number generator to sample the > space in a lucky manner. > Option (1) is most in keeping with the contract that KMeans || should provide > a precise number of cluster centers when possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org