[ https://issues.apache.org/jira/browse/SPARK-6068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341634#comment-14341634 ]
Sean Owen commented on SPARK-6068: ---------------------------------- Yes, it seems like too much change to the existing version. From https://github.com/apache/spark/pull/2634 it seems like there are just some differences of opinion about what's worth doing and how. I think the only way forward would be to propose integration what you've done for the new version in the {{.ml}} package, because it's not clear the existing PR isn't going to proceed. I'm hoping to just drive a resolution to what is almost one big issue rather than leave it hanging. I'm looking at the ~8 JIRAs for k-means you created: https://issues.apache.org/jira/issues/?jql=project%20%3D%20SPARK%20AND%20reporter%20%3D%20%22Derrick%20Burns%22%20AND%20resolution%20%3D%20Unresolved I assume a couple (like this one) are 'back-portable' from your work to the existing impl. Can we zap those and close them with a PR? This would be great and I'd like to help get those quick wins in. The rest sound like interdependent aspects of one proposal: create a new k-means implementation with different design and properties X / Y / Z, and use it in the new pipelines API. (I can't say whether this would be accepted or not but that's what's on the table). I'd rather coherently collect that rather than have it live in pieces in JIRA, esp. since I'm getting the sense these remaining pieces won't otherwise move forward. > KMeans Parallel test may fail > ----------------------------- > > Key: SPARK-6068 > URL: https://issues.apache.org/jira/browse/SPARK-6068 > Project: Spark > Issue Type: Bug > Components: MLlib > Affects Versions: 1.2.1 > Reporter: Derrick Burns > Labels: clustering > Original Estimate: 24h > Remaining Estimate: 24h > > The test "k-means|| initialization in KMeansSuite can fail when the random > number generator is truly random. > The test is predicated on the assumption that each round of K-Means || will > add at least one new cluster center. The current implementation of K-Means > || adds 2*k cluster centers with high probability. However, there is no > deterministic lower bound on the number of cluster centers added. > Choices are: > 1) change the KMeans || implementation to iterate on selecting points until > it has satisfied a lower bound on the number of points chosen. > 2) eliminate the test > 3) ignore the problem and depend on the random number generator to sample the > space in a lucky manner. > Option (1) is most in keeping with the contract that KMeans || should provide > a precise number of cluster centers when possible. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org