[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
GitHub user mengxr opened a pull request: https://github.com/apache/spark/pull/117 [MLLIB-18] [WIP] Adding sparse data support and update KMeans Continue our discussions from https://github.com/apache/incubator-spark/pull/575 This PR is WIP because it depends on a SNAPSHOT version of breeze. Per previous discussions and benchmarks, I switched to breeze for linear algebra operations. @dlwh and I made some improvements to breeze to keep its performance comparable to the bare-bone implementation, including norm computation and squared distance. This is why this PR needs to depend on a SNAPSHOT version of breeze. @fommil , please find the notice of using netlib-core in `NOTICE`. This is following Apache's instructions on appropriate labeling. I'm going to update this PR to include: 1. Fast distance computation: using `\|a\|_2^2 + \|b\|_2^2 - 2 a^T b` when it doesn't introduce too much numerical error. The squared norms are pre-computed. Otherwise, computing the distance between the center (dense) and a point (possibly sparse) always takes O(n) time. 2. Some numbers about the performance. 3. A released version of breeze. @dlwh, a minor release of breeze will help this PR get merged early. Do you mind sharing breeze's release plan? Thanks! You can merge this pull request into a Git repository by running: $ git pull https://github.com/mengxr/spark sparse-kmeans Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/117.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #117 commit 07ffaf2b7f3300a6c0afc2a21a0134c76d3ec8dc Author: Xiangrui Meng Date: 2014-03-10T21:27:46Z add dense/sparse vector data models and conversions to/from breeze vectors use breeze to implement KMeans in order to support both dense and sparse data --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37243343 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37243346 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37247880 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13101/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37247879 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37252898 Okay, sbt was able to fetch breeze_2.10-0.7-SNAPSHOT from Sonatype, so tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user fommil commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37284933 @mengxr this `NOTICE` is not quite correct. `netlib-core` is released under a 3 clause BSD, which I believe *is* compatible with Apache. Although you should check this to be comfortable. However, use of netlib `native` components is optional and will pull in both LGPL libraries and machine-specific libraries that use arbitrary (potentially highly commercially sensitive) licenses (e.g. Intel Math Kernel Library) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37320613 @fommil I didn't realize the bottom of https://github.com/fommil/netlib-java/blob/master/LICENSE.txt is 3-clause BSD. It is Apache authorized, so I don't need to mention it in NOTICE. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user fommil commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37353456 @mengxr cool :-D --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37372450 @fommil We will mention native libraries in the documentation once this PR gets merged. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37372615 I added fast distance computation and updated the implementation of KMeans. Squared norms of the points are pre-computed and cached in order to compute distance faster for sparse data. See the test in KMeansSuite on sparse data, which has dimension 1 (though many zeros) but runs under 5 seconds. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37372652 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37372650 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37372715 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37372716 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13120/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37377103 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37377104 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37379575 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13125/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37379573 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37461164 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37461163 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37470402 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37470409 One or more automated tests failed Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13136/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37487419 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37487420 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37491139 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/13138/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37491137 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [MLLIB-18] [WIP] Adding sparse data support an...
Github user dlwh commented on the pull request: https://github.com/apache/spark/pull/117#issuecomment-37572686 I'll release something this weekend. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
