GitHub user mengxr opened a pull request:
https://github.com/apache/spark/pull/117
[MLLIB-18] [WIP] Adding sparse data support and update KMeans
Continue our discussions from
https://github.com/apache/incubator-spark/pull/575
This PR is WIP because it depends on a SNAPSHOT version of breeze.
Per previous discussions and benchmarks, I switched to breeze for linear
algebra operations. @dlwh and I made some improvements to breeze to keep its
performance comparable to the bare-bone implementation, including norm
computation and squared distance. This is why this PR needs to depend on a
SNAPSHOT version of breeze.
@fommil , please find the notice of using netlib-core in `NOTICE`. This is
following Apache's instructions on appropriate labeling.
I'm going to update this PR to include:
1. Fast distance computation: using `\|a\|_2^2 + \|b\|_2^2 - 2 a^T b` when
it doesn't introduce too much numerical error. The squared norms are
pre-computed. Otherwise, computing the distance between the center (dense) and
a point (possibly sparse) always takes O(n) time.
2. Some numbers about the performance.
3. A released version of breeze. @dlwh, a minor release of breeze will help
this PR get merged early. Do you mind sharing breeze's release plan? Thanks!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mengxr/spark sparse-kmeans
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/117.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #117
----
commit 07ffaf2b7f3300a6c0afc2a21a0134c76d3ec8dc
Author: Xiangrui Meng <[email protected]>
Date: 2014-03-10T21:27:46Z
add dense/sparse vector data models and conversions to/from breeze vectors
use breeze to implement KMeans in order to support both dense and sparse
data
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
