[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK
Github user yangliuyu commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-48037519 @mengxr yes, userSize is 10 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK
Github user yangliuyu commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-47886598 @mengxr k is 100, rCond, tol and maxIterations are all defaults, i.e. 1e-9, 1e-10, 300. Change iteration number from 300 to 200 will not reduce too much time cost 10% (test on another small size of dataset 61794 x 100, k=99 ~110s) btw, improve RowMatrix multiply patch works well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK
Github user yangliuyu commented on the pull request:
https://github.com/apache/spark/pull/964#issuecomment-48007755
@vrilleup I missed persist the RDD[Vector], but after added, it only help
reduce time cost from 20+s to 10+s, subsequent aggregate tasks still take more
than 10s. For the case stage 47, Scheduler delay and gc takes too much time.
The matrix is 800371 x 10, 29898284 non-zeros. Our testing env only has 16
cores, so don't know whether it will get better performance on large number of
cores.
```scala
val data = input.map { case (sid, uid) =
(songId2IndexMap(sid), userId2IndexMap(uid))
}.groupByKey().sortByKey()
.map({ case (sid, uids) =
val uidList = uids.toSet.toList.sorted
val uidSeq = uidList.map(uid = (uid, v))
Vectors.sparse(userSize, uidSeq)
}).persist()
val mat = new RowMatrix(data)
val svd = mat.computeSparseSVD(100, computeU = true)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: SPARK-1782: svd for sparse matrix using ARPACK
Github user yangliuyu commented on the pull request: https://github.com/apache/spark/pull/964#issuecomment-47795986 @vrilleup about your performance test result for real matrices, what's the cpu usage rate for each executor? We run svd on a 205899 x 1000 sparse matrix, 1850566 non-zeros, (Spark standalone mode, run 2 worker instance on one machine, one executor for each worker, 16 cores total) it cost 18 minutes not including getting the U matrix, cpu usage is about 200%-400% per executor. Seems not a reasonable time cost even on small number of executors. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
