7000x7000 is not tall-and-skinny matrix. Storing the dense matrix
requires 784MB. The driver needs more storage for collecting result
from executors as well as making a copy for LAPACK's dgesvd. So you
need more memory. Do you need the full SVD? If not, try to use a small
k, e.g, 50. -Xiangrui
On
Hi Xianguri,
After setting SVD to smaller value (200) its working.
Thanks,
Shailesh
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-OutOfMemoryError-while-running-SVD-MLLib-example-tp14972p15179.html
Sent from the Apache Spark User List
Note, the data is random numbers (double).
Any suggestions/pointers will be highly appreciated.
Thanks,
Shailesh
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/java-lang-OutOfMemoryError-while-running-SVD-MLLib-example-tp14972p15083.html
Sent from the
Hello,
I am new to Spark. I have downloaded Spark 1.1.0 and trying to run the
TallSkinnySVD.scala example with different input data sizes. I tried with
input data with 1000X1000 matrix, 5000X5000 matrix.
Though I had faced some Java Heap issues I added following parameters in
spark-defaults.conf
