Re: SVD computation limit

2017-09-19 Thread Vadim Semenov
This may also be related to https://issues.apache.org/jira/browse/SPARK-22033 On Tue, Sep 19, 2017 at 3:40 PM, Mark Bittmann wrote: > I've run into this before. The EigenValueDecomposition creates a Java > Array with 2*k*n elements. The Java Array is indexed with a native

Re: SVD computation limit

2017-09-19 Thread Mark Bittmann
I've run into this before. The EigenValueDecomposition creates a Java Array with 2*k*n elements. The Java Array is indexed with a native integer type, so 2*k*n cannot exceed Integer.MAX_VALUE values. The array is created here: https://github.com/apache/spark/blob/master/mllib/src/

SVD computation limit

2017-09-19 Thread Alexander Ovcharenko
Hello guys, While trying to compute SVD using computeSVD() function, i am getting the following warning with the follow up exception: 17/09/14 12:29:02 WARN RowMatrix: computing svd with k=49865 and n=191077, please check necessity IllegalArgumentException: u'requirement failed: k = 49865 and/or