Re: TallSkinnyQR

when using multiple executors/machines; Only the R (not the Q) in tallSkinnyQR is correct when using multiple executors/machines. U and Q were being stored in RowMatrix format. There is no index information about RowMatrix, so it does not make sense for U and Q.Others have run into this same pro

Re: RowMatrix: tallSkinnyQR

Since this function is used to compute QR decomposition for RowMatrix of a tall and skinny shape, the output R is always with small rank. [image: Inline image 1] On Fri, Jun 9, 2017 at 10:33 PM, Arun <arunbm...@gmail.com> wrote: > hi > > *def tallSkinnyQR(computeQ:

RowMatrix: tallSkinnyQR

hi *def tallSkinnyQR(computeQ: Boolean = false): QRDecomposition[RowMatrix, Matrix]* *In output of this method Q is distributed matrix* *and R is local Matrix* *Whats the reason R is Local Matrix?* -Arun

Re: TallSkinnyQR

to be a bug in the section of code that converts the RowMatrix > format back into indexedRowMatrix format. > > For RowMatrix, I think the singular values and right singular vectors > (not the left singular vectors U) that computeSVD computes are correct when > using multiple executors/mach

Re: TallSkinnyQR

here seems to be a bug in the section of code that converts the >>>> RowMatrix format back into indexedRowMatrix format. >>>> >>>> For RowMatrix, I think the singular values and right singular vectors >>>> (not the left singular vector

Re: TallSkinnyQR

nverts the >>> RowMatrix format back into indexedRowMatrix format. >>> >>> For RowMatrix, I think the singular values and right singular vectors >>> (not the left singular vectors U) that computeSVD computes are correct when >>> using multiple executors

Re: TallSkinnyQR

gt; RowMatrix format back into indexedRowMatrix format. >> >> For RowMatrix, I think the singular values and right singular vectors >> (not the left singular vectors U) that computeSVD computes are correct when >> using multiple executors/machines; Only the R (not the Q) in t

Re: TallSkinnyQR

> > For RowMatrix, I think the singular values and right singular vectors > (not the left singular vectors U) that computeSVD computes are correct when > using multiple executors/machines; Only the R (not the Q) in tallSkinnyQR > is correct when using multiple executors/mach

Re: TallSkinnyQR

/machines; Only the R (not the Q) in tallSkinnyQR is correct when using multiple executors/machines. U and Q were being stored in RowMatrix format. There is no index information about RowMatrix, so it does not make sense for U and Q. Others have run into this same problem ( https://issues.apache.org

Re: TallSkinnyQR

I have a different question that might be trivial for you (although not to me :)) Maybe you can answer this? Here is a MapReduce Example implemented in Java. It reads each line of text and for each word in the line of text determines if it starts with an upper case. If so, it creates a key value

Re: TallSkinnyQR

Ok thanks. On Fri, Dec 2, 2016 at 8:19 AM Sean Owen wrote: > I tried, but enforcing the ordering changed a fair bit of behavior and I > gave up. I think the way to think of it is: a RowMatrix has whatever > ordering you made it with, so you need to give it ordered rows if

Re: TallSkinnyQR

I tried, but enforcing the ordering changed a fair bit of behavior and I gave up. I think the way to think of it is: a RowMatrix has whatever ordering you made it with, so you need to give it ordered rows if you're going to use a method like the QR decomposition. That works. I don't think the QR

Re: TallSkinnyQR

Hi guys, Was this bug ever resolved? Iman On Fri, Nov 11, 2016 at 9:59 AM Iman Mohtashemi wrote: > Yes this would be helpful, otherwise the Q part of the decomposition is > useless. One can use that to solve the system by transposing it and > multiplying with b and

Re: TallSkinnyQR

Yes this would be helpful, otherwise the Q part of the decomposition is useless. One can use that to solve the system by transposing it and multiplying with b and solving for x (Ax = b) where A = R and b = Qt*b since the Upper triangular matrix is correctly available (R) On Fri, Nov 11, 2016 at

Re: TallSkinnyQR

@Xiangrui / @Joseph, do you think it would be reasonable to have CoordinateMatrix sort the rows it creates to make an IndexedRowMatrix? in order to make the ultimate output of toRowMatrix less surprising when it's not ordered? On Tue, Nov 8, 2016 at 3:29 PM Sean Owen wrote:

Re: TallSkinnyQR

Thanks Sean! Let me take a look! Iman On Nov 8, 2016 7:29 AM, "Sean Owen" wrote: > I think the problem here is that IndexedRowMatrix.toRowMatrix does *not* > result in a RowMatrix with rows in order of their indices, necessarily: > > // Drop its row indices. > RowMatrix

Re: TallSkinnyQR

I think the problem here is that IndexedRowMatrix.toRowMatrix does *not* result in a RowMatrix with rows in order of their indices, necessarily: // Drop its row indices. RowMatrix rowMat = indexedRowMatrix.toRowMatrix(); What you get is a matrix where the rows are arranged in whatever order they

Re: TallSkinnyQR

So b = 0.89 0.42 0.0 0.88 0.97 The solution at the bottom is the solution to Ax = b solved using Gaussian elimination. I guess another question is, is there another way to solve this problem? I'm trying to solve the least squares fit with a huge A (5MM x 1MM) x =

Re: TallSkinnyQR

Hi Sean, Here you go: sparsematrix.txt = row, col ,val 0,0,.42 0,1,.28 0,2,.89 1,0,.83 1,1,.34 1,2,.42 2,0,.23 3,0,.42 3,1,.98 3,2,.88 4,0,.23 4,1,.36 4,2,.97 The vector is just the third column of the matrix which should give the trivial solution of [0,0,1] This translates to this which is

Re: TallSkinnyQR

Rather than post a large section of code, please post a small example of the input matrix and its decomposition, to illustrate what you're saying is out of order. On Tue, Nov 8, 2016 at 3:50 AM im281 wrote: > I am getting the correct rows but they are out of order. Is

TallSkinnyQR

e.i(), e.j(), e.value()); //} //}); //entries.saveAsTextFile("Data/output1"); //output.saveAsTextFile("Data/output1"); //ata.toCoordinateMatrix().entries().toJavaRDD().