Using a RowMatrix inside a map

2015-01-14 Thread Alex Minnaar
I am working with a RowMatrix and I noticed in the multiply() method that the local matrix with which it is being multiplied is being distributed to all of the rows of the RowMatrix. If this is the case, then is it impossible to multiply a row matrix within a map operation? Because this would

Re: Using a RowMatrix inside a map

2015-01-14 Thread Xiangrui Meng
Yes, you can only use RowMatrix.multiply() within the driver. We are working on distributed block matrices and linear algebra operations on top of it, which would fit your use cases well. It may take several PRs to finish. You can find the first one here: https://github.com/apache/spark/pull/3200