Re: Matrix multiplication in spark
We are working on a PRs to add block partitioned matrix formats and dense matrix multiply methods. This should be out in the next few weeks or so. The sparse methods still need some research on partitioning schemes etc. and we will do that after the dense methods are in place. Thanks Shivaram On Wed, Nov 5, 2014 at 2:00 AM, Duy Huynh wrote: > ok great. when will this be ready? > > On Wed, Nov 5, 2014 at 4:27 AM, Xiangrui Meng wrote: > >> We are working on distributed block matrices. The main JIRA is at: >> >> https://issues.apache.org/jira/browse/SPARK-3434 >> >> The goal is to support basic distributed linear algebra, (dense first >> and then sparse). >> >> -Xiangrui >> >> On Wed, Nov 5, 2014 at 12:23 AM, ll wrote: >> > @sowen.. i am looking for distributed operations, especially very large >> > sparse matrix x sparse matrix multiplication. what is the best way to >> > implement this in spark? >> > >> > >> > >> > -- >> > View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html >> > Sent from the Apache Spark User List mailing list archive at Nabble.com. >> > >> > - >> > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> > For additional commands, e-mail: user-h...@spark.apache.org >> > >> > >
Re: Matrix multiplication in spark
ok great. when will this be ready? On Wed, Nov 5, 2014 at 4:27 AM, Xiangrui Meng wrote: > We are working on distributed block matrices. The main JIRA is at: > > https://issues.apache.org/jira/browse/SPARK-3434 > > The goal is to support basic distributed linear algebra, (dense first > and then sparse). > > -Xiangrui > > On Wed, Nov 5, 2014 at 12:23 AM, ll wrote: > > @sowen.. i am looking for distributed operations, especially very large > > sparse matrix x sparse matrix multiplication. what is the best way to > > implement this in spark? > > > > > > > > -- > > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html > > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > > - > > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > > For additional commands, e-mail: user-h...@spark.apache.org > > >
Re: Matrix multiplication in spark
We are working on distributed block matrices. The main JIRA is at: https://issues.apache.org/jira/browse/SPARK-3434 The goal is to support basic distributed linear algebra, (dense first and then sparse). -Xiangrui On Wed, Nov 5, 2014 at 12:23 AM, ll wrote: > @sowen.. i am looking for distributed operations, especially very large > sparse matrix x sparse matrix multiplication. what is the best way to > implement this in spark? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Matrix multiplication in spark
@sowen.. i am looking for distributed operations, especially very large sparse matrix x sparse matrix multiplication. what is the best way to implement this in spark? -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Matrix-multiplication-in-spark-tp12562p18164.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
Re: Matrix multiplication in spark
Scala defines transpose.
On Thu, Aug 21, 2014 at 4:22 PM, x wrote:
> Yes.
> Now Spark API doesn't provide transpose function. You have to define it
> like below.
>
> def transpose(m: Array[Array[Double]]): Array[Array[Double]] = {
> (for {
> c <- m(0).indices
> } yield m.map(_(c)) ).toArray
> }
>
> xj @ Tokyo
>
>
> On Thu, Aug 21, 2014 at 10:12 PM, phoenix bai
> wrote:
>
>> this is exactly what I was looking for. thank you.
>>
>> one thing though, it doesn`t have transpose() function defined, so I have
>> to do the transpose myself for the localMat in your case.
>> hoping it will be supported in the future :-)
>>
>>
>>
>> On Thu, Aug 21, 2014 at 7:30 PM, x wrote:
>>
>>> You could create a distributed matrix with RowMatrix.
>>>
>>> val rmat = new RowMatrix(rows)
>>>
>>> And then make a local DenseMatrix.
>>>
>>> val localMat = Matrices.dense(m, n, mat)
>>>
>>> Then multiply them.
>>>
>>> rmat.multiply(localMat)
>>>
>>>
>>> xj @ Tokyo
>>>
>>> On Thu, Aug 21, 2014 at 6:37 PM, Sean Owen wrote:
>>>
Are you trying to multiply dense or sparse matrices? if sparse, are
they very large -- meaning, are you looking for distributed
operations?
On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai
wrote:
> there is RowMatrix implemented in spark.
> and I check for a while but failed to find any matrix operations (like
> multiplication etc) are defined in the class yet.
>
> so, my question is, if I want to do matrix multiplication, (to do
vector x
> matrix multiplication to be precise), need to convert the
vector/matrix to
> the the matrix type defined in breeze package right?
>
> thanks
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>
>
Re: Matrix multiplication in spark
Yes.
Now Spark API doesn't provide transpose function. You have to define it
like below.
def transpose(m: Array[Array[Double]]): Array[Array[Double]] = {
(for {
c <- m(0).indices
} yield m.map(_(c)) ).toArray
}
xj @ Tokyo
On Thu, Aug 21, 2014 at 10:12 PM, phoenix bai wrote:
> this is exactly what I was looking for. thank you.
>
> one thing though, it doesn`t have transpose() function defined, so I have
> to do the transpose myself for the localMat in your case.
> hoping it will be supported in the future :-)
>
>
>
> On Thu, Aug 21, 2014 at 7:30 PM, x wrote:
>
>> You could create a distributed matrix with RowMatrix.
>>
>> val rmat = new RowMatrix(rows)
>>
>> And then make a local DenseMatrix.
>>
>> val localMat = Matrices.dense(m, n, mat)
>>
>> Then multiply them.
>>
>> rmat.multiply(localMat)
>>
>>
>> xj @ Tokyo
>>
>> On Thu, Aug 21, 2014 at 6:37 PM, Sean Owen wrote:
>>
>>> Are you trying to multiply dense or sparse matrices? if sparse, are
>>> they very large -- meaning, are you looking for distributed
>>> operations?
>>>
>>> On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai
>>> wrote:
>>> > there is RowMatrix implemented in spark.
>>> > and I check for a while but failed to find any matrix operations (like
>>> > multiplication etc) are defined in the class yet.
>>> >
>>> > so, my question is, if I want to do matrix multiplication, (to do
>>> vector x
>>> > matrix multiplication to be precise), need to convert the
>>> vector/matrix to
>>> > the the matrix type defined in breeze package right?
>>> >
>>> > thanks
>>>
>>> -
>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>
>>>
>>
>
Re: Matrix multiplication in spark
You could create a distributed matrix with RowMatrix. val rmat = new RowMatrix(rows) And then make a local DenseMatrix. val localMat = Matrices.dense(m, n, mat) Then multiply them. rmat.multiply(localMat) xj @ Tokyo On Thu, Aug 21, 2014 at 6:37 PM, Sean Owen wrote: > Are you trying to multiply dense or sparse matrices? if sparse, are > they very large -- meaning, are you looking for distributed > operations? > > On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai > wrote: > > there is RowMatrix implemented in spark. > > and I check for a while but failed to find any matrix operations (like > > multiplication etc) are defined in the class yet. > > > > so, my question is, if I want to do matrix multiplication, (to do vector > x > > matrix multiplication to be precise), need to convert the vector/matrix > to > > the the matrix type defined in breeze package right? > > > > thanks > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Re: Matrix multiplication in spark
Are you trying to multiply dense or sparse matrices? if sparse, are they very large -- meaning, are you looking for distributed operations? On Thu, Aug 21, 2014 at 10:07 AM, phoenix bai wrote: > there is RowMatrix implemented in spark. > and I check for a while but failed to find any matrix operations (like > multiplication etc) are defined in the class yet. > > so, my question is, if I want to do matrix multiplication, (to do vector x > matrix multiplication to be precise), need to convert the vector/matrix to > the the matrix type defined in breeze package right? > > thanks - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
