Who is doing multiplication of large dense matrices using Hadoop? What is
a good way to do that computation using Hadoop?
Thanks,
Mike
I'm not sure, but I would suspect that Mahout has some low level map/reduce
jobs for this. You might start there.
On Fri, Nov 18, 2011 at 8:59 AM, Mike Spreitzer wrote:
> Who is doing multiplication of large dense matrices using Hadoop? What is
> a good way to do that computation using Hadoop
I wrote up a basic algorithm for this here:
http://math.columbia.edu/~tpeters/teh-codez/hadoop/hadoop-matrix-mult.html
It's almost certainly not optimal, but might get you some ideas.
Here is another approach http://www.norstad.org/matrix-multiply/index.html
Cheers,
Tom
On Fri, Nov 18, 2011 at
Is Hadoop the best tool for doing large matrix math.
Sure you can do it, but, aren't there better tools for these types of problems?
Sent from a remote device. Please excuse any typos...
Mike Segel
On Nov 18, 2011, at 10:59 AM, Mike Spreitzer wrote:
> Who is doing multiplication of large den
That's also an interesting question, but right now I am studying Hadoop
and want to know how well dense MM can be done in Hadoop.
Thanks,
Mike
From: Michel Segel
To: "common-user@hadoop.apache.org"
Date: 11/18/2011 12:34 PM
Subject: Re: Matrix multiplication
t; Subject: Re: Matrix multiplication in Hadoop
> From: mspre...@us.ibm.com
> Date: Fri, 18 Nov 2011 12:39:00 -0500
>
> That's also an interesting question, but right now I am studying Hadoop
> and want to know how well dense MM can be done in Hadoop.
>
> Thanks,
>
ut my Blog for answers to commonly asked questions.
From: Michel Segel
To: "common-user@hadoop.apache.org"
Sent: Friday, November 18, 2011 9:33 AM
Subject: Re: Matrix multiplication in Hadoop
Is Hadoop the best tool for doing large matrix math.
Sur
A problem with matrix multiplication in hadoop is that hadoop is row
oriented for the most part. I have thought about this use case however and
you can theoretically turn a 2D matrix into a 1D matrix and then that fits
into the row oriented nature of hadoop. Also being that the typical mapper
can
: Matrix multiplication in Hadoop
Ok Mike,
First I admire that you are studying Hadoop.
To answer your question... not well.
Might I suggest that if you want to learn Hadoop, you try and find a
problem which can easily be broken in to a series of parallel tasks where
there is minimal
On Friday, November 18, 2011, Mike Spreitzer wrote:
> Why is matrix multiplication ill-suited for Hadoop?
IMHO, a huge issue here is the JVM's inability to fully support cpu vendor
specific SIMD instructions and, by extension, optimized BLAS routines.
Running a large MM task using intel's MKL ra
Perhaps this is a good candidate for a native library, then?
From: Mike Davis [xmikeda...@gmail.com]
Sent: Friday, November 18, 2011 7:39 PM
To: common-user@hadoop.apache.org
Subject: Re: Matrix multiplication in Hadoop
On Friday, November 18, 2011, Mike
ke Davis [xmikeda...@gmail.com]
> Sent: Friday, November 18, 2011 7:39 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Matrix multiplication in Hadoop
>
> On Friday, November 18, 2011, Mike Spreitzer wrote:
>> Why is matrix multiplication ill-suited for Hadoop?
&g
AM, Tim Broberg wrote:
> Perhaps this is a good candidate for a native library, then?
>
>
> From: Mike Davis [xmikeda...@gmail.com]
> Sent: Friday, November 18, 2011 7:39 PM
> To: common-user@hadoop.apache.org
> Subject: Re: Matrix mul
2011, Tim Broberg wrote:
> > Perhaps this is a good candidate for a native library, then?
> >
> >
> > From: Mike Davis [xmikeda...@gmail.com]
> > Sent: Friday, November 18, 2011 7:39 PM
> > To: common-user@hadoop.apache
[xmikeda...@gmail.com]
> > Sent: Friday, November 18, 2011 7:39 PM
> > To: common-user@hadoop.apache.org
> > Subject: Re: Matrix multiplication in Hadoop
> >
> > On Friday, November 18, 2011, Mike Spreitzer
> wrote:
> > > Why is matrix multiplication ill-suited fo
Perhaps this is a good candidate for a native library, then?
>>
>>
>> From: Mike Davis [xmikeda...@gmail.com]
>> Sent: Friday, November 18, 2011 7:39 PM
>> To: common-user@hadoop.apache.org
>> Subject: Re: Matrix multiplic
and then the matrix
multiplication takes place in there.
Regards
Bejoy K S
-Original Message-
From: Mike Spreitzer
Date: Fri, 18 Nov 2011 14:52:05
To:
Reply-To: common-user@hadoop.apache.org
Subject: RE: Matrix multiplication in Hadoop
Well, this mismatch may tell me something
user vector is treated as a single column matrix and then
> the matrix multiplication takes place in there.
>
> Regards
> Bejoy K S
>
> -Original Message-
> From: Mike Spreitzer
> Date: Fri, 18 Nov 2011 14:52:05
> To:
> Reply-To: common-user@hadoop.apache.org
&
gt; > the matrix multiplication takes place in there.
> >
> > Regards
> > Bejoy K S
> >
> > -Original Message-
> > From: Mike Spreitzer
> > Date: Fri, 18 Nov 2011 14:52:05
> > To:
> > Reply-To: common-user@hadoop.apache
I am looking at large dense matrix multiplication as an example problem
for a class of middleware. I am also interested in sparse matrices, but
am taking things one step at a time.
There is a paper in IEEE CloudCom '10 about Hama, including a matrix
multiplication technique. It is essentially
on but its intermediate data
> volume is no greater than the input.
>
> Thanks,
> Mike
>
> --
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Matrix-multiplication-in-Hadoop-tp3519089p3529843
21 matches
Mail list logo
