Re: Matrix inversion

[Ted Dunning]

Mahout is considerably better at sparse operations and optimizations than
dense ones.

Beyond that, I would expect that you would do better with traditional math
libraries.

And, are you really trying to invert a matrix? The common maxim is that
this implies an error in your method because inversion is O(n^3) and often
ill-conditioned to boot.  Usually, an implicit form of inversion via a
decompositional representation is far better than a true inversion. For
large systems the situation is even more stark, numerical accuracy
limitations and noise in the original data make it impossible to do better
than an approximate AND implicit inverse such as a limited rank SVD.




On Thu, May 5, 2016 at 1:56 PM, Dmitriy Lyubimov <dlie...@gmail.com> wrote:

> BTW, Thibaut, in the paper you mention, MPI based implementation beats
> Spark at least 2 times on performance of the inversion. Kinda what i was
> saying -- and in this case it doesn't seem that algorithm is as highly
> interconnected as, e.g., naive blockwise multiplication.
>
> On Thu, May 5, 2016 at 1:50 PM, Dmitriy Lyubimov <dlie...@gmail.com>
> wrote:
>
> > The mantra i keep hearing is that if someone needs matrix inversion then
> > he/she must be doing something wrong. Not sure how true that is, but in
> all
> > cases i have encountered, people try to avoid matrix inversion one way or
> > another.
> >
> > Re: libraries: Mahout is more about apis now than any particular in-core
> > library. Unfortunately, mahout's in-memory operations are rooted in
> > single-threaded colt and are pretty slow at the moment. We are looking
> for
> > ways of doing in-memory operations faster and integrating something
> better
> > and native.
> >
> > However, the really limiting factor seems to be Spark programming model
> > and the effects it brings to interconnected I/O problems with high degree
> > of scattering. Cf. , for example, to performances you can get with MKL
> MPI
> > wrapper. If you are looking for performance of distributed algebra on
> CPUs,
> > there's very few things that can compete with MKL MPI wrapper.
> >
> > My personal opinion is that for as long as the problem fits in memory
> (and
> > most of them do nowadays), no algorithm on spark is going to beat Matlab
> in
> > matrix multiplication and such, all things being equal, no matter how
> many
> > cores spark cluster gets, on 1gbit networks. The same seems to be 10-fold
> > true when comparing to GPU based algorithms (case in point: BidMach).
> >
> > On Thu, May 5, 2016 at 12:45 PM, thibaut <thibaut.gensol...@gmail.com>
> > wrote:
> >
> >>
> >> My askings are:
> >> - Is it better for what we want to do to use Mahout, or Spark ?
> >>
> >
> > Mahout at this point is better for declarative prototyping as it contains
> > distributed optimizer and compact expression dsl.
> >
> > - I saw that you already have a distributed PCA. Do you have a really
> >> efficient matrix inversion algorithm in Mahout ?
> >>
> > PCA underpinnings are described in detail in the "AM:Beyond MapReduce"
> > book.
> >
> >> - How good is the linear algebra library in compare to Matlab for
> example
> >> ?
> >>
> > See my opinion above about algorithms on spark. Yes, i did some
> > benchmarking and digging around. Some things could be on-par, but
> > interconnected things are decidedly worse than single node Matlab (in
> terms
> > of speed).
> >
> >>
> >> Finally, our main concern for using Spark is about the linear algebra
> >> library that is used with Spark. And we were wondering how good is the
> >> Mahout one ?
> >
> > What do you mean specifically? Speed? As i said, the in-core speed is
> what
> > one can expect from java based implementation, but in-core speed factor
> > seems to be far overshadowed by I/O programming model issues in highly
> > interconnected problems once certain size of the problem is reached.
> >
> >>
> >>
> > Thanking you in advance,
> >>
> >> Best regards.
> >> Thibaut
> >
> >
> >
>

Re: Matrix inversion

[Dmitriy Lyubimov]

BTW, Thibaut, in the paper you mention, MPI based implementation beats
Spark at least 2 times on performance of the inversion. Kinda what i was
saying -- and in this case it doesn't seem that algorithm is as highly
interconnected as, e.g., naive blockwise multiplication.

On Thu, May 5, 2016 at 1:50 PM, Dmitriy Lyubimov <dlie...@gmail.com> wrote:

> The mantra i keep hearing is that if someone needs matrix inversion then
> he/she must be doing something wrong. Not sure how true that is, but in all
> cases i have encountered, people try to avoid matrix inversion one way or
> another.
>
> Re: libraries: Mahout is more about apis now than any particular in-core
> library. Unfortunately, mahout's in-memory operations are rooted in
> single-threaded colt and are pretty slow at the moment. We are looking for
> ways of doing in-memory operations faster and integrating something better
> and native.
>
> However, the really limiting factor seems to be Spark programming model
> and the effects it brings to interconnected I/O problems with high degree
> of scattering. Cf. , for example, to performances you can get with MKL MPI
> wrapper. If you are looking for performance of distributed algebra on CPUs,
> there's very few things that can compete with MKL MPI wrapper.
>
> My personal opinion is that for as long as the problem fits in memory (and
> most of them do nowadays), no algorithm on spark is going to beat Matlab in
> matrix multiplication and such, all things being equal, no matter how many
> cores spark cluster gets, on 1gbit networks. The same seems to be 10-fold
> true when comparing to GPU based algorithms (case in point: BidMach).
>
> On Thu, May 5, 2016 at 12:45 PM, thibaut <thibaut.gensol...@gmail.com>
> wrote:
>
>>
>> My askings are:
>> - Is it better for what we want to do to use Mahout, or Spark ?
>>
>
> Mahout at this point is better for declarative prototyping as it contains
> distributed optimizer and compact expression dsl.
>
> - I saw that you already have a distributed PCA. Do you have a really
>> efficient matrix inversion algorithm in Mahout ?
>>
> PCA underpinnings are described in detail in the "AM:Beyond MapReduce"
> book.
>
>> - How good is the linear algebra library in compare to Matlab for example
>> ?
>>
> See my opinion above about algorithms on spark. Yes, i did some
> benchmarking and digging around. Some things could be on-par, but
> interconnected things are decidedly worse than single node Matlab (in terms
> of speed).
>
>>
>> Finally, our main concern for using Spark is about the linear algebra
>> library that is used with Spark. And we were wondering how good is the
>> Mahout one ?
>
> What do you mean specifically? Speed? As i said, the in-core speed is what
> one can expect from java based implementation, but in-core speed factor
> seems to be far overshadowed by I/O programming model issues in highly
> interconnected problems once certain size of the problem is reached.
>
>>
>>
> Thanking you in advance,
>>
>> Best regards.
>> Thibaut
>
>
>

Re: Matrix inversion

[Dmitriy Lyubimov]

The mantra i keep hearing is that if someone needs matrix inversion then
he/she must be doing something wrong. Not sure how true that is, but in all
cases i have encountered, people try to avoid matrix inversion one way or
another.

Re: libraries: Mahout is more about apis now than any particular in-core
library. Unfortunately, mahout's in-memory operations are rooted in
single-threaded colt and are pretty slow at the moment. We are looking for
ways of doing in-memory operations faster and integrating something better
and native.

However, the really limiting factor seems to be Spark programming model and
the effects it brings to interconnected I/O problems with high degree of
scattering. Cf. , for example, to performances you can get with MKL MPI
wrapper. If you are looking for performance of distributed algebra on CPUs,
there's very few things that can compete with MKL MPI wrapper.

My personal opinion is that for as long as the problem fits in memory (and
most of them do nowadays), no algorithm on spark is going to beat Matlab in
matrix multiplication and such, all things being equal, no matter how many
cores spark cluster gets, on 1gbit networks. The same seems to be 10-fold
true when comparing to GPU based algorithms (case in point: BidMach).

On Thu, May 5, 2016 at 12:45 PM, thibaut <thibaut.gensol...@gmail.com>
wrote:

>
> My askings are:
> - Is it better for what we want to do to use Mahout, or Spark ?
>

Mahout at this point is better for declarative prototyping as it contains
distributed optimizer and compact expression dsl.

- I saw that you already have a distributed PCA. Do you have a really
> efficient matrix inversion algorithm in Mahout ?
>
PCA underpinnings are described in detail in the "AM:Beyond MapReduce"
book.

> - How good is the linear algebra library in compare to Matlab for example ?
>
See my opinion above about algorithms on spark. Yes, i did some
benchmarking and digging around. Some things could be on-par, but
interconnected things are decidedly worse than single node Matlab (in terms
of speed).

>
> Finally, our main concern for using Spark is about the linear algebra
> library that is used with Spark. And we were wondering how good is the
> Mahout one ?

What do you mean specifically? Speed? As i said, the in-core speed is what
one can expect from java based implementation, but in-core speed factor
seems to be far overshadowed by I/O programming model issues in highly
interconnected problems once certain size of the problem is reached.

>
>
Thanking you in advance,
>
> Best regards.
> Thibaut

Re: matrix inversion in plan ?

[go canal]

Unfortunately I do not know much details of these. The steps of these 
calculation is passed to me from a research team. I am helping them with coding 
part only. I myself is not good at math :-( 
btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I can get 
inverse of matrix from SVD right ? thanks, canal 


 On Monday, October 5, 2015 2:25 PM, Ted Dunning  
wrote:
   

 That isn't enough detail.

How do you mean to compute degrees of freedom?  WHy do you need the inverse
to do this?

Where did you get this algorithm?

Is this even appropriate at large scale?

Is this a stable computation?



On Sun, Oct 4, 2015 at 11:18 PM, go canal  wrote:

> I will be more than interested to extend to complex double, when the
> solver is ready for double data type.  thanks, canal
>
>
>      On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>
>
>  On Sun, Oct 4, 2015 at 10:32 PM, go canal 
> wrote:
>
> > in fact i need to support both double and complex double for either
> > distributed memory based or out-of-core.
>
>
> Ahh...
>
> Well Mahout doesn't support complex anything. So this isn't going to help
> you.
>
>
>


  

Re: matrix inversion in plan ?

[Andrew Musselman]

Go, if you'd like to put your research team in touch with the list we may
be able to help work through a good approach; let us know.

On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov  wrote:

> Mahout translation (approximation, since ssvd is reduced-rank, not the true
> thing):
>
> val (drmU, drmV, s) = dssvd(drmA, k = 100)
> val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
>
> Still, technically, it is a right inverse as in reality m is rarely the
> same as n. Also, k must be k<= drmA.nrow min drmA.ncol
>
>
> On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> andrew.mussel...@gmail.com
> > wrote:
>
> > Yeah, nice trick Ted; here's a how-to for the list:
> > http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
> >
> > On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning 
> wrote:
> >
> > > Yes. You can get the inverse from an SVD or emulate its effect.
> > >
> > > Can you share the actual mathematical specification for your problem?
> > >
> > > If you can't, then there is little we can do to help.
> > >
> > >
> > >
> > > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> > > wrote:
> > >
> > > > Unfortunately I do not know much details of these. The steps of these
> > > > calculation is passed to me from a research team. I am helping them
> > with
> > > > coding part only. I myself is not good at math :-(
> > > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so,
> I
> > > can
> > > > get inverse of matrix from SVD right ? thanks, canal
> > > >
> > > >
> > > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > > > ted.dunn...@gmail.com> wrote:
> > > >
> > > >
> > > >  That isn't enough detail.
> > > >
> > > > How do you mean to compute degrees of freedom?  WHy do you need the
> > > inverse
> > > > to do this?
> > > >
> > > > Where did you get this algorithm?
> > > >
> > > > Is this even appropriate at large scale?
> > > >
> > > > Is this a stable computation?
> > > >
> > > >
> > > >
> > > > On Sun, Oct 4, 2015 at 11:18 PM, go canal  >
> > > > wrote:
> > > >
> > > > > I will be more than interested to extend to complex double, when
> the
> > > > > solver is ready for double data type.  thanks, canal
> > > > >
> > > > >
> > > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > > > ted.dunn...@gmail.com> wrote:
> > > > >
> > > > >
> > > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal
>  > >
> > > > > wrote:
> > > > >
> > > > > > in fact i need to support both double and complex double for
> either
> > > > > > distributed memory based or out-of-core.
> > > > >
> > > > >
> > > > > Ahh...
> > > > >
> > > > > Well Mahout doesn't support complex anything. So this isn't going
> to
> > > help
> > > > > you.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: matrix inversion in plan ?

[Dmitriy Lyubimov]

or pseudoinverse really, i guess

On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov  wrote:

> Mahout translation (approximation, since ssvd is reduced-rank, not the
> true thing):
>
> val (drmU, drmV, s) = dssvd(drmA, k = 100)
> val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
>
> Still, technically, it is a right inverse as in reality m is rarely the
> same as n. Also, k must be k<= drmA.nrow min drmA.ncol
>
>
> On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> andrew.mussel...@gmail.com> wrote:
>
>> Yeah, nice trick Ted; here's a how-to for the list:
>> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
>>
>> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning 
>> wrote:
>>
>> > Yes. You can get the inverse from an SVD or emulate its effect.
>> >
>> > Can you share the actual mathematical specification for your problem?
>> >
>> > If you can't, then there is little we can do to help.
>> >
>> >
>> >
>> > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
>> > wrote:
>> >
>> > > Unfortunately I do not know much details of these. The steps of these
>> > > calculation is passed to me from a research team. I am helping them
>> with
>> > > coding part only. I myself is not good at math :-(
>> > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I
>> > can
>> > > get inverse of matrix from SVD right ? thanks, canal
>> > >
>> > >
>> > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
>> > > ted.dunn...@gmail.com> wrote:
>> > >
>> > >
>> > >  That isn't enough detail.
>> > >
>> > > How do you mean to compute degrees of freedom?  WHy do you need the
>> > inverse
>> > > to do this?
>> > >
>> > > Where did you get this algorithm?
>> > >
>> > > Is this even appropriate at large scale?
>> > >
>> > > Is this a stable computation?
>> > >
>> > >
>> > >
>> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal 
>> > > wrote:
>> > >
>> > > > I will be more than interested to extend to complex double, when the
>> > > > solver is ready for double data type.  thanks, canal
>> > > >
>> > > >
>> > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
>> > > > ted.dunn...@gmail.com> wrote:
>> > > >
>> > > >
>> > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal
>> 
>> > > > wrote:
>> > > >
>> > > > > in fact i need to support both double and complex double for
>> either
>> > > > > distributed memory based or out-of-core.
>> > > >
>> > > >
>> > > > Ahh...
>> > > >
>> > > > Well Mahout doesn't support complex anything. So this isn't going to
>> > help
>> > > > you.
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>>
>
>

Re: matrix inversion in plan ?

[Andrew Musselman]

Yeah, nice trick Ted; here's a how-to for the list:
http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf

On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning  wrote:

> Yes. You can get the inverse from an SVD or emulate its effect.
>
> Can you share the actual mathematical specification for your problem?
>
> If you can't, then there is little we can do to help.
>
>
>
> On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> wrote:
>
> > Unfortunately I do not know much details of these. The steps of these
> > calculation is passed to me from a research team. I am helping them with
> > coding part only. I myself is not good at math :-(
> > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I
> can
> > get inverse of matrix from SVD right ? thanks, canal
> >
> >
> >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > ted.dunn...@gmail.com> wrote:
> >
> >
> >  That isn't enough detail.
> >
> > How do you mean to compute degrees of freedom?  WHy do you need the
> inverse
> > to do this?
> >
> > Where did you get this algorithm?
> >
> > Is this even appropriate at large scale?
> >
> > Is this a stable computation?
> >
> >
> >
> > On Sun, Oct 4, 2015 at 11:18 PM, go canal 
> > wrote:
> >
> > > I will be more than interested to extend to complex double, when the
> > > solver is ready for double data type.  thanks, canal
> > >
> > >
> > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > ted.dunn...@gmail.com> wrote:
> > >
> > >
> > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal 
> > > wrote:
> > >
> > > > in fact i need to support both double and complex double for either
> > > > distributed memory based or out-of-core.
> > >
> > >
> > > Ahh...
> > >
> > > Well Mahout doesn't support complex anything. So this isn't going to
> help
> > > you.
> > >
> > >
> > >
> >
> >
> >
>

Re: matrix inversion in plan ?

[Andrew Musselman]

Totally an approximation; depends on why people are asking for the inverse
and whether it'd do.

On Thu, Oct 8, 2015 at 4:20 PM, Dmitriy Lyubimov  wrote:

> or pseudoinverse really, i guess
>
> On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov 
> wrote:
>
> > Mahout translation (approximation, since ssvd is reduced-rank, not the
> > true thing):
> >
> > val (drmU, drmV, s) = dssvd(drmA, k = 100)
> > val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
> >
> > Still, technically, it is a right inverse as in reality m is rarely the
> > same as n. Also, k must be k<= drmA.nrow min drmA.ncol
> >
> >
> > On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> > andrew.mussel...@gmail.com> wrote:
> >
> >> Yeah, nice trick Ted; here's a how-to for the list:
> >> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
> >>
> >> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning 
> >> wrote:
> >>
> >> > Yes. You can get the inverse from an SVD or emulate its effect.
> >> >
> >> > Can you share the actual mathematical specification for your problem?
> >> >
> >> > If you can't, then there is little we can do to help.
> >> >
> >> >
> >> >
> >> > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> >> > wrote:
> >> >
> >> > > Unfortunately I do not know much details of these. The steps of
> these
> >> > > calculation is passed to me from a research team. I am helping them
> >> with
> >> > > coding part only. I myself is not good at math :-(
> >> > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If
> so, I
> >> > can
> >> > > get inverse of matrix from SVD right ? thanks, canal
> >> > >
> >> > >
> >> > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> >> > > ted.dunn...@gmail.com> wrote:
> >> > >
> >> > >
> >> > >  That isn't enough detail.
> >> > >
> >> > > How do you mean to compute degrees of freedom?  WHy do you need the
> >> > inverse
> >> > > to do this?
> >> > >
> >> > > Where did you get this algorithm?
> >> > >
> >> > > Is this even appropriate at large scale?
> >> > >
> >> > > Is this a stable computation?
> >> > >
> >> > >
> >> > >
> >> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal  >
> >> > > wrote:
> >> > >
> >> > > > I will be more than interested to extend to complex double, when
> the
> >> > > > solver is ready for double data type.  thanks, canal
> >> > > >
> >> > > >
> >> > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> >> > > > ted.dunn...@gmail.com> wrote:
> >> > > >
> >> > > >
> >> > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal
> >> 
> >> > > > wrote:
> >> > > >
> >> > > > > in fact i need to support both double and complex double for
> >> either
> >> > > > > distributed memory based or out-of-core.
> >> > > >
> >> > > >
> >> > > > Ahh...
> >> > > >
> >> > > > Well Mahout doesn't support complex anything. So this isn't going
> to
> >> > help
> >> > > > you.
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: matrix inversion in plan ?

[Ted Dunning]

Yes. You can get the inverse from an SVD or emulate its effect.

Can you share the actual mathematical specification for your problem?

If you can't, then there is little we can do to help.



On Wed, Oct 7, 2015 at 11:35 PM, go canal  wrote:

> Unfortunately I do not know much details of these. The steps of these
> calculation is passed to me from a research team. I am helping them with
> coding part only. I myself is not good at math :-(
> btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I can
> get inverse of matrix from SVD right ? thanks, canal
>
>
>  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>
>
>  That isn't enough detail.
>
> How do you mean to compute degrees of freedom?  WHy do you need the inverse
> to do this?
>
> Where did you get this algorithm?
>
> Is this even appropriate at large scale?
>
> Is this a stable computation?
>
>
>
> On Sun, Oct 4, 2015 at 11:18 PM, go canal 
> wrote:
>
> > I will be more than interested to extend to complex double, when the
> > solver is ready for double data type.  thanks, canal
> >
> >
> >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > ted.dunn...@gmail.com> wrote:
> >
> >
> >  On Sun, Oct 4, 2015 at 10:32 PM, go canal 
> > wrote:
> >
> > > in fact i need to support both double and complex double for either
> > > distributed memory based or out-of-core.
> >
> >
> > Ahh...
> >
> > Well Mahout doesn't support complex anything. So this isn't going to help
> > you.
> >
> >
> >
>
>
>

Re: matrix inversion in plan ?

[Dmitriy Lyubimov]

Mahout translation (approximation, since ssvd is reduced-rank, not the true
thing):

val (drmU, drmV, s) = dssvd(drmA, k = 100)
val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t

Still, technically, it is a right inverse as in reality m is rarely the
same as n. Also, k must be k<= drmA.nrow min drmA.ncol


On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman  wrote:

> Yeah, nice trick Ted; here's a how-to for the list:
> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
>
> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning  wrote:
>
> > Yes. You can get the inverse from an SVD or emulate its effect.
> >
> > Can you share the actual mathematical specification for your problem?
> >
> > If you can't, then there is little we can do to help.
> >
> >
> >
> > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> > wrote:
> >
> > > Unfortunately I do not know much details of these. The steps of these
> > > calculation is passed to me from a research team. I am helping them
> with
> > > coding part only. I myself is not good at math :-(
> > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I
> > can
> > > get inverse of matrix from SVD right ? thanks, canal
> > >
> > >
> > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > > ted.dunn...@gmail.com> wrote:
> > >
> > >
> > >  That isn't enough detail.
> > >
> > > How do you mean to compute degrees of freedom?  WHy do you need the
> > inverse
> > > to do this?
> > >
> > > Where did you get this algorithm?
> > >
> > > Is this even appropriate at large scale?
> > >
> > > Is this a stable computation?
> > >
> > >
> > >
> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal 
> > > wrote:
> > >
> > > > I will be more than interested to extend to complex double, when the
> > > > solver is ready for double data type.  thanks, canal
> > > >
> > > >
> > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > > ted.dunn...@gmail.com> wrote:
> > > >
> > > >
> > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal  >
> > > > wrote:
> > > >
> > > > > in fact i need to support both double and complex double for either
> > > > > distributed memory based or out-of-core.
> > > >
> > > >
> > > > Ahh...
> > > >
> > > > Well Mahout doesn't support complex anything. So this isn't going to
> > help
> > > > you.
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>

Re: matrix inversion in plan ?

[Ted Dunning]

This sort of thing would definitely compute the inverse.

And it is definitely to be avoided.

How about you give some specifics so I can say what should be done?



On Sun, Oct 4, 2015 at 7:31 PM, go canal <goca...@yahoo.com.invalid> wrote:

> Thank you all, the solver is something like this, am I correct:
> Matrix m = 
> Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1,
> m.rowSize()));
>
> The problem I have is that the matrix is too big, I need distributed, or
> out-of-core solution.
>
>  thanks, canal
>
>
>  On Monday, October 5, 2015 6:25 AM, Peter Jaumann <
> peter.jauma...@gmail.com> wrote:
>
>
>  This should be done with a matrix solver indeed!!!
>
>
>
> On Oct 4, 2015 11:53 AM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
> >
> >
> > It is almost certain that starting with an inversion is a serious error.
> >
> > Are you sure you don't want a matrix solver instead?
> >
> > Sent from my iPhone
> >
> > > On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
> > >
> > > oh, it is so unfortunate that the first step of my project requires the
> inversion of a very large matrix. will have to revert back to scalapack or
> MR based solutions I guess.
> > >  thanks, canal
> > >
> > >
> > >On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
> > >
> > >
> > > I doubt seriously that Samsara will support matrix inversion per se.
> The
> > > problem is
> > >
> > > a) it densifies sparse matrices
> > >
> > > b) it is much more costly than solving a linear system
> > >
> > > Samsara is roughly memory based, but different back-ends will try to
> spill
> > > to disk if necessary.  It is likely that the resulting degradation in
> > > performance would be dramatic and thus unacceptable to most users.
> > >
> > >
> > >
> > >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
> wrote:
> > >>
> > >> HiI saw some distributed matrix functions included in Samsara now.
> > >> Wondering if we have a plan to support matrix inversion ?BTW, am I
> correct
> > >> that it is distributed memory based, not out-of-core ? thanks, canal
> > >
> > >
>
>
>
>

Re: matrix inversion in plan ?

[Ted Dunning]

On Sun, Oct 4, 2015 at 10:32 PM, go canal  wrote:

> in fact i need to support both double and complex double for either
> distributed memory based or out-of-core.


Ahh...

Well Mahout doesn't support complex anything. So this isn't going to help
you.

Re: matrix inversion in plan ?

[go canal]

I will be more than interested to extend to complex double, when the solver is 
ready for double data type.  thanks, canal 


 On Monday, October 5, 2015 2:02 PM, Ted Dunning  
wrote:
   

 On Sun, Oct 4, 2015 at 10:32 PM, go canal  wrote:

> in fact i need to support both double and complex double for either
> distributed memory based or out-of-core.


Ahh...

Well Mahout doesn't support complex anything. So this isn't going to help
you.


  

Re: matrix inversion in plan ?

[Ted Dunning]

That isn't enough detail.

How do you mean to compute degrees of freedom?  WHy do you need the inverse
to do this?

Where did you get this algorithm?

Is this even appropriate at large scale?

Is this a stable computation?



On Sun, Oct 4, 2015 at 11:18 PM, go canal  wrote:

> I will be more than interested to extend to complex double, when the
> solver is ready for double data type.  thanks, canal
>
>
>  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>
>
>  On Sun, Oct 4, 2015 at 10:32 PM, go canal 
> wrote:
>
> > in fact i need to support both double and complex double for either
> > distributed memory based or out-of-core.
>
>
> Ahh...
>
> Well Mahout doesn't support complex anything. So this isn't going to help
> you.
>
>
>

Re: matrix inversion in plan ?

[go canal]

Thank you all, the solver is something like this, am I correct:
Matrix m = 
Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, 
m.rowSize()));

The problem I have is that the matrix is too big, I need distributed, or 
out-of-core solution.

 thanks, canal 


 On Monday, October 5, 2015 6:25 AM, Peter Jaumann 
<peter.jauma...@gmail.com> wrote:
   

 This should be done with a matrix solver indeed!!!



On Oct 4, 2015 11:53 AM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
>
>
> It is almost certain that starting with an inversion is a serious error.
>
> Are you sure you don't want a matrix solver instead?
>
> Sent from my iPhone
>
> > On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
> >
> > oh, it is so unfortunate that the first step of my project requires the
inversion of a very large matrix. will have to revert back to scalapack or
MR based solutions I guess.
> >  thanks, canal
> >
> >
> >    On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
ted.dunn...@gmail.com> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se. The
> > problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
spill
> > to disk if necessary.  It is likely that the resulting degradation in
> > performance would be dramatic and thus unacceptable to most users.
> >
> >
> >
> >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
wrote:
> >>
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
correct
> >> that it is distributed memory based, not out-of-core ? thanks, canal
> >
> >


  

Re: matrix inversion in plan ?

[Allen McIntosh]

1) Is m sparse?
2) Once you have computed "inverse", what are you going to do with it?

On 10/04/2015 10:31 PM, go canal wrote:
> Thank you all, the solver is something like this, am I correct:
> Matrix m = 
> Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, 
> m.rowSize()));
> 
> The problem I have is that the matrix is too big, I need distributed, or 
> out-of-core solution.
> 
>  thanks, canal 
> 
> 
>  On Monday, October 5, 2015 6:25 AM, Peter Jaumann 
> <peter.jauma...@gmail.com> wrote:
>
> 
>  This should be done with a matrix solver indeed!!!
> 
> 
> 
> On Oct 4, 2015 11:53 AM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
>>
>>
>> It is almost certain that starting with an inversion is a serious error.
>>
>> Are you sure you don't want a matrix solver instead?
>>
>> Sent from my iPhone
>>
>>> On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
>>>
>>> oh, it is so unfortunate that the first step of my project requires the
> inversion of a very large matrix. will have to revert back to scalapack or
> MR based solutions I guess.
>>>   thanks, canal
>>>
>>>
>>> On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>>>
>>>
>>> I doubt seriously that Samsara will support matrix inversion per se. The
>>> problem is
>>>
>>> a) it densifies sparse matrices
>>>
>>> b) it is much more costly than solving a linear system
>>>
>>> Samsara is roughly memory based, but different back-ends will try to
> spill
>>> to disk if necessary.  It is likely that the resulting degradation in
>>> performance would be dramatic and thus unacceptable to most users.
>>>
>>>
>>>
>>>> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
> wrote:
>>>>
>>>> HiI saw some distributed matrix functions included in Samsara now.
>>>> Wondering if we have a plan to support matrix inversion ?BTW, am I
> correct
>>>> that it is distributed memory based, not out-of-core ? thanks, canal
>>>
>>>
> 
> 
>   
> 

Re: matrix inversion in plan ?

[Ted Dunning]

It is almost certain that starting with an inversion is a serious error. 

Are you sure you don't want a matrix solver instead?

Sent from my iPhone

> On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
> 
> oh, it is so unfortunate that the first step of my project requires the 
> inversion of a very large matrix. will have to revert back to scalapack or MR 
> based solutions I guess.
>  thanks, canal 
> 
> 
> On Saturday, October 3, 2015 11:31 PM, Ted Dunning 
> <ted.dunn...@gmail.com> wrote:
> 
> 
> I doubt seriously that Samsara will support matrix inversion per se. The
> problem is
> 
> a) it densifies sparse matrices
> 
> b) it is much more costly than solving a linear system
> 
> Samsara is roughly memory based, but different back-ends will try to spill
> to disk if necessary.  It is likely that the resulting degradation in
> performance would be dramatic and thus unacceptable to most users.
> 
> 
> 
>> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid> wrote:
>> 
>> HiI saw some distributed matrix functions included in Samsara now.
>> Wondering if we have a plan to support matrix inversion ?BTW, am I correct
>> that it is distributed memory based, not out-of-core ? thanks, canal
> 
> 

Re: matrix inversion in plan ?

[Peter Jaumann]

This should be done with a matrix solver indeed!!!



On Oct 4, 2015 11:53 AM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
>
>
> It is almost certain that starting with an inversion is a serious error.
>
> Are you sure you don't want a matrix solver instead?
>
> Sent from my iPhone
>
> > On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
> >
> > oh, it is so unfortunate that the first step of my project requires the
inversion of a very large matrix. will have to revert back to scalapack or
MR based solutions I guess.
> >  thanks, canal
> >
> >
> > On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
ted.dunn...@gmail.com> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se. The
> > problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
spill
> > to disk if necessary.  It is likely that the resulting degradation in
> > performance would be dramatic and thus unacceptable to most users.
> >
> >
> >
> >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
wrote:
> >>
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
correct
> >> that it is distributed memory based, not out-of-core ? thanks, canal
> >
> >

Re: matrix inversion in plan ?

[go canal]

both sparse and dense. 
there are a few steps for the whole calculation:. inverse of M. get degree of 
freedom. multiplication. addition
in fact i need to support both double and complex double for either distributed 
memory based or out-of-core.
I found one MR based solution for large matrix inversion 
https://github.com/JingenXiang/MatrixInversion and I have modified the code to 
support complex double. Execution seems ok but i do not understand the final 
output format. It seems that the columns are swapped.  thanks, canal 


 On Monday, October 5, 2015 12:26 PM, Allen McIntosh 
<amcint...@appcomsci.com> wrote:
   

 1) Is m sparse?
2) Once you have computed "inverse", what are you going to do with it?

On 10/04/2015 10:31 PM, go canal wrote:
> Thank you all, the solver is something like this, am I correct:
> Matrix m = 
> Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, 
> m.rowSize()));
> 
> The problem I have is that the matrix is too big, I need distributed, or 
> out-of-core solution.
> 
>  thanks, canal 
> 
> 
>      On Monday, October 5, 2015 6:25 AM, Peter Jaumann 
><peter.jauma...@gmail.com> wrote:
>    
> 
>  This should be done with a matrix solver indeed!!!
> 
> 
> 
> On Oct 4, 2015 11:53 AM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
>>
>>
>> It is almost certain that starting with an inversion is a serious error.
>>
>> Are you sure you don't want a matrix solver instead?
>>
>> Sent from my iPhone
>>
>>> On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
>>>
>>> oh, it is so unfortunate that the first step of my project requires the
> inversion of a very large matrix. will have to revert back to scalapack or
> MR based solutions I guess.
>>>  thanks, canal
>>>
>>>
>>>    On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>>>
>>>
>>> I doubt seriously that Samsara will support matrix inversion per se. The
>>> problem is
>>>
>>> a) it densifies sparse matrices
>>>
>>> b) it is much more costly than solving a linear system
>>>
>>> Samsara is roughly memory based, but different back-ends will try to
> spill
>>> to disk if necessary.  It is likely that the resulting degradation in
>>> performance would be dramatic and thus unacceptable to most users.
>>>
>>>
>>>
>>>> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
> wrote:
>>>>
>>>> HiI saw some distributed matrix functions included in Samsara now.
>>>> Wondering if we have a plan to support matrix inversion ?BTW, am I
> correct
>>>> that it is distributed memory based, not out-of-core ? thanks, canal
>>>
>>>
> 
> 
>  
> 



  

Re: matrix inversion in plan ?

[Ted Dunning]

I doubt seriously that Samsara will support matrix inversion per se. The
problem is

a) it densifies sparse matrices

b) it is much more costly than solving a linear system

Samsara is roughly memory based, but different back-ends will try to spill
to disk if necessary.  It is likely that the resulting degradation in
performance would be dramatic and thus unacceptable to most users.



On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid> wrote:

> HiI saw some distributed matrix functions included in Samsara now.
> Wondering if we have a plan to support matrix inversion ?BTW, am I correct
> that it is distributed memory based, not out-of-core ? thanks, canal

Re: matrix inversion in plan ?

[go canal]

oh, it is so unfortunate that the first step of my project requires the 
inversion of a very large matrix. will have to revert back to scalapack or MR 
based solutions I guess.
 thanks, canal 


 On Saturday, October 3, 2015 11:31 PM, Ted Dunning <ted.dunn...@gmail.com> 
wrote:
   

 I doubt seriously that Samsara will support matrix inversion per se. The
problem is

a) it densifies sparse matrices

b) it is much more costly than solving a linear system

Samsara is roughly memory based, but different back-ends will try to spill
to disk if necessary.  It is likely that the resulting degradation in
performance would be dramatic and thus unacceptable to most users.



On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid> wrote:

> HiI saw some distributed matrix functions included in Samsara now.
> Wondering if we have a plan to support matrix inversion ?BTW, am I correct
> that it is distributed memory based, not out-of-core ? thanks, canal


  

Re: matrix inversion in plan ?

[Andrew Musselman]

If there's a need for inversion that's good info; would love to know the
purpose to get a sense of how people want to use the product.

On Saturday, October 3, 2015, Allen McIntosh <amcint...@appcomsci.com>
wrote:

> Can you explain why you feel you must invert a very large matrix.  This
> can be a Bad Idea.
>
> On 10/03/2015 08:09 PM, go canal wrote:
> > oh, it is so unfortunate that the first step of my project requires
> > the inversion of a very large matrix. will have to revert back to
> > scalapack or MR based solutions I guess. thanks, canal
> >
> >
> > On Saturday, October 3, 2015 11:31 PM, Ted Dunning
> > <ted.dunn...@gmail.com <javascript:;>> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se.
> > The problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
> > spill to disk if necessary.  It is likely that the resulting
> > degradation in performance would be dramatic and thus unacceptable to
> > most users.
> >
> >
> >
> > On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
> > wrote:
> >
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
> >> correct that it is distributed memory based, not out-of-core ?
> >> thanks, canal
> >
> >
> >
> >
>
>

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Koobas]

This is a standard problem in dense linear algebra.
The most established package to solve this problem is LAPACK.
There are newer packages, but this is a good reference point.
You first factor the matrix, DGETRF for a general matrix,
DSYTRF for a symmetric matrix, DPOTRF for a symmetric positive definite
matrix.
Then you call one of the functions: DGETRI, DSYTRI, DPOTRI, to do the rest.
DPOTRI is just a call to DTRTRI (inverse of the triangular factor),
and DLAUUM (triangular matrix multiply).

If you are on a multicore, MKL has that routine decently optimized.
If you are adventurous, you can try an academic package, such as PLASMA.
(May beat MKL on matrix inversion, but still requires MKL for BLAS.)
If you're going for the kill (performance-wise), you can try GPU
acceleration with MAGMA.
My guess is that MAGMA will invert a 20K x 20K matrix for you in the matter
of seconds.


On Mon, Jan 21, 2013 at 1:12 AM, Colin Wang colin.bin.wang.mah...@gmail.com
 wrote:

 Hi Koobas,

 I am trying on dense matrix in Hadoop, thousand times thousand square size.
 How do HPC guys to solve this problem? Any references?

 Thank you,
 Colin

 On Mon, Jan 21, 2013 at 11:49 AM, Koobas koo...@gmail.com wrote:

  Colin,
  I am more of an HPC guys.
  I am a Mahout noob myself.
  Are we talking about a dense matrix?
  What size?
 
 
  On Sun, Jan 20, 2013 at 9:34 PM, Colin Wang 
  colin.bin.wang.mah...@gmail.com
   wrote:
 
   Hi Koobas,
  
   I want the first one. Do you have any suggestions?
  
   Thank you,
   Colin
   On Fri, Jan 18, 2013 at 12:49 PM, Koobas koo...@gmail.com wrote:
  
Martix inversion
  
 

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Ted Dunning]

This is a relatively small matrix.  You can decompose a matrix like that,
at least approximately in a pretty short time.  With stock R (has no BLAS
acceleration), it takes 9ms to decompose a 100 x 100 dense matrix and 6.2
seconds to decompose a 1K x 1K dense matrix.  Since this is so fast, there
is little point in using Hadoop for this.

HPC guys pretty much never invert a matrix larger than 10x10 unless it is
diagonal or other special form.

Even if you think you need an inverse, usually all you need to do is
multiply by the inverse which is better done using a decomposed form
instead of a real inverse.  Matrix libraries will often cheat a bit and
substitute a matrix object that is really a decomposition when you ask for
an inverse.  Unless you look at the elements of the matrix, they can leave
it at that and you get speed and a warm feeling that you have an inverse
matrix.

One of the major reasons for avoiding the inverse at the size that you are
talking about is that using the inverse directly will probably cause you to
have a large loss of precision.

On Mon, Jan 21, 2013 at 1:12 AM, Colin Wang colin.bin.wang.mah...@gmail.com
 wrote:

 Hi Koobas,

 I am trying on dense matrix in Hadoop, thousand times thousand square size.
 How do HPC guys to solve this problem? Any references?

 Thank you,
 Colin

 On Mon, Jan 21, 2013 at 11:49 AM, Koobas koo...@gmail.com wrote:

  Colin,
  I am more of an HPC guys.
  I am a Mahout noob myself.
  Are we talking about a dense matrix?
  What size?
 
 
  On Sun, Jan 20, 2013 at 9:34 PM, Colin Wang 
  colin.bin.wang.mah...@gmail.com
   wrote:
 
   Hi Koobas,
  
   I want the first one. Do you have any suggestions?
  
   Thank you,
   Colin
   On Fri, Jan 18, 2013 at 12:49 PM, Koobas koo...@gmail.com wrote:
  
Martix inversion
  
 

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Colin Wang]

Hi Koobas,

I want the first one. Do you have any suggestions?

Thank you,
Colin
On Fri, Jan 18, 2013 at 12:49 PM, Koobas koo...@gmail.com wrote:

 Martix inversion

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Colin Wang]

Hi Sean,

I start to realize that the full inverse may not be realistic.
Matrix decomposition may be a better idea.
But I want to have a try and to show it as negative example in Map/Reduce.

Thank you,
Colin
On Fri, Jan 18, 2013 at 8:10 PM, Sean Owen sro...@gmail.com wrote:

 And, do you really need an inverse, or pseudo-inverse?
 But, no, there are really no direct utilities for this. But we could
 probably tell you how to do it efficiently, as long as you don't
 actually mean a full inverse.

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Colin Wang]

Hi Ted,

Thank you for the valuable experience.

Colin
On Fri, Jan 18, 2013 at 7:58 PM, Ted Dunning ted.dunn...@gmail.com wrote:

 h O(n^2) cannot be scaled to arbitrary size n.  Sparse systems with only
 k items on average per row can often be handled with o(n) com

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Koobas]

Colin,
I am more of an HPC guys.
I am a Mahout noob myself.
Are we talking about a dense matrix?
What size?


On Sun, Jan 20, 2013 at 9:34 PM, Colin Wang colin.bin.wang.mah...@gmail.com
 wrote:

 Hi Koobas,

 I want the first one. Do you have any suggestions?

 Thank you,
 Colin
 On Fri, Jan 18, 2013 at 12:49 PM, Koobas koo...@gmail.com wrote:

  Martix inversion

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Ted Dunning]

Left unsaid in this comment is the fact that matrix inversion of any
sizable matrix is almost always a mistake because it is (a) inaccurate, (b)
slow.

In scalable numerics it is also commonly true that the only really scalable
problems are sparse.  The reason for that is that systems whose cost grows
with O(n^2) cannot be scaled to arbitrary size n.  Sparse systems with only
k items on average per row can often be handled with o(n) complexity which
a requirement for a practical system.

On Thu, Jan 17, 2013 at 8:49 PM, Koobas koo...@gmail.com wrote:

 Martix inversion, as in explicitly computing the inverse,
 e.g. computing variance / covariance,
 or matrix inversion, as in solving a linear system of equations?


 On Thu, Jan 17, 2013 at 7:49 PM, Colin Wang 
 colin.bin.wang.mah...@gmail.com
  wrote:

  Hi All,
 
  I want to solve the matrix inversion, of course, big size, in Map/Reduce
  way.
  I don't know if Mahout offers this kind of utility. Could you please give
  me some tips?
 
  Thank you,
  Colin
 

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Sean Owen]

And, do you really need an inverse, or pseudo-inverse?
But, no, there are really no direct utilities for this. But we could
probably tell you how to do it efficiently, as long as you don't
actually mean a full inverse.

On Fri, Jan 18, 2013 at 11:58 AM, Ted Dunning ted.dunn...@gmail.com wrote:
 Left unsaid in this comment is the fact that matrix inversion of any
 sizable matrix is almost always a mistake because it is (a) inaccurate, (b)
 slow.

 In scalable numerics it is also commonly true that the only really scalable
 problems are sparse.  The reason for that is that systems whose cost grows
 with O(n^2) cannot be scaled to arbitrary size n.  Sparse systems with only
 k items on average per row can often be handled with o(n) complexity which
 a requirement for a practical system.

 On Thu, Jan 17, 2013 at 8:49 PM, Koobas koo...@gmail.com wrote:

 Martix inversion, as in explicitly computing the inverse,
 e.g. computing variance / covariance,
 or matrix inversion, as in solving a linear system of equations?


 On Thu, Jan 17, 2013 at 7:49 PM, Colin Wang 
 colin.bin.wang.mah...@gmail.com
  wrote:

  Hi All,
 
  I want to solve the matrix inversion, of course, big size, in Map/Reduce
  way.
  I don't know if Mahout offers this kind of utility. Could you please give
  me some tips?
 
  Thank you,
  Colin
 

Re: Any utility to solve the matrix inversion in Map/Reduce Way

[Koobas]

Martix inversion, as in explicitly computing the inverse,
e.g. computing variance / covariance,
or matrix inversion, as in solving a linear system of equations?


On Thu, Jan 17, 2013 at 7:49 PM, Colin Wang colin.bin.wang.mah...@gmail.com
 wrote:

 Hi All,

 I want to solve the matrix inversion, of course, big size, in Map/Reduce
 way.
 I don't know if Mahout offers this kind of utility. Could you please give
 me some tips?

 Thank you,
 Colin