Re: matrix inversion in plan ?

[go canal]

Unfortunately I do not know much details of these. The steps of these 
calculation is passed to me from a research team. I am helping them with coding 
part only. I myself is not good at math :-( 
btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I can get 
inverse of matrix from SVD right ? thanks, canal 


 On Monday, October 5, 2015 2:25 PM, Ted Dunning  
wrote:
   

 That isn't enough detail.

How do you mean to compute degrees of freedom?  WHy do you need the inverse
to do this?

Where did you get this algorithm?

Is this even appropriate at large scale?

Is this a stable computation?



On Sun, Oct 4, 2015 at 11:18 PM, go canal  wrote:

> I will be more than interested to extend to complex double, when the
> solver is ready for double data type.  thanks, canal
>
>
>      On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>
>
>  On Sun, Oct 4, 2015 at 10:32 PM, go canal 
> wrote:
>
> > in fact i need to support both double and complex double for either
> > distributed memory based or out-of-core.
>
>
> Ahh...
>
> Well Mahout doesn't support complex anything. So this isn't going to help
> you.
>
>
>


  

Re: matrix inversion in plan ?

[Andrew Musselman]

Go, if you'd like to put your research team in touch with the list we may
be able to help work through a good approach; let us know.

On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov  wrote:

> Mahout translation (approximation, since ssvd is reduced-rank, not the true
> thing):
>
> val (drmU, drmV, s) = dssvd(drmA, k = 100)
> val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
>
> Still, technically, it is a right inverse as in reality m is rarely the
> same as n. Also, k must be k<= drmA.nrow min drmA.ncol
>
>
> On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> andrew.mussel...@gmail.com
> > wrote:
>
> > Yeah, nice trick Ted; here's a how-to for the list:
> > http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
> >
> > On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning 
> wrote:
> >
> > > Yes. You can get the inverse from an SVD or emulate its effect.
> > >
> > > Can you share the actual mathematical specification for your problem?
> > >
> > > If you can't, then there is little we can do to help.
> > >
> > >
> > >
> > > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> > > wrote:
> > >
> > > > Unfortunately I do not know much details of these. The steps of these
> > > > calculation is passed to me from a research team. I am helping them
> > with
> > > > coding part only. I myself is not good at math :-(
> > > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so,
> I
> > > can
> > > > get inverse of matrix from SVD right ? thanks, canal
> > > >
> > > >
> > > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > > > ted.dunn...@gmail.com> wrote:
> > > >
> > > >
> > > >  That isn't enough detail.
> > > >
> > > > How do you mean to compute degrees of freedom?  WHy do you need the
> > > inverse
> > > > to do this?
> > > >
> > > > Where did you get this algorithm?
> > > >
> > > > Is this even appropriate at large scale?
> > > >
> > > > Is this a stable computation?
> > > >
> > > >
> > > >
> > > > On Sun, Oct 4, 2015 at 11:18 PM, go canal  >
> > > > wrote:
> > > >
> > > > > I will be more than interested to extend to complex double, when
> the
> > > > > solver is ready for double data type.  thanks, canal
> > > > >
> > > > >
> > > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > > > ted.dunn...@gmail.com> wrote:
> > > > >
> > > > >
> > > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal
>  > >
> > > > > wrote:
> > > > >
> > > > > > in fact i need to support both double and complex double for
> either
> > > > > > distributed memory based or out-of-core.
> > > > >
> > > > >
> > > > > Ahh...
> > > > >
> > > > > Well Mahout doesn't support complex anything. So this isn't going
> to
> > > help
> > > > > you.
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > >
> >
>

Re: matrix inversion in plan ?

[Dmitriy Lyubimov]

or pseudoinverse really, i guess

On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov  wrote:

> Mahout translation (approximation, since ssvd is reduced-rank, not the
> true thing):
>
> val (drmU, drmV, s) = dssvd(drmA, k = 100)
> val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
>
> Still, technically, it is a right inverse as in reality m is rarely the
> same as n. Also, k must be k<= drmA.nrow min drmA.ncol
>
>
> On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> andrew.mussel...@gmail.com> wrote:
>
>> Yeah, nice trick Ted; here's a how-to for the list:
>> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
>>
>> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning 
>> wrote:
>>
>> > Yes. You can get the inverse from an SVD or emulate its effect.
>> >
>> > Can you share the actual mathematical specification for your problem?
>> >
>> > If you can't, then there is little we can do to help.
>> >
>> >
>> >
>> > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
>> > wrote:
>> >
>> > > Unfortunately I do not know much details of these. The steps of these
>> > > calculation is passed to me from a research team. I am helping them
>> with
>> > > coding part only. I myself is not good at math :-(
>> > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I
>> > can
>> > > get inverse of matrix from SVD right ? thanks, canal
>> > >
>> > >
>> > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
>> > > ted.dunn...@gmail.com> wrote:
>> > >
>> > >
>> > >  That isn't enough detail.
>> > >
>> > > How do you mean to compute degrees of freedom?  WHy do you need the
>> > inverse
>> > > to do this?
>> > >
>> > > Where did you get this algorithm?
>> > >
>> > > Is this even appropriate at large scale?
>> > >
>> > > Is this a stable computation?
>> > >
>> > >
>> > >
>> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal 
>> > > wrote:
>> > >
>> > > > I will be more than interested to extend to complex double, when the
>> > > > solver is ready for double data type.  thanks, canal
>> > > >
>> > > >
>> > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
>> > > > ted.dunn...@gmail.com> wrote:
>> > > >
>> > > >
>> > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal
>> 
>> > > > wrote:
>> > > >
>> > > > > in fact i need to support both double and complex double for
>> either
>> > > > > distributed memory based or out-of-core.
>> > > >
>> > > >
>> > > > Ahh...
>> > > >
>> > > > Well Mahout doesn't support complex anything. So this isn't going to
>> > help
>> > > > you.
>> > > >
>> > > >
>> > > >
>> > >
>> > >
>> > >
>> >
>>
>
>

Re: matrix inversion in plan ?

[Andrew Musselman]

Yeah, nice trick Ted; here's a how-to for the list:
http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf

On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning  wrote:

> Yes. You can get the inverse from an SVD or emulate its effect.
>
> Can you share the actual mathematical specification for your problem?
>
> If you can't, then there is little we can do to help.
>
>
>
> On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> wrote:
>
> > Unfortunately I do not know much details of these. The steps of these
> > calculation is passed to me from a research team. I am helping them with
> > coding part only. I myself is not good at math :-(
> > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I
> can
> > get inverse of matrix from SVD right ? thanks, canal
> >
> >
> >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > ted.dunn...@gmail.com> wrote:
> >
> >
> >  That isn't enough detail.
> >
> > How do you mean to compute degrees of freedom?  WHy do you need the
> inverse
> > to do this?
> >
> > Where did you get this algorithm?
> >
> > Is this even appropriate at large scale?
> >
> > Is this a stable computation?
> >
> >
> >
> > On Sun, Oct 4, 2015 at 11:18 PM, go canal 
> > wrote:
> >
> > > I will be more than interested to extend to complex double, when the
> > > solver is ready for double data type.  thanks, canal
> > >
> > >
> > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > ted.dunn...@gmail.com> wrote:
> > >
> > >
> > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal 
> > > wrote:
> > >
> > > > in fact i need to support both double and complex double for either
> > > > distributed memory based or out-of-core.
> > >
> > >
> > > Ahh...
> > >
> > > Well Mahout doesn't support complex anything. So this isn't going to
> help
> > > you.
> > >
> > >
> > >
> >
> >
> >
>

Re: matrix inversion in plan ?

[Andrew Musselman]

Totally an approximation; depends on why people are asking for the inverse
and whether it'd do.

On Thu, Oct 8, 2015 at 4:20 PM, Dmitriy Lyubimov  wrote:

> or pseudoinverse really, i guess
>
> On Thu, Oct 8, 2015 at 3:58 PM, Dmitriy Lyubimov 
> wrote:
>
> > Mahout translation (approximation, since ssvd is reduced-rank, not the
> > true thing):
> >
> > val (drmU, drmV, s) = dssvd(drmA, k = 100)
> > val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t
> >
> > Still, technically, it is a right inverse as in reality m is rarely the
> > same as n. Also, k must be k<= drmA.nrow min drmA.ncol
> >
> >
> > On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman <
> > andrew.mussel...@gmail.com> wrote:
> >
> >> Yeah, nice trick Ted; here's a how-to for the list:
> >> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
> >>
> >> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning 
> >> wrote:
> >>
> >> > Yes. You can get the inverse from an SVD or emulate its effect.
> >> >
> >> > Can you share the actual mathematical specification for your problem?
> >> >
> >> > If you can't, then there is little we can do to help.
> >> >
> >> >
> >> >
> >> > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> >> > wrote:
> >> >
> >> > > Unfortunately I do not know much details of these. The steps of
> these
> >> > > calculation is passed to me from a research team. I am helping them
> >> with
> >> > > coding part only. I myself is not good at math :-(
> >> > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If
> so, I
> >> > can
> >> > > get inverse of matrix from SVD right ? thanks, canal
> >> > >
> >> > >
> >> > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> >> > > ted.dunn...@gmail.com> wrote:
> >> > >
> >> > >
> >> > >  That isn't enough detail.
> >> > >
> >> > > How do you mean to compute degrees of freedom?  WHy do you need the
> >> > inverse
> >> > > to do this?
> >> > >
> >> > > Where did you get this algorithm?
> >> > >
> >> > > Is this even appropriate at large scale?
> >> > >
> >> > > Is this a stable computation?
> >> > >
> >> > >
> >> > >
> >> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal  >
> >> > > wrote:
> >> > >
> >> > > > I will be more than interested to extend to complex double, when
> the
> >> > > > solver is ready for double data type.  thanks, canal
> >> > > >
> >> > > >
> >> > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> >> > > > ted.dunn...@gmail.com> wrote:
> >> > > >
> >> > > >
> >> > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal
> >> 
> >> > > > wrote:
> >> > > >
> >> > > > > in fact i need to support both double and complex double for
> >> either
> >> > > > > distributed memory based or out-of-core.
> >> > > >
> >> > > >
> >> > > > Ahh...
> >> > > >
> >> > > > Well Mahout doesn't support complex anything. So this isn't going
> to
> >> > help
> >> > > > you.
> >> > > >
> >> > > >
> >> > > >
> >> > >
> >> > >
> >> > >
> >> >
> >>
> >
> >
>

Re: matrix inversion in plan ?

[Ted Dunning]

Yes. You can get the inverse from an SVD or emulate its effect.

Can you share the actual mathematical specification for your problem?

If you can't, then there is little we can do to help.



On Wed, Oct 7, 2015 at 11:35 PM, go canal  wrote:

> Unfortunately I do not know much details of these. The steps of these
> calculation is passed to me from a research team. I am helping them with
> coding part only. I myself is not good at math :-(
> btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I can
> get inverse of matrix from SVD right ? thanks, canal
>
>
>  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>
>
>  That isn't enough detail.
>
> How do you mean to compute degrees of freedom?  WHy do you need the inverse
> to do this?
>
> Where did you get this algorithm?
>
> Is this even appropriate at large scale?
>
> Is this a stable computation?
>
>
>
> On Sun, Oct 4, 2015 at 11:18 PM, go canal 
> wrote:
>
> > I will be more than interested to extend to complex double, when the
> > solver is ready for double data type.  thanks, canal
> >
> >
> >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > ted.dunn...@gmail.com> wrote:
> >
> >
> >  On Sun, Oct 4, 2015 at 10:32 PM, go canal 
> > wrote:
> >
> > > in fact i need to support both double and complex double for either
> > > distributed memory based or out-of-core.
> >
> >
> > Ahh...
> >
> > Well Mahout doesn't support complex anything. So this isn't going to help
> > you.
> >
> >
> >
>
>
>

Re: matrix inversion in plan ?

[Dmitriy Lyubimov]

Mahout translation (approximation, since ssvd is reduced-rank, not the true
thing):

val (drmU, drmV, s) = dssvd(drmA, k = 100)
val drmInvA = drmV %*% diagv(1 /=: s) %*% drmU.t

Still, technically, it is a right inverse as in reality m is rarely the
same as n. Also, k must be k<= drmA.nrow min drmA.ncol


On Thu, Oct 8, 2015 at 2:52 PM, Andrew Musselman  wrote:

> Yeah, nice trick Ted; here's a how-to for the list:
> http://www.cse.unr.edu/~bebis/CS791E/Notes/SVD.pdf
>
> On Thu, Oct 8, 2015 at 2:31 PM, Ted Dunning  wrote:
>
> > Yes. You can get the inverse from an SVD or emulate its effect.
> >
> > Can you share the actual mathematical specification for your problem?
> >
> > If you can't, then there is little we can do to help.
> >
> >
> >
> > On Wed, Oct 7, 2015 at 11:35 PM, go canal 
> > wrote:
> >
> > > Unfortunately I do not know much details of these. The steps of these
> > > calculation is passed to me from a research team. I am helping them
> with
> > > coding part only. I myself is not good at math :-(
> > > btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I
> > can
> > > get inverse of matrix from SVD right ? thanks, canal
> > >
> > >
> > >  On Monday, October 5, 2015 2:25 PM, Ted Dunning <
> > > ted.dunn...@gmail.com> wrote:
> > >
> > >
> > >  That isn't enough detail.
> > >
> > > How do you mean to compute degrees of freedom?  WHy do you need the
> > inverse
> > > to do this?
> > >
> > > Where did you get this algorithm?
> > >
> > > Is this even appropriate at large scale?
> > >
> > > Is this a stable computation?
> > >
> > >
> > >
> > > On Sun, Oct 4, 2015 at 11:18 PM, go canal 
> > > wrote:
> > >
> > > > I will be more than interested to extend to complex double, when the
> > > > solver is ready for double data type.  thanks, canal
> > > >
> > > >
> > > >  On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> > > > ted.dunn...@gmail.com> wrote:
> > > >
> > > >
> > > >  On Sun, Oct 4, 2015 at 10:32 PM, go canal  >
> > > > wrote:
> > > >
> > > > > in fact i need to support both double and complex double for either
> > > > > distributed memory based or out-of-core.
> > > >
> > > >
> > > > Ahh...
> > > >
> > > > Well Mahout doesn't support complex anything. So this isn't going to
> > help
> > > > you.
> > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>

Re: Exception in thread "main" java.lang.IllegalArgumentException: Unable to read output from "mahout -spark classpath"

[Andrew Palumbo]

The Mahout 0.11.0 Shell requires Spark 1.3. Please try with Spark 1.3.1.

On 10/08/2015 10:37 PM, go canal wrote:
> I tried Spark 1.4.1, same error. Then I saw the same error from shell 
> command. So I suspect that it is the environment configuration problem.
> I have followed this https://mahout.apache.org/general/downloads.html for 
> Mahout configuration.
> So it seems to be a Spark configuration problem, I guess, although I can run 
> spark-example without errors. Will need to figure out what are missing.
>   thanks, canal
>
>
>   On Monday, October 5, 2015 12:23 AM, Pat Ferrel 
>  wrote:
> 
>
>   Mahout 0.11.0 is built on Spark 1.4 and so 1.5.1 is a bit unknown. I think 
> the Mahout Shell does not run on 1.5.1.
>
> That may not be the error below, which is caused when Mahout tries to create 
> a set of jars to use in the Spark executors. The code runs `mahout -spark 
> classpath` to get these. So something is missing in your env in Eclipse. Does 
> `mahout -spark classpath` run in a shell, if so check to see if you env 
> matches in Eclipse.
>
> Also what are you trying to do? I have some example Spark Context creation 
> code if you are using Mahout as a Library.
>
>
> On Oct 3, 2015, at 2:14 AM, go canal  wrote:
>
> Hello,I am running a very simple Mahout application in Eclipse, but got this 
> error:
> Exception in thread "main" java.lang.IllegalArgumentException: Unable to read 
> output from "mahout -spark classpath". Is SPARK_HOME defined?
> I have SPARK_HOME defined in Eclipse as an environment variable with value of 
> /usr/local/spark-1.5.1.
> What else I need to include/set ?
>
> thanks, canal
>
>
>

Re: Exception in thread "main" java.lang.IllegalArgumentException: Unable to read output from "mahout -spark classpath"

[go canal]

I tried Spark 1.4.1, same error. Then I saw the same error from shell command. 
So I suspect that it is the environment configuration problem.
I have followed this https://mahout.apache.org/general/downloads.html for 
Mahout configuration. 
So it seems to be a Spark configuration problem, I guess, although I can run 
spark-example without errors. Will need to figure out what are missing.
 thanks, canal 


 On Monday, October 5, 2015 12:23 AM, Pat Ferrel  
wrote:
   

 Mahout 0.11.0 is built on Spark 1.4 and so 1.5.1 is a bit unknown. I think the 
Mahout Shell does not run on 1.5.1.

That may not be the error below, which is caused when Mahout tries to create a 
set of jars to use in the Spark executors. The code runs `mahout -spark 
classpath` to get these. So something is missing in your env in Eclipse. Does 
`mahout -spark classpath` run in a shell, if so check to see if you env matches 
in Eclipse.

Also what are you trying to do? I have some example Spark Context creation code 
if you are using Mahout as a Library.


On Oct 3, 2015, at 2:14 AM, go canal  wrote:

Hello,I am running a very simple Mahout application in Eclipse, but got this 
error:
Exception in thread "main" java.lang.IllegalArgumentException: Unable to read 
output from "mahout -spark classpath". Is SPARK_HOME defined?
I have SPARK_HOME defined in Eclipse as an environment variable with value of 
/usr/local/spark-1.5.1.
What else I need to include/set ?

thanks, canal