Re: Is Mahout obsolete now?

[go canal]

I am now curious to know if Spark or Mahout is Out-Of-Core or In-Core solution 
? Specifically, for matrix multiplication and factorization. thanks, canal 


 On Tuesday, October 20, 2015 6:37 AM, Dmitriy Lyubimov  
wrote:
   

 On Mon, Oct 19, 2015 at 3:29 PM, Pat Ferrel  wrote:

> Even have code running using the PredicitonIO framework. This includesa
> SDK to event store to realtime query. Loosely speaking a lambda
> architecture. Most of the whole enchilada running except the content part
> of the equation, which only works on metadata for how.
> https://github.com/pferrel/scala-parallel-universal-recommendation
>
> We even do custom versions at actionML.com
>

oh, nice!


  

Re: matrix inversion in plan ?

[go canal]

Unfortunately I do not know much details of these. The steps of these 
calculation is passed to me from a research team. I am helping them with coding 
part only. I myself is not good at math :-( 
btw,  I think Mahout supports out-of-core SVD, am I correct ? If so, I can get 
inverse of matrix from SVD right ? thanks, canal 


 On Monday, October 5, 2015 2:25 PM, Ted Dunning <ted.dunn...@gmail.com> 
wrote:
   

 That isn't enough detail.

How do you mean to compute degrees of freedom?  WHy do you need the inverse
to do this?

Where did you get this algorithm?

Is this even appropriate at large scale?

Is this a stable computation?



On Sun, Oct 4, 2015 at 11:18 PM, go canal <goca...@yahoo.com.invalid> wrote:

> I will be more than interested to extend to complex double, when the
> solver is ready for double data type.  thanks, canal
>
>
>      On Monday, October 5, 2015 2:02 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>
>
>  On Sun, Oct 4, 2015 at 10:32 PM, go canal <goca...@yahoo.com.invalid>
> wrote:
>
> > in fact i need to support both double and complex double for either
> > distributed memory based or out-of-core.
>
>
> Ahh...
>
> Well Mahout doesn't support complex anything. So this isn't going to help
> you.
>
>
>


  

Re: Exception in thread "main" java.lang.IllegalArgumentException: Unable to read output from "mahout -spark classpath"

[go canal]

I tried Spark 1.4.1, same error. Then I saw the same error from shell command. 
So I suspect that it is the environment configuration problem.
I have followed this https://mahout.apache.org/general/downloads.html for 
Mahout configuration. 
So it seems to be a Spark configuration problem, I guess, although I can run 
spark-example without errors. Will need to figure out what are missing.
 thanks, canal 


 On Monday, October 5, 2015 12:23 AM, Pat Ferrel <p...@occamsmachete.com> 
wrote:
   

 Mahout 0.11.0 is built on Spark 1.4 and so 1.5.1 is a bit unknown. I think the 
Mahout Shell does not run on 1.5.1.

That may not be the error below, which is caused when Mahout tries to create a 
set of jars to use in the Spark executors. The code runs `mahout -spark 
classpath` to get these. So something is missing in your env in Eclipse. Does 
`mahout -spark classpath` run in a shell, if so check to see if you env matches 
in Eclipse.

Also what are you trying to do? I have some example Spark Context creation code 
if you are using Mahout as a Library.


On Oct 3, 2015, at 2:14 AM, go canal <goca...@yahoo.com.INVALID> wrote:

Hello,I am running a very simple Mahout application in Eclipse, but got this 
error:
Exception in thread "main" java.lang.IllegalArgumentException: Unable to read 
output from "mahout -spark classpath". Is SPARK_HOME defined?
I have SPARK_HOME defined in Eclipse as an environment variable with value of 
/usr/local/spark-1.5.1.
What else I need to include/set ?

thanks, canal


  

Re: matrix inversion in plan ?

[go canal]

I will be more than interested to extend to complex double, when the solver is 
ready for double data type.  thanks, canal 


 On Monday, October 5, 2015 2:02 PM, Ted Dunning <ted.dunn...@gmail.com> 
wrote:
   

 On Sun, Oct 4, 2015 at 10:32 PM, go canal <goca...@yahoo.com.invalid> wrote:

> in fact i need to support both double and complex double for either
> distributed memory based or out-of-core.


Ahh...

Well Mahout doesn't support complex anything. So this isn't going to help
you.


  

Re: Exception in thread "main" java.lang.IllegalArgumentException: Unable to read output from "mahout -spark classpath"

[go canal]

Thank you very much for the help. I will try Spark 1.4. 
I would like to try distributed matrix multiplication. not sure if there are 
sample codes available. I am very new to this stack. thanks, canal 


 On Monday, October 5, 2015 12:23 AM, Pat Ferrel <p...@occamsmachete.com> 
wrote:
   

 Mahout 0.11.0 is built on Spark 1.4 and so 1.5.1 is a bit unknown. I think the 
Mahout Shell does not run on 1.5.1.

That may not be the error below, which is caused when Mahout tries to create a 
set of jars to use in the Spark executors. The code runs `mahout -spark 
classpath` to get these. So something is missing in your env in Eclipse. Does 
`mahout -spark classpath` run in a shell, if so check to see if you env matches 
in Eclipse.

Also what are you trying to do? I have some example Spark Context creation code 
if you are using Mahout as a Library.


On Oct 3, 2015, at 2:14 AM, go canal <goca...@yahoo.com.INVALID> wrote:

Hello,I am running a very simple Mahout application in Eclipse, but got this 
error:
Exception in thread "main" java.lang.IllegalArgumentException: Unable to read 
output from "mahout -spark classpath". Is SPARK_HOME defined?
I have SPARK_HOME defined in Eclipse as an environment variable with value of 
/usr/local/spark-1.5.1.
What else I need to include/set ?

thanks, canal


  

Re: matrix inversion in plan ?

[go canal]

Thank you all, the solver is something like this, am I correct:
Matrix m = 
Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, 
m.rowSize()));

The problem I have is that the matrix is too big, I need distributed, or 
out-of-core solution.

 thanks, canal 


 On Monday, October 5, 2015 6:25 AM, Peter Jaumann 
<peter.jauma...@gmail.com> wrote:
   

 This should be done with a matrix solver indeed!!!



On Oct 4, 2015 11:53 AM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
>
>
> It is almost certain that starting with an inversion is a serious error.
>
> Are you sure you don't want a matrix solver instead?
>
> Sent from my iPhone
>
> > On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
> >
> > oh, it is so unfortunate that the first step of my project requires the
inversion of a very large matrix. will have to revert back to scalapack or
MR based solutions I guess.
> >  thanks, canal
> >
> >
> >    On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
ted.dunn...@gmail.com> wrote:
> >
> >
> > I doubt seriously that Samsara will support matrix inversion per se. The
> > problem is
> >
> > a) it densifies sparse matrices
> >
> > b) it is much more costly than solving a linear system
> >
> > Samsara is roughly memory based, but different back-ends will try to
spill
> > to disk if necessary.  It is likely that the resulting degradation in
> > performance would be dramatic and thus unacceptable to most users.
> >
> >
> >
> >> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
wrote:
> >>
> >> HiI saw some distributed matrix functions included in Samsara now.
> >> Wondering if we have a plan to support matrix inversion ?BTW, am I
correct
> >> that it is distributed memory based, not out-of-core ? thanks, canal
> >
> >


  

Re: matrix inversion in plan ?

[go canal]

both sparse and dense. 
there are a few steps for the whole calculation:. inverse of M. get degree of 
freedom. multiplication. addition
in fact i need to support both double and complex double for either distributed 
memory based or out-of-core.
I found one MR based solution for large matrix inversion 
https://github.com/JingenXiang/MatrixInversion and I have modified the code to 
support complex double. Execution seems ok but i do not understand the final 
output format. It seems that the columns are swapped.  thanks, canal 


 On Monday, October 5, 2015 12:26 PM, Allen McIntosh 
<amcint...@appcomsci.com> wrote:
   

 1) Is m sparse?
2) Once you have computed "inverse", what are you going to do with it?

On 10/04/2015 10:31 PM, go canal wrote:
> Thank you all, the solver is something like this, am I correct:
> Matrix m = 
> Matrix inverse = new QRDecomposition(m).solve(new DiagonalMatrix(1, 
> m.rowSize()));
> 
> The problem I have is that the matrix is too big, I need distributed, or 
> out-of-core solution.
> 
>  thanks, canal 
> 
> 
>      On Monday, October 5, 2015 6:25 AM, Peter Jaumann 
><peter.jauma...@gmail.com> wrote:
>    
> 
>  This should be done with a matrix solver indeed!!!
> 
> 
> 
> On Oct 4, 2015 11:53 AM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:
>>
>>
>> It is almost certain that starting with an inversion is a serious error.
>>
>> Are you sure you don't want a matrix solver instead?
>>
>> Sent from my iPhone
>>
>>> On Oct 3, 2015, at 20:09, go canal <goca...@yahoo.com.INVALID> wrote:
>>>
>>> oh, it is so unfortunate that the first step of my project requires the
> inversion of a very large matrix. will have to revert back to scalapack or
> MR based solutions I guess.
>>>  thanks, canal
>>>
>>>
>>>    On Saturday, October 3, 2015 11:31 PM, Ted Dunning <
> ted.dunn...@gmail.com> wrote:
>>>
>>>
>>> I doubt seriously that Samsara will support matrix inversion per se. The
>>> problem is
>>>
>>> a) it densifies sparse matrices
>>>
>>> b) it is much more costly than solving a linear system
>>>
>>> Samsara is roughly memory based, but different back-ends will try to
> spill
>>> to disk if necessary.  It is likely that the resulting degradation in
>>> performance would be dramatic and thus unacceptable to most users.
>>>
>>>
>>>
>>>> On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid>
> wrote:
>>>>
>>>> HiI saw some distributed matrix functions included in Samsara now.
>>>> Wondering if we have a plan to support matrix inversion ?BTW, am I
> correct
>>>> that it is distributed memory based, not out-of-core ? thanks, canal
>>>
>>>
> 
> 
>  
> 



  

Re: matrix inversion in plan ?

[go canal]

oh, it is so unfortunate that the first step of my project requires the 
inversion of a very large matrix. will have to revert back to scalapack or MR 
based solutions I guess.
 thanks, canal 


 On Saturday, October 3, 2015 11:31 PM, Ted Dunning <ted.dunn...@gmail.com> 
wrote:
   

 I doubt seriously that Samsara will support matrix inversion per se. The
problem is

a) it densifies sparse matrices

b) it is much more costly than solving a linear system

Samsara is roughly memory based, but different back-ends will try to spill
to disk if necessary.  It is likely that the resulting degradation in
performance would be dramatic and thus unacceptable to most users.



On Fri, Oct 2, 2015 at 8:47 PM, go canal <goca...@yahoo.com.invalid> wrote:

> HiI saw some distributed matrix functions included in Samsara now.
> Wondering if we have a plan to support matrix inversion ?BTW, am I correct
> that it is distributed memory based, not out-of-core ? thanks, canal


  

Re: Matrix inverse

[go canal]

'in core' as in whole matrix loaded into memory ? I am looking for 
'distributed' matrix inverse because the matrix is very big (2 million x 2m), 
and it is a Complex Double matrix.
btw, which API is for the in-core inverse ? thanks,
canal 


 On Sunday, August 9, 2015 10:40 PM, Dmitriy Lyubimov dlie...@gmail.com 
wrote:
   

 Do you mean in core matrix inversion? It is supported via solve. Actually
it is supported both in Java and Scala.
On Aug 5, 2015 9:11 PM, go canal goca...@yahoo.com.invalid wrote:

 Hello,I am new to Mahout. Would appreciate if someone could tell me if
 matrix inverse is still supported in the latest release (0.10) ? I thought
 it was supported in the earlier release, for example, 0.3, in the class
  org.apache.mahout.math.matrix.linalq.Algebra ? thanks,
 canal


  

Re: Matrix inverse

[go canal]

I found one message posted before - is this the solution:
 Matrix inverse = new QRDecomposition(a).solve(new DiagonalMatrix(n)) thanks,
canal 


 On Tuesday, August 11, 2015 12:20 PM, go canal goca...@yahoo.com.INVALID 
wrote:
   

 'in core' as in whole matrix loaded into memory ? I am looking for 
'distributed' matrix inverse because the matrix is very big (2 million x 2m), 
and it is a Complex Double matrix.
btw, which API is for the in-core inverse ? thanks,
canal 


    On Sunday, August 9, 2015 10:40 PM, Dmitriy Lyubimov dlie...@gmail.com 
wrote:
  

 Do you mean in core matrix inversion? It is supported via solve. Actually
it is supported both in Java and Scala.
On Aug 5, 2015 9:11 PM, go canal goca...@yahoo.com.invalid wrote:

 Hello,I am new to Mahout. Would appreciate if someone could tell me if
 matrix inverse is still supported in the latest release (0.10) ? I thought
 it was supported in the earlier release, for example, 0.3, in the class
  org.apache.mahout.math.matrix.linalq.Algebra ? thanks,
 canal




  

Matrix inverse

[go canal]

Hello,I am new to Mahout. Would appreciate if someone could tell me if matrix 
inverse is still supported in the latest release (0.10) ? I thought it was 
supported in the earlier release, for example, 0.3, in the class  
org.apache.mahout.math.matrix.linalq.Algebra ? thanks,
canal