I think the eigenvalues and eigenvectors you are talking about is that of
M^T*M or M*M^T, if we get M=U*s*V^T as SVD. What I want is to get
eigenvectors and eigenvalues of M itself. Is this my misunderstanding of
linear algebra or the API?
[image: M^{*} M = V \Sigma^{*} U^{*}\, U \Sigma V^{*} = V (\Sigma^{*}
\Sigma) V^{*}\,][image: M M^{*} = U \Sigma V^{*} \, V \Sigma^{*} U^{*} = U
(\Sigma \Sigma^{*}) U^{*}\,]
2014-08-08 11:19 GMT+08:00 x <wasedax...@gmail.com>:
> U.rows.toArray.take(1).foreach(println) and
> V.toArray.take(s.size).foreach(println)
> are not eigenvectors corresponding to the biggest eigenvalue
> s.toArray(0)*s.toArray(0)?
>
> xj @ Tokyo
>
>
> On Fri, Aug 8, 2014 at 12:07 PM, Chunnan Yao <yaochun...@gmail.com> wrote:
>
>> Hi there, what you've suggested are all meaningful. But to make myself
>> clearer, my essential problems are:
>> 1. My matrix is asymmetric, and it is a probabilistic adjacency matrix,
>> whose entries(a_ij) represents the likelihood that user j will broadcast
>> the information generated by user i. Apparently, a_ij and a_ji is
>> different, caus I love you doesn't necessarily mean you love me(What a sad
>> story~). All entries are real.
>> 2. I know I can get eigenvalues through SVD. My problem is I can't get
>> the corresponding eigenvectors, which requires solving equations, and I
>> also need eigenvectors in my calculation.In my simulation of this paper, I
>> only need the biggest eigenvalues and corresponding eigenvectors.
>> The paper posted by Shivaram Venkataraman is also concerned about
>> symmetric matrix. Could any one help me out?
>>
>>
>> 2014-08-08 9:41 GMT+08:00 x <wasedax...@gmail.com>:
>>
>> The SVD computed result already contains descending order of singular
>>> values, you can get the biggest eigenvalue.
>>>
>>> ---
>>>
>>> val svd = matrix.computeSVD(matrix.numCols().toInt, computeU = true)
>>> val U: RowMatrix = svd.U
>>> val s: Vector = svd.s
>>> val V: Matrix = svd.V
>>>
>>> U.rows.toArray.take(1).foreach(println)
>>>
>>> println(s.toArray(0)*s.toArray(0))
>>>
>>> println(V.toArray.take(s.size).foreach(println))
>>>
>>> ---
>>>
>>> xj @ Tokyo
>>>
>>>
>>> On Fri, Aug 8, 2014 at 3:06 AM, Shivaram Venkataraman <
>>> shiva...@eecs.berkeley.edu> wrote:
>>>
>>>> If you just want to find the top eigenvalue / eigenvector you can do
>>>> something like the Lanczos method. There is a description of a MapReduce
>>>> based algorithm in Section 4.2 of [1]
>>>>
>>>> [1] http://www.cs.cmu.edu/~ukang/papers/HeigenPAKDD2011.pdf
>>>>
>>>>
>>>> On Thu, Aug 7, 2014 at 10:54 AM, Li Pu <l...@twitter.com.invalid> wrote:
>>>>
>>>>> @Miles, the latest SVD implementation in mllib is partially
>>>>> distributed. Matrix-vector multiplication is computed among all workers,
>>>>> but the right singular vectors are all stored in the driver. If your
>>>>> symmetric matrix is n x n and you want the first k eigenvalues, you will
>>>>> need to fit n x k doubles in driver's memory. Behind the scene, it calls
>>>>> ARPACK to compute eigen-decomposition of A^T A. You can look into the
>>>>> source code for the details.
>>>>>
>>>>> @Sean, the SVD++ implementation in graphx is not the canonical
>>>>> definition of SVD. It doesn't have the orthogonality that SVD holds. But
>>>>> we
>>>>> might want to use graphx as the underlying matrix representation for
>>>>> mllib.SVD to address the problem of skewed entry distribution.
>>>>>
>>>>>
>>>>> On Thu, Aug 7, 2014 at 10:51 AM, Evan R. Sparks <evan.spa...@gmail.com
>>>>> > wrote:
>>>>>
>>>>>> Reza Zadeh has contributed the distributed implementation of
>>>>>> (Tall/Skinny) SVD (
>>>>>> http://spark.apache.org/docs/latest/mllib-dimensionality-reduction.html),
>>>>>> which is in MLlib (Spark 1.0) and a distributed sparse SVD coming in
>>>>>> Spark
>>>>>> 1.1. (https://issues.apache.org/jira/browse/SPARK-1782). If your
>>>>>> data is sparse (which it often is in social networks), you may have
>>>>>> better
>>>>>> luck with this.
>>>>>>
>>>>>> I haven't tried the GraphX implementation, but those algorithms are
>>>>>> often well-suited for power-law distributed graphs as you might see in
>>>>>> social networks.
>>>>>>
>>>>>> FWIW, I believe you need to square elements of the sigma matrix from
>>>>>> the SVD to get the eigenvalues.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Aug 7, 2014 at 10:20 AM, Sean Owen <so...@cloudera.com>
>>>>>> wrote:
>>>>>>
>>>>>>> (-incubator, +user)
>>>>>>>
>>>>>>> If your matrix is symmetric (and real I presume), and if my linear
>>>>>>> algebra isn't too rusty, then its SVD is its eigendecomposition. The
>>>>>>> SingularValueDecomposition object you get back has U and V, both of
>>>>>>> which have columns that are the eigenvectors.
>>>>>>>
>>>>>>> There are a few SVDs in the Spark code. The one in mllib is not
>>>>>>> distributed (right?) and is probably not an efficient means of
>>>>>>> computing eigenvectors if you really just want a decomposition of a
>>>>>>> symmetric matrix.
>>>>>>>
>>>>>>> The one I see in graphx is distributed? I haven't used it though.
>>>>>>> Maybe it could be part of a solution.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Aug 7, 2014 at 2:21 PM, yaochunnan <yaochun...@gmail.com>
>>>>>>> wrote:
>>>>>>> > Our lab need to do some simulation on online social networks. We
>>>>>>> need to
>>>>>>> > handle a 5000*5000 adjacency matrix, namely, to get its largest
>>>>>>> eigenvalue
>>>>>>> > and corresponding eigenvector. Matlab can be used but it is
>>>>>>> time-consuming.
>>>>>>> > Is Spark effective in linear algebra calculations and
>>>>>>> transformations? Later
>>>>>>> > we would have 5000000*5000000 matrix processed. It seems emergent
>>>>>>> that we
>>>>>>> > should find some distributed computation platform.
>>>>>>> >
>>>>>>> > I see SVD has been implemented and I can get eigenvalues of a
>>>>>>> matrix through
>>>>>>> > this API. But when I want to get both eigenvalues and
>>>>>>> eigenvectors or at
>>>>>>> > least the biggest eigenvalue and the corresponding eigenvector, it
>>>>>>> seems
>>>>>>> > that current Spark doesn't have such API. Is it possible that I
>>>>>>> write
>>>>>>> > eigenvalue decomposition from scratch? What should I do? Thanks a
>>>>>>> lot!
>>>>>>> >
>>>>>>> >
>>>>>>> > Miles Yao
>>>>>>> >
>>>>>>> > ________________________________
>>>>>>> > View this message in context: How can I implement eigenvalue
>>>>>>> decomposition
>>>>>>> > in Spark?
>>>>>>> > Sent from the Apache Spark User List mailing list archive at
>>>>>>> Nabble.com.
>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Li
>>>>> @vrilleup
>>>>>
>>>>
>>>>
>>>
>>
>
