Re: Possible long lineage issue when using DStream to update a normal RDD

2015-05-07 Thread Chunnan Yao
Thank you for this suggestion! But may I ask what's the advantage to use checkpoint instead of cache here? Cuz they both cut lineage. I only know checkpoint saves RDD in disk, while cache in memory. So may be it's for reliability? Also on http://spark.apache.org/docs/latest/streaming-programming-g

Re: How can I implement eigenvalue decomposition in Spark?

2014-08-08 Thread Chunnan Yao
e biggest eigenvalue > s.toArray(0)*s.toArray(0)? > > xj @ Tokyo > > > On Fri, Aug 8, 2014 at 12:07 PM, Chunnan Yao wrote: > >> Hi there, what you've suggested are all meaningful. But to make myself >> clearer, my essential problems are: >> 1. My matrix is a

Re: How can I implement eigenvalue decomposition in Spark?

2014-08-07 Thread Chunnan Yao
Hi there, what you've suggested are all meaningful. But to make myself clearer, my essential problems are: 1. My matrix is asymmetric, and it is a probabilistic adjacency matrix, whose entries(a_ij) represents the likelihood that user j will broadcast the information generated by user i. Apparently