Re: How could I do this algorithm in Spark?

m my Verizon Wireless 4G LTE smartphone > > > Original message > From: Guillermo Ortiz <konstt2...@gmail.com> > Date: 02/24/2016 5:26 PM (GMT-05:00) > To: user <user@spark.apache.org> > Subject: How could I do this algorithm in Spark? > > I want to do some

RE: How could I do this algorithm in Spark?

be interesting. Sent from my Verizon Wireless 4G LTE smartphone Original message From: Guillermo Ortiz <konstt2...@gmail.com> Date: 02/24/2016 5:26 PM (GMT-05:00) To: user <user@spark.apache.org> Subject: How could I do this algorithm in Spark? I want to do some algori

Re: How could I do this algorithm in Spark?

Thank you!, I'm trying to do it with Pregel,, it's being hard because I have never used GraphX and Pregel before. 2016-02-25 14:00 GMT+01:00 Sabarish Sasidharan : > Like Robin said, pls explore Pregel. You could do it without Pregel but it > might be laborious. I have a

Re: How could I do this algorithm in Spark?

Like Robin said, pls explore Pregel. You could do it without Pregel but it might be laborious. I have a simple outline below. You will need more iterations if the number of levels is higher. a-b b-c c-d b-e e-f f-c flatmaptopair a -> (a-b) b -> (a-b) b -> (b-c) c -> (b-c) c -> (c-d) d -> (c-d)

Re: How could I do this algorithm in Spark?

I'm taking a look to Pregel. It seems it's a good way to do it. The only negative thing that I see it's not a really complex graph with a lot of edges between the vertex .. They are more like a lot of isolated small graphs 2016-02-25 12:32 GMT+01:00 Robin East : > The

Re: How could I do this algorithm in Spark?

Oh, the letters were just an example, it could be: a , t b, o t, k k, c So.. a -> t -> k -> c and the result is: a,c; t,c; k,c and b,o I don't know if you were thinking about sortBy because the another example where letter were consecutive. 2016-02-25 9:42 GMT+01:00 Guillermo Ortiz

Re: How could I do this algorithm in Spark?

I don't see that sorting the data helps. The answer has to be all the associations. In this case the answer has to be: a , b --> it was a error in the question, sorry. b , d c , d x , y y , y I feel like all the data which is associate should be in the same executor. On this case if I order the

Re: How could I do this algorithm in Spark?

Guillermo, I think you're after an associative algorithm where A is ultimately associated with D, correct? Jakob would correct if that is a typo--a sort would be all that is necessary in that case. I believe you're looking for something else though, if I understand correctly. This seems like a

Re: How could I do this algorithm in Spark?

Hi Guillermo, assuming that the first "a,b" is a typo and you actually meant "a,d", this is a sorting problem. You could easily model your data as an RDD or tuples (or as a dataframe/set) and use the sortBy (or orderBy for dataframe/sets) methods. best, --Jakob On Wed, Feb 24, 2016 at 2:26 PM,

How could I do this algorithm in Spark?

I want to do some algorithm in Spark.. I know how to do it in a single machine where all data are together, but I don't know a good way to do it in Spark. If someone has an idea.. I have some data like this a , b x , y b , c y , y c , d I want something like: a , d b , d c , d x , y y , y I