Hi, Thanks for your reply.
I basically want to check if my understanding what parallelize() on RDDs is correct. In my case, I create a vertex RDD and edge RDD and distribute them by calling parallelize(). Now does Spark perform any operation on these RDDs in parallel? For example, if I apply groupBy on the edge RDD (grouping by source vertex) and call a function F on the grouped RDD, will F be applied on each group in parallel and will Spark determine how to do this in parallel regardless of the number of groups? Thanks. On Tue, Feb 17, 2015 at 5:03 PM, Yifan LI <iamyifa...@gmail.com> wrote: > Hi Kannan, > > I am not sure I have understood what your question is exactly, but maybe > the reduceByKey or reduceByKeyLocally functionality is better to your need. > > > > Best, > Yifan LI > > > > > > On 17 Feb 2015, at 17:37, Vijayasarathy Kannan <kvi...@vt.edu> wrote: > > Hi, > > I am working on a Spark application that processes graphs and I am trying > to do the following. > > - group the vertices (key - vertex, value - set of its outgoing edges) > - distribute each key to separate processes and process them (like mapper) > - reduce the results back at the main process > > Does the "groupBy" functionality do the distribution by default? > Do we have to explicitly use RDDs to enable automatic distribution? > > It'd be great if you could help me understand these and how to go about > with the problem. > > Thanks. > > >
