Thanks for your reply. I think that the problem was that SparkR tried to serialize the whole environment. Mind that the large dataframe was part of it. So every worker received their slice / partition (which is very small) plus the whole thing!
So I deleted the large dataframe and list before parallelizing and the cluster ran without memory issues. Best, Carlos J. Gil Bellosta http://www.datanalytics.com 2014-08-15 3:53 GMT+02:00 Shivaram Venkataraman <shiva...@eecs.berkeley.edu>: > Could you try increasing the number of slices with the large data set ? > SparkR assumes that each slice (or partition in Spark terminology) can fit > in memory of a single machine. Also is the error happening when you do the > map function or does it happen when you combine the results ? > > Thanks > Shivaram > > > On Thu, Aug 14, 2014 at 3:53 PM, Carlos J. Gil Bellosta > <gilbello...@gmail.com> wrote: >> >> Hello, >> >> I am having problems trying to apply the split-apply-combine strategy >> for dataframes using SparkR. >> >> I have a largish dataframe and I would like to achieve something similar >> to what >> >> ddply(df, .(id), foo) >> >> would do, only that using SparkR as computing engine. My df has a few >> million records and I would like to split it by "id" and operate on >> the pieces. These pieces are quite small in size: just a few hundred >> records. >> >> I do something along the following lines: >> >> 1) Use split to transform df into a list of dfs. >> 2) parallelize the resulting list as a RDD (using a few thousand slices) >> 3) map my function on the pieces using Spark. >> 4) recombine the results (do.call, rbind, etc.) >> >> My cluster works and I can perform medium sized batch jobs. >> >> However, it fails with my full df: I get a heap space out of memory >> error. It is funny as the slices are very small in size. >> >> Should I send smaller batches to my cluster? Is there any recommended >> general approach to these kind of split-apply-combine problems? >> >> Best, >> >> Carlos J. Gil Bellosta >> http://www.datanalytics.com >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
