Is the OOM happening to the Driver JVM or one of the Executor JVMs? What memory size is each JVM?
How large is the data you're trying to broadcast? If it's large enough, you may want to consider just persisting the data to distributed storage (like HDFS) and read it in through the normal read RDD methods like sc.textFile(). Maybe someone else can comment on what the largest recommended data collection sizes are to use with Broadcast... On Thu, Dec 11, 2014 at 10:14 AM, ll <duy.huynh....@gmail.com> wrote: > hi. i'm running into this OutOfMemory issue when i'm broadcasting a large > array. what is the best way to handle this? > > should i split the array into smaller arrays before broadcasting, and then > combining them locally at each node? > > thanks! > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/broadcast-OutOfMemoryError-tp20633.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
