Once you have generated the final RDD before submitting it to reducer try to repartition the RDD either using coalesce(partitions) or repartition() into known partitions. 2. Rule of thumb to create number of data partitions (3 * num_executors * cores_per_executor).
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-0-9-1-java-lang-outOfMemoryError-Java-Heap-Space-tp7861p7970.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
