I got rid of most of my heap errors by increasing the number of partitions of my RDDs by 8-16x. I found in the tuning page <http://spark.apache.org/docs/latest/tuning.html> that heap space errors can be caused by a hash table that's generated during the shuffle functions, so by splitting up how much is in each shuffle function with partitions, I was able to get rid of the errors. Thanks for putting me on the right path with the partitions.
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-java-lang-outOfMemoryError-Java-Heap-Space-tp7735p7843.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
