Re: Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

[Sguj]

I got rid of most of my heap errors by increasing the number of partitions of
my RDDs by 8-16x. I found in the  tuning page
   that heap space errors
can be caused by a hash table that's generated during the shuffle functions,
so by splitting up how much is in each shuffle function with partitions, I
was able to get rid of the errors. Thanks for putting me on the right path
with the partitions.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-java-lang-outOfMemoryError-Java-Heap-Space-tp7735p7843.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

[Sguj]

Am I trying to reduce it to the minimum number of partitions, or increase the
number of partitions with that change?



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-java-lang-outOfMemoryError-Java-Heap-Space-tp7735p7739.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Spark 1.0.0 java.lang.outOfMemoryError: Java Heap Space

[abhiguruvayya]

Try repartitioning the RDD using coalsce(int partitions) before performing
any transforms.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Spark-1-0-0-java-lang-outOfMemoryError-Java-Heap-Space-tp7735p7736.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.