I don't need to cache RDDs in my spark Application, but there is a big
shuffle in the data processing. I can always find Shuffle spill (memory)
and Shuffle spill (disk). I'm wondering if I can give more memory to
shuffle to avoid spill to disk.

export SPARK_JAVA_OPTS='-Dspark.shuffle.memoryFraction=.7
-Dspark.storage.memoryFraction=.1 -Dspark.io.compression.codec=org.apach
e.spark.io.SnappyCompressionCodec -Dspark.shuffle.compress=true'

Using standalone with the following parameters in spark-submit
--num-executors 3 --executor-memory 15G --executor-cores 6 --driver-memory
2G

And it turns out that a real bad idea. The worker keeps hanging some time
in reduceByKey step. Sometime I had to go to each data node, and kill the
worker manually.

If I remove SPARK_JAVA_OPTS, the application finished successfully.

Will all RDDs be stored in spark.storage.memoryFraction? if so, why tab
"executors" always show "0.0 B / 8.6 GB" for memory used? Also table
"storage" shows nothing in Application Detail UI.

Any idea how to set those parameters to avoid spill more data?

I'm using Spark-1.0.0 in CDH 5.1.0.

Reply via email to