I don't need to cache RDDs in my spark Application, but there is a big shuffle in the data processing. I can always find Shuffle spill (memory) and Shuffle spill (disk). I'm wondering if I can give more memory to shuffle to avoid spill to disk.
export SPARK_JAVA_OPTS='-Dspark.shuffle.memoryFraction=.7 -Dspark.storage.memoryFraction=.1 -Dspark.io.compression.codec=org.apach e.spark.io.SnappyCompressionCodec -Dspark.shuffle.compress=true' Using standalone with the following parameters in spark-submit --num-executors 3 --executor-memory 15G --executor-cores 6 --driver-memory 2G And it turns out that a real bad idea. The worker keeps hanging some time in reduceByKey step. Sometime I had to go to each data node, and kill the worker manually. If I remove SPARK_JAVA_OPTS, the application finished successfully. Will all RDDs be stored in spark.storage.memoryFraction? if so, why tab "executors" always show "0.0 B / 8.6 GB" for memory used? Also table "storage" shows nothing in Application Detail UI. Any idea how to set those parameters to avoid spill more data? I'm using Spark-1.0.0 in CDH 5.1.0.
