Hi, On the Spark Configuration page ( http://spark.apache.org/docs/1.5.2/configuration.html), the documentation for spark.shuffle.memoryFraction mentions that the fraction is taken from the Java heap. However, the documentation for spark.shuffle.io.preferDirectBufs implies that off-heap memory might be used instead during shuffles. Do these two explanation conflict each other?
In a related question, with Tungsten enabled, when does Spark use off-heap memory? Thanks, Jianneng -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Questions-about-Spark-Shuffle-and-Heap-tp25564.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
