Thanks Himanshu and RahulKumar!
The databricks forum post was extremely useful. It is great to see an
article that clearly details how and when shuffles are cleaned up.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage
'spark.shuffle.memoryFraction' to 0.4 which is by default 0.2
this should make a significant difference in disk use of shuffle.
Thank you
-
Himanshu Mehra
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage-tp23279p23334.html
Sent from the Apache Spark
(SigmoidAnalytics), India
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage-tp23279p23323.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
there for a good
reason.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage-tp23279.html
Sent from the Apache Spark User List mailing list archive at Nabble.com
in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Limit-Spark-Shuffle-Disk-Usage-tp23279.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr