I have a Spark application which is running Spark Streaming and Spark SQL. I observed the size of shuffle files under "spark.local.dir" folder keeps increase and never decreases. Eventually it will run out-of-disk-space error.
The question is: when will Spark delete these shuffle files? In the application, I'm use some operations like "updateStateByKey" and enabling checkpoint already. Thank you! --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
