Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/21390 Context for other reviewers: the issue addressed by this patch is actually a real issue in practice, especially for long-lived Spark clusters; I have seen this specific problem play a large contributing role to certain production out-of-disk-space failures. One thing I'd like to note: as implemented here, this patch only addresses this problem for Spark's built-in "Standalone" cluster manager. @jiangxb1987, could you mention that limitation in the PR title and description? My personal preference is to proceed incrementally by merging this Standalone-only PR and and deferring support for other cluster managers to future PRs (perhaps from experts familiar with those other cluster managers). I'll take a more detailed look tomorrow, but just wanted to provide motivation for other reviewers who might leave comments before then.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org