huangsheng created SPARK-52124:
----------------------------------
Summary: Actively Releasing Disk Space After Application
Completion in Spark Standalone Mode
Key: SPARK-52124
URL: https://issues.apache.org/jira/browse/SPARK-52124
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.5.5
Reporter: huangsheng
When submitting tasks using Spark in standalone mode, a folder is generated
under the {{work}} directory on each node every time a task is submitted. The
naming convention for these folders is, for example,
{{{}app-20250212191730-0249{}}}. These folders contain the resource files that
each node downloads from the master node when the task is submitted. Although
there is a scheduled cleanup mechanism ({{{}spark.worker.cleanup.enabled{}}}),
it is not immediate. {color:#FF0000}If a large number of tasks are submitted in
a short period of time, and each task depends on a significant amount of
external resources, the disk space can be quickly exhausted.{color}
Therefore, I suggest actively deleting the disk space occupied under the
{{work}} directory after each task is completed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
