Jackey Lee created SPARK-31438:
----------------------------------
Summary: Support JobCleaned Status in SparkListener
Key: SPARK-31438
URL: https://issues.apache.org/jira/browse/SPARK-31438
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.1.0
Reporter: Jackey Lee
In Spark, we need do some hook, such as hive external temporary paths cleaning,
after job cleaned, which is discussed in SPARK-31346 and [GitHub Pull Request
#28129|https://github.com/apache/spark/pull/28129].
The JobEnd Status is not suitable for this. As JobEnd is responsible for Job
finished, once all result has generated, it should be finished. After finish,
Scheduler will leave the still running tasks to be zombie tasks and delete
abnormal tasks asynchronously.
Thus, we add JobCleaned Status to enable user to do some hook after all tasks
cleaned in Job. The JobCleaned Status can get from TaskSetManagers, which is
related to a stage, and once all stages of the job has been cleaned, then the
job is cleaned.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
