Hu Ziqian created SPARK-42744:
---------------------------------

             Summary: delete uploaded file when job finish
                 Key: SPARK-42744
                 URL: https://issues.apache.org/jira/browse/SPARK-42744
             Project: Spark
          Issue Type: Improvement
          Components: Kubernetes
    Affects Versions: 3.1.2
            Reporter: Hu Ziqian


On kubernetes, spark use spark.kubernetes.file.upload.path to upload local 
files to 

Hadoop compatible file system. But spark do not delete those files at all.

In this issue, we add a configuration 
spark.kubernetes.uploaded.file.delete.on.termination to delete those files by 
driver when job finishs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

Reply via email to