[
https://issues.apache.org/jira/browse/SPARK-27696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Henry Yu updated SPARK-27696:
-----------------------------
Description:
When submit to k8s, driver pod not deleted after job completion.
While k8s checks driver pod name not existing, It is especially painful when we
use workflow tool to resubmit the failed spark job. (by the way, client always
exist with 0 is another painful issue)
I have fix this with a new config
spark.kubernetes.submission.deleteCompletedPod=true in our home maintained
spark version.
Do you guys have more insights or I can make a pr on this issue?
was:
when submit to k8s, driver pod not deleted after job completion.
While k8s checks driver pod name not existing, It is especially painful when we
use workflow tool to resubmit the failed spark job. (by the way, client always
exist with 0 is another painful issue)
I have fix this with a new config
spark.kubernetes.submission.deleteCompletedPod=true in our home maintained
spark version.
> kubernetes driver pod not deleted after finish.
> -----------------------------------------------
>
> Key: SPARK-27696
> URL: https://issues.apache.org/jira/browse/SPARK-27696
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.4.0
> Reporter: Henry Yu
> Priority: Major
>
> When submit to k8s, driver pod not deleted after job completion.
> While k8s checks driver pod name not existing, It is especially painful when
> we use workflow tool to resubmit the failed spark job. (by the way, client
> always exist with 0 is another painful issue)
> I have fix this with a new config
> spark.kubernetes.submission.deleteCompletedPod=true in our home maintained
> spark version.
> Do you guys have more insights or I can make a pr on this issue?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
