[ https://issues.apache.org/jira/browse/SPARK-27696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17038778#comment-17038778 ]
Henry Yu commented on SPARK-27696: ---------------------------------- Too muck un deleted pod will lead to Etcd heavy pressure. we have fixed this by pull driver log when finish job and delete driver pod. > kubernetes driver pod not deleted after finish. > ----------------------------------------------- > > Key: SPARK-27696 > URL: https://issues.apache.org/jira/browse/SPARK-27696 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 2.4.0 > Reporter: Henry Yu > Priority: Minor > > When submit to k8s, driver pod not deleted after job completion. > While k8s checks driver pod name not existing, It is especially painful when > we use workflow tool to resubmit the failed spark job. (by the way, client > always exist with 0 is another painful issue) > I have fix this with a new config > spark.kubernetes.submission.deleteCompletedPod=true in our home maintained > spark version. > Do you guys have more insights or I can make a pr on this issue? -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org