Re: Spark driver pod garbage collection

2018-05-23 Thread Anirudh Ramanathan
There's a flag to the controller manager that is in charge of retention policy for terminated or completed pods. https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/#options --terminated-pod-gc-threshold int32 Default: 12500 Number of terminated pods that

Spark driver pod garbage collection

2018-05-23 Thread purna pradeep
Hello, Currently I observe dead pods are not getting garbage collected (aka spark driver pods which have completed execution). So pods could sit in the namespace for weeks potentially. This makes listing, parsing, and reading pods slower and well as having junk sit on the cluster. I believe