Re: [K8S] ExecutorPodsWatchSnapshotSource with no spark-exec-inactive label in 3.1?

[attilapiros]

Hi,

I do not think this could cause any problem.

The PODs polling snapshots would contain those PODs which were inactivated
before. Every time when the polling is triggered, so here it make sense to
handle them like deleted PODs and skip those.

On the other hand PODs watcher only informed about state changes
(https://kubernetes.io/docs/reference/kubernetes-api/common-parameters/common-parameters/#watch)
and here we are talking about inactivated PODs which I do not expect to have
many state changes.

Best regards,
Attila




--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/

-
To unsubscribe e-mail: dev-unsubscr...@spark.apache.org

[K8S] ExecutorPodsWatchSnapshotSource with no spark-exec-inactive label in 3.1?

[Jacek Laskowski]

Hi,

spark-exec-inactive label is not used to watch executor pods
in ExecutorPodsWatchSnapshotSource [1] as it happens in PollRunnable [2].

Any reasons? Am I correct to consider it a bug?

[1]
https://github.com/apache/spark/blob/270a9a3cac25f3e799460320d0fc94ccd7ecfaea/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsWatchSnapshotSource.scala#L39-L42
[2]
https://github.com/apache/spark/blob/ee624821a903a263c844ed849d6833df8e9ad43e/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsPollingSnapshotSource.scala#L62

Pozdrawiam,
Jacek Laskowski

https://about.me/JacekLaskowski
"The Internals Of" Online Books 
Follow me on https://twitter.com/jaceklaskowski