[ https://issues.apache.org/jira/browse/SPARK-25543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dongjoon Hyun updated SPARK-25543: ---------------------------------- Affects Version/s: (was: 2.5.0) 2.4.0 > Confusing log messages at DEBUG level, in K8s mode. > --------------------------------------------------- > > Key: SPARK-25543 > URL: https://issues.apache.org/jira/browse/SPARK-25543 > Project: Spark > Issue Type: Bug > Components: Kubernetes > Affects Versions: 2.4.0 > Reporter: Prashant Sharma > Assignee: Prashant Sharma > Priority: Minor > Fix For: 2.4.1, 2.5.0 > > > Steps to reproduce. > Start spark shell by providing a K8s master. Then turn the debug log on, > {code} > scala> sc.setLogLevel("DEBUG") > {code} > {code} > sc.setLogLevel("DEBUG") > scala> 2018-09-26 09:33:54 DEBUG ExecutorPodsLifecycleManager:58 - Removed > executors with ids from Spark that were either found to be deleted or > non-existent in the cluster. > 2018-09-26 09:33:55 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:33:56 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:33:56 DEBUG ExecutorPodsPollingSnapshotSource:58 - > Resynchronizing full executor pod state from Kubernetes. > 2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Currently have 1 running > executors and 0 pending executors. Map() executors have been requested but > are pending appearance in the cluster. > 2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Current number of > running executors is equal to the number of requested executors. Not scaling > up further. > 2018-09-26 09:33:57 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:33:58 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:33:59 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:34:00 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:34:01 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:34:02 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:34:03 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:34:04 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:34:05 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from Spark that were either found to be deleted or non-existent in > the cluster. > 2018-09-26 09:34:06 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors > with ids from ... > {code} > The fix is easy, first check if there are any removed executors, before > producing the log message. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org