[
https://issues.apache.org/jira/browse/SPARK-25543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun reassigned SPARK-25543:
-------------------------------------
Assignee: Prashant Sharma
> Confusing log messages at DEBUG level, in K8s mode.
> ---------------------------------------------------
>
> Key: SPARK-25543
> URL: https://issues.apache.org/jira/browse/SPARK-25543
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes
> Affects Versions: 2.5.0
> Reporter: Prashant Sharma
> Assignee: Prashant Sharma
> Priority: Minor
> Fix For: 2.4.1, 2.5.0
>
>
> Steps to reproduce.
> Start spark shell by providing a K8s master. Then turn the debug log on,
> {code}
> scala> sc.setLogLevel("DEBUG")
> {code}
> {code}
> sc.setLogLevel("DEBUG")
> scala> 2018-09-26 09:33:54 DEBUG ExecutorPodsLifecycleManager:58 - Removed
> executors with ids from Spark that were either found to be deleted or
> non-existent in the cluster.
> 2018-09-26 09:33:55 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:33:56 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:33:56 DEBUG ExecutorPodsPollingSnapshotSource:58 -
> Resynchronizing full executor pod state from Kubernetes.
> 2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Currently have 1 running
> executors and 0 pending executors. Map() executors have been requested but
> are pending appearance in the cluster.
> 2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Current number of
> running executors is equal to the number of requested executors. Not scaling
> up further.
> 2018-09-26 09:33:57 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:33:58 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:33:59 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:34:00 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:34:01 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:34:02 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:34:03 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:34:04 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:34:05 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from Spark that were either found to be deleted or non-existent in
> the cluster.
> 2018-09-26 09:34:06 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
> with ids from ...
> {code}
> The fix is easy, first check if there are any removed executors, before
> producing the log message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
