Prashant Sharma created SPARK-25543:
---------------------------------------
Summary: Confusing log messages at DEBUG level, in K8s mode.
Key: SPARK-25543
URL: https://issues.apache.org/jira/browse/SPARK-25543
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 2.5.0
Reporter: Prashant Sharma
Steps to reproduce.
Start spark shell by providing a K8s master. Then turn the debug log on,
{code}
scala> sc.setLogLevel("DEBUG")
{code}
{code}
sc.setLogLevel("DEBUG")
scala> 2018-09-26 09:33:54 DEBUG ExecutorPodsLifecycleManager:58 - Removed
executors with ids from Spark that were either found to be deleted or
non-existent in the cluster.
2018-09-26 09:33:55 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:33:56 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:33:56 DEBUG ExecutorPodsPollingSnapshotSource:58 -
Resynchronizing full executor pod state from Kubernetes.
2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Currently have 1 running
executors and 0 pending executors. Map() executors have been requested but are
pending appearance in the cluster.
2018-09-26 09:33:57 DEBUG ExecutorPodsAllocator:58 - Current number of running
executors is equal to the number of requested executors. Not scaling up further.
2018-09-26 09:33:57 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:33:58 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:33:59 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:34:00 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:34:01 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:34:02 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:34:03 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:34:04 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:34:05 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from Spark that were either found to be deleted or non-existent in
the cluster.
2018-09-26 09:34:06 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors
with ids from ...
{code}
The fix is easy, first check if there are any removed executors, before
producing the log message.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
