This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 3b8465b41ec [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO
level in `TaskSchedulerImpl`
3b8465b41ec is described below
commit 3b8465b41ec3ecc1f97701e93f7043bf0cacb42e
Author: Dongjoon Hyun <dongj...@apache.org>
AuthorDate: Sat May 28 10:38:16 2022 -0700
[SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in
`TaskSchedulerImpl`
### What changes were proposed in this pull request?
This PR aims to log `ExecutorDecommission` as INFO level in
`TaskSchedulerImpl` in Apache Spark 3.4.
### Why are the changes needed?
Like the other modules, `TaskSchedulerImpl` had better log the decommission
as `INFO` level instead of `ERROR`.
```
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission
executors: 8
22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8
to decommissioning.
22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers
(BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning.
22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on
100.103.40.13: Executor decommission.
22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove
reason statistics: ...
22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7)
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove
executor 8 from BlockManagerMaster.
22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager
BlockManagerId(8, 100.103.40.13, 43353, None)
22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in
removeExecutor
22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8
(epoch 7)
22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested
22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove
executor 8 from BlockManagerMaster.
22/05/28 01:25:34 INFO
KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove
non-existent executor 8
```
### Does this PR introduce _any_ user-facing change?
No. This is a change on log level.
### How was this patch tested?
Pass the CIs.
Closes #36707 from dongjoon-hyun/SPARK-39324.
Authored-by: Dongjoon Hyun <dongj...@apache.org>
Signed-off-by: Dongjoon Hyun <dongj...@apache.org>
---
core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
index 55db73ab2a0..55938d8b77e 100644
--- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
+++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala
@@ -1018,6 +1018,8 @@ private[spark] class TaskSchedulerImpl(
logDebug(s"Executor $executorId on $hostPort lost, but reason not yet
known.")
case ExecutorKilled =>
logInfo(s"Executor $executorId on $hostPort killed by driver.")
+ case _: ExecutorDecommission =>
+ logInfo(s"Executor $executorId on $hostPort is decommissioned.")
case _ =>
logError(s"Lost executor $executorId on $hostPort: $reason")
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
