This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push: new 3b8465b41ec [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` 3b8465b41ec is described below commit 3b8465b41ec3ecc1f97701e93f7043bf0cacb42e Author: Dongjoon Hyun <dongj...@apache.org> AuthorDate: Sat May 28 10:38:16 2022 -0700 [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` ### What changes were proposed in this pull request? This PR aims to log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` in Apache Spark 3.4. ### Why are the changes needed? Like the other modules, `TaskSchedulerImpl` had better log the decommission as `INFO` level instead of `ERROR`. ``` 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission executors: 8 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to decommissioning. 22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning. 22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: Executor decommission. 22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason statistics: ... 22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7) 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(8, 100.103.40.13, 43353, None) 22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in removeExecutor 22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 7) 22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested 22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. 22/05/28 01:25:34 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 8 ``` ### Does this PR introduce _any_ user-facing change? No. This is a change on log level. ### How was this patch tested? Pass the CIs. Closes #36707 from dongjoon-hyun/SPARK-39324. Authored-by: Dongjoon Hyun <dongj...@apache.org> Signed-off-by: Dongjoon Hyun <dongj...@apache.org> --- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala index 55db73ab2a0..55938d8b77e 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala @@ -1018,6 +1018,8 @@ private[spark] class TaskSchedulerImpl( logDebug(s"Executor $executorId on $hostPort lost, but reason not yet known.") case ExecutorKilled => logInfo(s"Executor $executorId on $hostPort killed by driver.") + case _: ExecutorDecommission => + logInfo(s"Executor $executorId on $hostPort is decommissioned.") case _ => logError(s"Lost executor $executorId on $hostPort: $reason") } --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org