[spark] branch master updated: [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl`
This is an automated email from the ASF dual-hosted git repository. dongjoon pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 3b8465b41ec [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` 3b8465b41ec is described below commit 3b8465b41ec3ecc1f97701e93f7043bf0cacb42e Author: Dongjoon Hyun AuthorDate: Sat May 28 10:38:16 2022 -0700 [SPARK-39324][CORE] Log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` ### What changes were proposed in this pull request? This PR aims to log `ExecutorDecommission` as INFO level in `TaskSchedulerImpl` in Apache Spark 3.4. ### Why are the changes needed? Like the other modules, `TaskSchedulerImpl` had better log the decommission as `INFO` level instead of `ERROR`. ``` 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Decommission executors: 8 22/05/28 01:25:28 INFO KubernetesClusterSchedulerBackend: Notify executor 8 to decommissioning. 22/05/28 01:25:28 INFO BlockManagerMasterEndpoint: Mark BlockManagers (BlockManagerId(8, 100.103.40.13, 43353, None)) as being decommissioning. 22/05/28 01:25:29 ERROR TaskSchedulerImpl: Lost executor 8 on 100.103.40.13: Executor decommission. 22/05/28 01:25:29 INFO ExecutorMonitor: Executor 8 is removed. Remove reason statistics: ... 22/05/28 01:25:29 INFO DAGScheduler: Executor lost: 8 (epoch 7) 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. 22/05/28 01:25:29 INFO BlockManagerMasterEndpoint: Removing block manager BlockManagerId(8, 100.103.40.13, 43353, None) 22/05/28 01:25:29 INFO BlockManagerMaster: Removed 8 successfully in removeExecutor 22/05/28 01:25:29 INFO DAGScheduler: Shuffle files lost for executor: 8 (epoch 7) 22/05/28 01:25:34 INFO BlockManagerMaster: Removal of executor 8 requested 22/05/28 01:25:34 INFO BlockManagerMasterEndpoint: Trying to remove executor 8 from BlockManagerMaster. 22/05/28 01:25:34 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asked to remove non-existent executor 8 ``` ### Does this PR introduce _any_ user-facing change? No. This is a change on log level. ### How was this patch tested? Pass the CIs. Closes #36707 from dongjoon-hyun/SPARK-39324. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala | 2 ++ 1 file changed, 2 insertions(+) diff --git a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala index 55db73ab2a0..55938d8b77e 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/TaskSchedulerImpl.scala @@ -1018,6 +1018,8 @@ private[spark] class TaskSchedulerImpl( logDebug(s"Executor $executorId on $hostPort lost, but reason not yet known.") case ExecutorKilled => logInfo(s"Executor $executorId on $hostPort killed by driver.") +case _: ExecutorDecommission => + logInfo(s"Executor $executorId on $hostPort is decommissioned.") case _ => logError(s"Lost executor $executorId on $hostPort: $reason") } - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-39282][SQL] Replace If-Else branch with bitwise operators in roundNumberOfBytesToNearestWord
This is an automated email from the ASF dual-hosted git repository. srowen pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new b12c12790c7 [SPARK-39282][SQL] Replace If-Else branch with bitwise operators in roundNumberOfBytesToNearestWord b12c12790c7 is described below commit b12c12790c7c4453c1f9e62d49feba30957c0af3 Author: zhixingheyi-tian AuthorDate: Sat May 28 14:15:59 2022 -0500 [SPARK-39282][SQL] Replace If-Else branch with bitwise operators in roundNumberOfBytesToNearestWord ### What changes were proposed in this pull request? Use bitwise operators to avoid If-Else branch and improve computation performance. ### How was this patch tested? Existed UTs. Closes #36659 from zhixingheyi-tian/avoid_ifelse_branch. Authored-by: zhixingheyi-tian Signed-off-by: Sean Owen --- .../main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java | 6 +- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java b/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java index deb7d2bf1b0..500bc9de325 100644 --- a/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java +++ b/common/unsafe/src/main/java/org/apache/spark/unsafe/array/ByteArrayMethods.java @@ -39,11 +39,7 @@ public class ByteArrayMethods { public static long roundNumberOfBytesToNearestWord(long numBytes) { long remainder = numBytes & 0x07; // This is equivalent to `numBytes % 8` -if (remainder == 0) { - return numBytes; -} else { - return numBytes + (8 - remainder); -} +return numBytes + ((8 - remainder) & 0x7); } // Some JVMs can't allocate arrays of length Integer.MAX_VALUE; actual max is somewhat smaller. - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated: [SPARK-39261][CORE][FOLLOWUP] Improve newline formatting for error messages
This is an automated email from the ASF dual-hosted git repository. yumwang pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/master by this push: new 00277f5370c [SPARK-39261][CORE][FOLLOWUP] Improve newline formatting for error messages 00277f5370c is described below commit 00277f5370c56d207aae437293958d1b5e4470d9 Author: sychen AuthorDate: Sun May 29 09:38:34 2022 +0800 [SPARK-39261][CORE][FOLLOWUP] Improve newline formatting for error messages ### What changes were proposed in this pull request? Use `java.nio.file.Files.delete` instead of `org.apache.commons.io.FileUtils#delete` ### Why are the changes needed? `org.apache.commons.io.FileUtils#delete` is a method only available in version 2.9, hadoop2 uses version 2.4 of commons-io. Build failed ``` ./dev/make-distribution.sh --tgz -Pyarn -Phive -Phive-thriftserver -Phadoop-2 -DskipTests -Dmaven.javadoc.skip=true ``` ``` spark/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala:92: value delete is not a member of object org.apache.commons.io.FileUtils ``` ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? exist UT Closes #36710 from cxzl25/SPARK-39261-FOLLOWUP. Authored-by: sychen Signed-off-by: Yuming Wang --- core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala index 6ad80950931..9092571b9b9 100644 --- a/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala +++ b/core/src/test/scala/org/apache/spark/SparkThrowableSuite.scala @@ -19,6 +19,7 @@ package org.apache.spark import java.io.File import java.nio.charset.StandardCharsets +import java.nio.file.Files import java.util.IllegalFormatException import com.fasterxml.jackson.annotation.JsonInclude.Include @@ -89,7 +90,7 @@ class SparkThrowableSuite extends SparkFunSuite { if (rewrittenString.trim != errorClassFileContents.trim) { val errorClassesFile = new File(errorClassDir, new File(errorClassesUrl.getPath).getName) logInfo(s"Regenerating error class file $errorClassesFile") -FileUtils.delete(errorClassesFile) +Files.delete(errorClassesFile.toPath) FileUtils.writeStringToFile(errorClassesFile, rewrittenString, StandardCharsets.UTF_8) } } else { - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org