This is an automated email from the ASF dual-hosted git repository. yao pushed a commit to branch branch-3.5 in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.5 by this push: new e07e291cc7a [SPARK-44929][TESTS] Standardize log output for console appender in tests e07e291cc7a is described below commit e07e291cc7a325e7b698d1a974ff68423e1781e5 Author: Kent Yao <y...@apache.org> AuthorDate: Thu Aug 24 13:51:26 2023 +0800 [SPARK-44929][TESTS] Standardize log output for console appender in tests ### What changes were proposed in this pull request? This PR set a character length limit for the error message and a stack depth limit for error stack traces to the console appender in tests. The original patterns are - %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex - %t: %m%n%ex And they're adjusted to the new consistent pattern - `%d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n` ### Why are the changes needed? In testing, intentional and unintentional failures are created to generate extensive log volumes. For instance, a single FileNotFound error may be logged multiple times in the writer, task runner, task set manager, and other areas, resulting in thousands of lines per failure. For example, tests in ParquetRebaseDatetimeSuite will be run with V1 and V2 Datasource, two or more specific values, and multiple configuration pairs. I have seen the SparkUpgradeException all over the CI logs ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? ``` build/sbt "sql/testOnly *ParquetRebaseDatetimeV1Suite" ``` ``` 15:59:55.446 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Job job_202308230059551630377040190578321_1301 aborted. 15:59:55.446 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 1301.0 (TID 1595) org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/Users/hzyaoqin/spark/target/tmp/spark-67cce58e-dfb2-4811-a9c0-50ec4c90d1f1. at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:765) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420) at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) 15:59:55.446 WARN org.apache.spark.scheduler.TaskSetManager: Lost task 0.0 in stage 1301.0 (TID 1595) (10.221.97.38 executor driver): org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/Users/hzyaoqin/spark/target/tmp/spark-67cce58e-dfb2-4811-a9c0-50ec4c90d1f1. at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:765) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420) at org.apache.spark.sql.execution.datasources.... 15:59:55.446 ERROR org.apache.spark.scheduler.TaskSetManager: Task 0 in stage 1301.0 failed 1 times; aborting job 15:59:55.447 ERROR org.apache.spark.sql.execution.datasources.FileFormatWriter: Aborting job 0ead031e-c9dd-446b-b20b-c76ec54978b1. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1301.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1301.0 (TID 1595) (10.221.97.38 executor driver): org.apache.spark.SparkException: [TASK_WRITE_FAILED] Task failed while writing rows to file:/Users/hzyaoqin/spark/target/tmp/spark-67cce58e-dfb2-4811-a9c0-50ec4c90d1f1. at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:765) at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:420) at org.apache.spark.sql.execution.datasources.WriteFilesExec.$anonfun$doExecuteWrite$1(WriteFiles.scala:100) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2(RDD.scala:890) at org.apache.spark.rdd.RDD.$anonfun$mapPartitionsInternal$2$adapted(RDD.scala:890) at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:52) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:364) 15:59:55.579 ERROR org.apache.spark.executor.Executor: Exception in task 0.0 in stage 1303.0 (TID 1597) ``` ### Was this patch authored or co-authored using generative AI tooling? no Closes #42627 from yaooqinn/SPARK-44929. Authored-by: Kent Yao <y...@apache.org> Signed-off-by: Kent Yao <y...@apache.org> (cherry picked from commit 830500150f7e3972d1fa5b47d0ab564bfa7e4b12) Signed-off-by: Kent Yao <y...@apache.org> --- connector/connect/client/jvm/src/test/resources/log4j2.properties | 2 +- connector/connect/server/src/test/resources/log4j2.properties | 2 +- connector/protobuf/src/test/resources/log4j2.properties | 2 +- core/src/test/resources/log4j2.properties | 2 +- hadoop-cloud/src/test/resources/log4j2.properties | 2 +- repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala | 2 +- .../src/test/resources/log-config-test-log4j.properties | 2 +- .../org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala | 2 +- .../test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala | 2 +- sql/core/src/test/resources/log4j2.properties | 2 +- sql/hive-thriftserver/src/test/resources/log4j2.properties | 2 +- .../apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala | 2 +- .../scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala | 2 +- sql/hive/src/test/resources/log4j2.properties | 2 +- 14 files changed, 14 insertions(+), 14 deletions(-) diff --git a/connector/connect/client/jvm/src/test/resources/log4j2.properties b/connector/connect/client/jvm/src/test/resources/log4j2.properties index ab02104c696..550fd261b6f 100644 --- a/connector/connect/client/jvm/src/test/resources/log4j2.properties +++ b/connector/connect/client/jvm/src/test/resources/log4j2.properties @@ -32,7 +32,7 @@ appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %t: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n # Ignore messages below warning level from Jetty, because it's a bit verbose logger.jetty.name = org.sparkproject.jetty diff --git a/connector/connect/server/src/test/resources/log4j2.properties b/connector/connect/server/src/test/resources/log4j2.properties index ac4c1de0e91..f782d7f3aaa 100644 --- a/connector/connect/server/src/test/resources/log4j2.properties +++ b/connector/connect/server/src/test/resources/log4j2.properties @@ -32,7 +32,7 @@ appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %t: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n # Ignore messages below warning level from Jetty, because it's a bit verbose logger.jetty.name = org.sparkproject.jetty diff --git a/connector/protobuf/src/test/resources/log4j2.properties b/connector/protobuf/src/test/resources/log4j2.properties index ab02104c696..550fd261b6f 100644 --- a/connector/protobuf/src/test/resources/log4j2.properties +++ b/connector/protobuf/src/test/resources/log4j2.properties @@ -32,7 +32,7 @@ appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %t: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n # Ignore messages below warning level from Jetty, because it's a bit verbose logger.jetty.name = org.sparkproject.jetty diff --git a/core/src/test/resources/log4j2.properties b/core/src/test/resources/log4j2.properties index ab02104c696..550fd261b6f 100644 --- a/core/src/test/resources/log4j2.properties +++ b/core/src/test/resources/log4j2.properties @@ -32,7 +32,7 @@ appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %t: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n # Ignore messages below warning level from Jetty, because it's a bit verbose logger.jetty.name = org.sparkproject.jetty diff --git a/hadoop-cloud/src/test/resources/log4j2.properties b/hadoop-cloud/src/test/resources/log4j2.properties index 01a9cafafa8..215cc8e4888 100644 --- a/hadoop-cloud/src/test/resources/log4j2.properties +++ b/hadoop-cloud/src/test/resources/log4j2.properties @@ -33,7 +33,7 @@ appender.console.type = Console appender.console.name = STDERR appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %t: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n # Ignore messages below warning level from Jetty, because it's a bit verbose logger.jetty.name = org.spark_project.jetty diff --git a/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala b/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala index 1f44826c100..bb2a85cfa0d 100644 --- a/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala +++ b/repl/src/test/scala/org/apache/spark/repl/ReplSuite.scala @@ -284,7 +284,7 @@ class ReplSuite extends SparkFunSuite { |appender.console.target = SYSTEM_ERR |appender.console.follow = true |appender.console.layout.type = PatternLayout - |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex + |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n | |# Set the log level for this class to ERROR same as the default setting. |logger.repl.name = org.apache.spark.repl.Main diff --git a/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties b/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties index 4aeea886251..c30c42e1ebd 100644 --- a/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties +++ b/resource-managers/kubernetes/integration-tests/src/test/resources/log-config-test-log4j.properties @@ -23,4 +23,4 @@ appender.console.type = Console appender.console.name = console appender.console.target = SYSTEM_ERR appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n diff --git a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala index aea4486e0e2..743de91b08f 100644 --- a/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala +++ b/resource-managers/kubernetes/integration-tests/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/DecommissionSuite.scala @@ -47,7 +47,7 @@ private[spark] trait DecommissionSuite { k8sSuite: KubernetesSuite => |appender.console.name = console |appender.console.target = SYSTEM_OUT |appender.console.layout.type = PatternLayout - |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex + |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n | |logger.spark.name = org.apache.spark |logger.spark.level = debug diff --git a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala index 472f29d1117..d166c50a1dd 100644 --- a/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala +++ b/resource-managers/yarn/src/test/scala/org/apache/spark/deploy/yarn/BaseYarnClusterSuite.scala @@ -51,7 +51,7 @@ abstract class BaseYarnClusterSuite extends SparkFunSuite with Matchers { |appender.console.name = console |appender.console.target = SYSTEM_ERR |appender.console.layout.type = PatternLayout - |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex + |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n |logger.jetty.name = org.sparkproject.jetty |logger.jetty.level = warn |logger.eclipse.name = org.eclipse.jetty diff --git a/sql/core/src/test/resources/log4j2.properties b/sql/core/src/test/resources/log4j2.properties index 49df23c4fc7..7ab47c16d4f 100644 --- a/sql/core/src/test/resources/log4j2.properties +++ b/sql/core/src/test/resources/log4j2.properties @@ -25,7 +25,7 @@ appender.console.type = Console appender.console.name = STDOUT appender.console.target = SYSTEM_OUT appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n appender.console.filter.threshold.type = ThresholdFilter appender.console.filter.threshold.level = warn diff --git a/sql/hive-thriftserver/src/test/resources/log4j2.properties b/sql/hive-thriftserver/src/test/resources/log4j2.properties index cc483f4671e..ebb3a6ccb2f 100644 --- a/sql/hive-thriftserver/src/test/resources/log4j2.properties +++ b/sql/hive-thriftserver/src/test/resources/log4j2.properties @@ -25,7 +25,7 @@ appender.console.type = Console appender.console.name = STDOUT appender.console.target = SYSTEM_OUT appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n appender.console.filter.1.type = Filters diff --git a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala index 8fd432148bb..b421a94a06c 100644 --- a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala +++ b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala @@ -1229,7 +1229,7 @@ abstract class HiveThriftServer2TestBase extends SparkFunSuite with BeforeAndAft |appender.console.name = console |appender.console.target = SYSTEM_ERR |appender.console.layout.type = PatternLayout - |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex + |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n """.stripMargin, new File(s"$tempLog4jConf/log4j2.properties"), StandardCharsets.UTF_8) diff --git a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala index 02ee8dd273d..53a69d964b1 100644 --- a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala +++ b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/UISeleniumSuite.scala @@ -80,7 +80,7 @@ class UISeleniumSuite |appender.console.name = console |appender.console.target = SYSTEM_ERR |appender.console.layout.type = PatternLayout - |appender.console.layout.pattern = %d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n%ex + |appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n """.stripMargin, new File(s"$tempLog4jConf/log4j2.properties"), StandardCharsets.UTF_8) diff --git a/sql/hive/src/test/resources/log4j2.properties b/sql/hive/src/test/resources/log4j2.properties index 6e2c01c4b14..c91f417bc6b 100644 --- a/sql/hive/src/test/resources/log4j2.properties +++ b/sql/hive/src/test/resources/log4j2.properties @@ -25,7 +25,7 @@ appender.console.type = Console appender.console.name = STDOUT appender.console.target = SYSTEM_OUT appender.console.layout.type = PatternLayout -appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %m%n%ex +appender.console.layout.pattern = %d{HH:mm:ss.SSS} %p %c: %maxLen{%m}{512}%n%ex{8}%n appender.console.filter.threshold.type = ThresholdFilter appender.console.filter.threshold.level = warn --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org