[
https://issues.apache.org/jira/browse/SPARK-37743?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-37743:
---------------------------------
Description:
{code}
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala: //
TODO(SPARK-9603): Building a package is flaky on Jenkins Maven builds.
core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala: //
This test suite has some weird behaviors when executed on Jenkins:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala: //
1. Sometimes it gets extremely slow out of unknown reason on Jenkins. Here we
add a
core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala: //
is only 2, while on Jenkins it's 32. For this specific test, 2 available
processors, which
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala:
* when running on a slow Jenkins machine) before records start to be removed.
To make sure a test
project/SparkBuild.scala: // with Jenkins flakiness.
repl/src/test/scala/org/apache/spark/repl/SparkShellSuite.scala: // This
test suite sometimes gets extremely slow out of unknown reason on Jenkins.
Here we
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
// TODO: Why fs.getContentSummary returns wrong size on Jenkins?
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
// Seems fs.getContentSummary returns wrong table size on Jenkins. So we use
sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala: // the
last dot due to a bug in SBT. This makes easier to debug via Jenkins test result
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreIntegrationSuite.scala:
// Should emit new progresses every 10 ms, but we could be facing a
slow Jenkins
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryStatusAndProgressSuite.scala:
// Should emit new progresses every 10 ms, but we could be facing a
slow Jenkins
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala:
// This test suite sometimes gets extremely slow out of unknown reason on
Jenkins. Here we
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala:
// started at a time, which is not Jenkins friendly.
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientBuilder.scala:
// In order to speed up test execution during development or in Jenkins, you
can specify the path
sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala:
// more cores, the issue can be reproduced steadily. Fortunately our Jenkins
builder meets this
streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala:
// If Jenkins is slow, we may not have a chance to run many threads
simultaneously. Having
core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
// This is to avoid any potential flakiness in the test because of large
pauses in jenkins
sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java:
// (e.g., /home/jenkins/workspace/SparkPullRequestBuilder@2)
sql/core/src/test/resources/sql-tests/inputs/datetime-parsing-invalid.sql:-- in
java 8 this case is invalid, but valid in java 11, disabled for jenkins
sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala:
// TODO: Figure out why StringType doesn't work on jenkins.
sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala:
// Weird DDL differences result in failures on jenkins.
{code}
was:
{code}
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala: //
TODO(SPARK-9603): Building a package is flaky on Jenkins Maven builds.
core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala: //
This test suite has some weird behaviors when executed on Jenkins:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala: //
1. Sometimes it gets extremely slow out of unknown reason on Jenkins. Here we
add a
core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala: //
is only 2, while on Jenkins it's 32. For this specific test, 2 available
processors, which
external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala:
* when running on a slow Jenkins machine) before records start to be removed.
To make sure a test
project/SparkBuild.scala: // with Jenkins flakiness.
repl/src/test/scala/org/apache/spark/repl/SparkShellSuite.scala: // This
test suite sometimes gets extremely slow out of unknown reason on Jenkins.
Here we
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
// TODO: Why fs.getContentSummary returns wrong size on Jenkins?
sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
// Seems fs.getContentSummary returns wrong table size on Jenkins. So we use
sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala: // the
last dot due to a bug in SBT. This makes easier to debug via Jenkins test result
sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreIntegrationSuite.scala:
// Should emit new progresses every 10 ms, but we could be facing a
slow Jenkins
sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryStatusAndProgressSuite.scala:
// Should emit new progresses every 10 ms, but we could be facing a
slow Jenkins
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala:
// This test suite sometimes gets extremely slow out of unknown reason on
Jenkins. Here we
sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala:
// started at a time, which is not Jenkins friendly.
sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientBuilder.scala:
// In order to speed up test execution during development or in Jenkins, you
can specify the path
sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala:
// more cores, the issue can be reproduced steadily. Fortunately our Jenkins
builder meets this
streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala:
// If Jenkins is slow, we may not have a chance to run many threads
simultaneously. Having
{code}
> Remove/update obsolete codes in Scala tests
> -------------------------------------------
>
> Key: SPARK-37743
> URL: https://issues.apache.org/jira/browse/SPARK-37743
> Project: Spark
> Issue Type: Sub-task
> Components: Tests
> Affects Versions: 3.3.0
> Reporter: Hyukjin Kwon
> Priority: Major
>
> {code}
> core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala: //
> TODO(SPARK-9603): Building a package is flaky on Jenkins Maven builds.
> core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala:
> // This test suite has some weird behaviors when executed on Jenkins:
> core/src/test/scala/org/apache/spark/deploy/SparkSubmitTestUtils.scala:
> // 1. Sometimes it gets extremely slow out of unknown reason on Jenkins.
> Here we add a
> core/src/test/scala/org/apache/spark/deploy/master/MasterSuite.scala: //
> is only 2, while on Jenkins it's 32. For this specific test, 2 available
> processors, which
> external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaDontFailOnDataLossSuite.scala:
> * when running on a slow Jenkins machine) before records start to be
> removed. To make sure a test
> project/SparkBuild.scala: // with Jenkins flakiness.
> repl/src/test/scala/org/apache/spark/repl/SparkShellSuite.scala: // This
> test suite sometimes gets extremely slow out of unknown reason on Jenkins.
> Here we
> sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
> // TODO: Why fs.getContentSummary returns wrong size on Jenkins?
> sql/core/src/main/scala/org/apache/spark/sql/execution/command/CommandUtils.scala:
> // Seems fs.getContentSummary returns wrong table size on Jenkins. So we
> use
> sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala: //
> the last dot due to a bug in SBT. This makes easier to debug via Jenkins test
> result
> sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreIntegrationSuite.scala:
> // Should emit new progresses every 10 ms, but we could be facing a
> slow Jenkins
> sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamingQueryStatusAndProgressSuite.scala:
> // Should emit new progresses every 10 ms, but we could be facing a
> slow Jenkins
> sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/CliSuite.scala:
> // This test suite sometimes gets extremely slow out of unknown reason
> on Jenkins. Here we
> sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/HiveThriftServer2Suites.scala:
> // started at a time, which is not Jenkins friendly.
> sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientBuilder.scala:
> // In order to speed up test execution during development or in Jenkins,
> you can specify the path
> sql/hive/src/test/scala/org/apache/spark/sql/sources/ParquetHadoopFsRelationSuite.scala:
> // more cores, the issue can be reproduced steadily. Fortunately our
> Jenkins builder meets this
> streaming/src/test/scala/org/apache/spark/streaming/util/WriteAheadLogSuite.scala:
> // If Jenkins is slow, we may not have a chance to run many threads
> simultaneously. Having
> core/src/test/scala/org/apache/spark/scheduler/TaskSchedulerImplSuite.scala:
> // This is to avoid any potential flakiness in the test because of large
> pauses in jenkins
> sql/core/src/test/java/test/org/apache/spark/sql/JavaDataFrameSuite.java:
> // (e.g., /home/jenkins/workspace/SparkPullRequestBuilder@2)
> sql/core/src/test/resources/sql-tests/inputs/datetime-parsing-invalid.sql:--
> in java 8 this case is invalid, but valid in java 11, disabled for jenkins
> sql/core/src/test/scala/org/apache/spark/sql/execution/vectorized/ColumnarBatchSuite.scala:
> // TODO: Figure out why StringType doesn't work on jenkins.
> sql/hive/compatibility/src/test/scala/org/apache/spark/sql/hive/execution/HiveCompatibilitySuite.scala:
> // Weird DDL differences result in failures on jenkins.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
