[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276891#comment-17276891
]
Dongjoon Hyun commented on SPARK-29220:
---
I agree with you, [~attilapiros].
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Spark Core, Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17275786#comment-17275786
]
Attila Zsolt Piros commented on SPARK-29220:
I think we can close this as with https://github.com/apache/spark/pull/31363
the exception is extended with the stack trace of the stuck thread and by
https://github.com/apache/spark/pull/31397 a fix is provided.
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Spark Core, Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937170#comment-16937170
]
Jungtaek Lim commented on SPARK-29220:
--
Could we expose a way to store these unit-test.log files in all modules to
somewhere, like build artifacts? For example, see files under "container 0" in
this page:
[https://circleci.com/gh/HeartSaVioR/spark-state-tools/68#artifacts/containers/0]
I understand Amplab CI runs plenty of builds so space might be matter - they
would be mostly only needed for failed builds. Zipped file would be also OK -
even better than nothing.
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
>
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937155#comment-16937155
]
shane knapp commented on SPARK-29220:
-
it's ok... let's try and keep an eye on these tests for other PRs and see if
it pops up again.
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937154#comment-16937154
]
Dongjoon Hyun commented on SPARK-29220:
---
Since I'm monitoring Jenkins jobs, I'll ping you if I see this failure again.
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937152#comment-16937152
]
Dongjoon Hyun commented on SPARK-29220:
---
Thank you for investigation, [~shaneknapp]. Yes. I merged that PR because I
though it's irrelevant to this flakiness. Do you want to re-trigger on that PR?
Then, sorry about that~
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937126#comment-16937126
]
shane knapp commented on SPARK-29220:
-
hmm, i see that https://github.com/apache/spark/pull/25901 has been merged in.
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937124#comment-16937124
]
shane knapp commented on SPARK-29220:
-
ARGH i just messed up and accidentally overwrote all of these logs. :(
i'll retrigger a build and if it happens again let me know here and we can
investigate fully. i'm not really sure what's happening, as i'm not a
java/scala experten, and other pull request builds are successfully running on
this worker. the system configs for these machines hasn't changed in a long
time, either.
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
>
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937120#comment-16937120
]
shane knapp commented on SPARK-29220:
-
for
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/
would any of the following logs be useful?
{noformat}
-bash-4.1$ find . | grep "unit-tests\.log"|egrep -i "yarn|sql"
./sql/hive/target/unit-tests.log
./sql/core/target/unit-tests.log
./sql/catalyst/target/unit-tests.log
./sql/hive-thriftserver/target/unit-tests.log
./external/kafka-0-10-sql/target/unit-tests.log
./resource-managers/yarn/target/unit-tests.log
{noformat}
let me know ASAP!
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936294#comment-16936294
]
Jungtaek Lim commented on SPARK-29220:
--
[~dongjoon] Ah yes, I guess you have been dealing with build system recently so
thought you know this. Thanks for forwarding!
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at
[jira] [Commented] (SPARK-29220) Flaky test: org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
[
https://issues.apache.org/jira/browse/SPARK-29220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16936286#comment-16936286
]
Dongjoon Hyun commented on SPARK-29220:
---
Maybe, [~shaneknapp] knows that~
> Flaky test:
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.handle large
> number of containers and tasks (SPARK-18750) [hadoop-3.2][java11]
> --
>
> Key: SPARK-29220
> URL: https://issues.apache.org/jira/browse/SPARK-29220
> Project: Spark
> Issue Type: Test
> Components: Tests, YARN
>Affects Versions: 3.0.0
>Reporter: Jungtaek Lim
>Priority: Minor
>
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111229/testReport/]
> [https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/111236/testReport/]
> {code:java}
> Error Messageorg.scalatest.exceptions.TestFailedException:
> java.lang.StackOverflowError did not equal
> nullStacktracesbt.ForkMain$ForkError:
> org.scalatest.exceptions.TestFailedException: java.lang.StackOverflowError
> did not equal null
> at
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:528)
> at
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:527)
> at
> org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1560)
> at
> org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:501)
> at
> org.apache.spark.deploy.yarn.LocalityPlacementStrategySuite.$anonfun$new$1(LocalityPlacementStrategySuite.scala:48)
> at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> at org.scalatest.Transformer.apply(Transformer.scala:22)
> at org.scalatest.Transformer.apply(Transformer.scala:20)
> at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
> at org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:149)
> at
> org.scalatest.FunSuiteLike.invokeWithFixture$1(FunSuiteLike.scala:184)
> at org.scalatest.FunSuiteLike.$anonfun$runTest$1(FunSuiteLike.scala:196)
> at org.scalatest.SuperEngine.runTestImpl(Engine.scala:289)
> at org.scalatest.FunSuiteLike.runTest(FunSuiteLike.scala:196)
> at org.scalatest.FunSuiteLike.runTest$(FunSuiteLike.scala:178)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterEach.runTest(BeforeAndAfterEach.scala:221)
> at
> org.scalatest.BeforeAndAfterEach.runTest$(BeforeAndAfterEach.scala:214)
> at org.apache.spark.SparkFunSuite.runTest(SparkFunSuite.scala:56)
> at
> org.scalatest.FunSuiteLike.$anonfun$runTests$1(FunSuiteLike.scala:229)
> at
> org.scalatest.SuperEngine.$anonfun$runTestsInBranch$1(Engine.scala:396)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:384)
> at org.scalatest.SuperEngine.runTestsInBranch(Engine.scala:379)
> at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:461)
> at org.scalatest.FunSuiteLike.runTests(FunSuiteLike.scala:229)
> at org.scalatest.FunSuiteLike.runTests$(FunSuiteLike.scala:228)
> at org.scalatest.FunSuite.runTests(FunSuite.scala:1560)
> at org.scalatest.Suite.run(Suite.scala:1147)
> at org.scalatest.Suite.run$(Suite.scala:1129)
> at
> org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1560)
> at org.scalatest.FunSuiteLike.$anonfun$run$1(FunSuiteLike.scala:233)
> at org.scalatest.SuperEngine.runImpl(Engine.scala:521)
> at org.scalatest.FunSuiteLike.run(FunSuiteLike.scala:233)
> at org.scalatest.FunSuiteLike.run$(FunSuiteLike.scala:232)
> at
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterAll$$super$run(SparkFunSuite.scala:56)
> at
> org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:213)
> at org.scalatest.BeforeAndAfterAll.run(BeforeAndAfterAll.scala:210)
> at org.scalatest.BeforeAndAfterAll.run$(BeforeAndAfterAll.scala:208)
> at org.apache.spark.SparkFunSuite.run(SparkFunSuite.scala:56)
> at
> org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:314)
> at
> org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:507)
> at sbt.ForkMain$Run$2.call(ForkMain.java:296)
> at sbt.ForkMain$Run$2.call(ForkMain.java:286)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
>
