[
https://issues.apache.org/jira/browse/SPARK-52864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-52864:
-----------------------------------
Labels: pull-request-available (was: )
> Let LocalSparkContext clear active context in beforeAll
> -------------------------------------------------------
>
> Key: SPARK-52864
> URL: https://issues.apache.org/jira/browse/SPARK-52864
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 3.5.6
> Reporter: Price Qian
> Priority: Trivial
> Labels: pull-request-available
>
> Tests that mix in trait
> [LocalSparkContext|https://github.com/apache/spark/blob/master/core/src/test/scala/org/apache/spark/LocalSparkContext.scala]
> will clean up SparkContext properly. The issue is when some tests do not mix
> in it and forget to clean up SparkContext. When running alone, they can all
> pass but when running together, the lingering SparkContext can cause issues
> when a different test tries to do {{new SparkContext}} because SparkContext
> doesn't allow multiple active instances. We can make this shared trait more
> defensive/robust by preemptively clear active SparkContext at the beginning.
>
> To reproduce the issue, you can add 2 test suites to {{core}} module:
> 1.
> core/src/test/scala/org/apache/spark/PriceTest1Suite.scala
>
> {code:java}
> package org.apache.spark
>
> class PriceTest1Suite extends SparkFunSuite {
> test("start a SparkContext without stopping") {
> val conf = new SparkConf().setAppName("test").setMaster("local")
> new SparkContext(conf)
> }
> }
> {code}
>
> 2.
> core/src/test/scala/org/apache/spark/PriceTest2Suite.scala
>
> {code:java}
> package org.apache.spark
>
> class PriceTest2Suite extends SparkFunSuite {
> test("start a SparkContext without stopping") {
> val conf = new SparkConf().setAppName("test").setMaster("local")
> new SparkContext(conf)
> }
> }
> {code}
>
> Run them together by:
> {code:java}
> mvn test -pl core -Dtest=none
> -DwildcardSuites=org.apache.spark.PriceTest1Suite,org.apache.spark.PriceTest2Suite{code}
> I get the following error:
>
> {code:java}
> Discovery starting.
> Discovery completed in 1 second, 511 milliseconds.
> Run starting. Expected test count is: 2
> PriceTest1Suite:
> - start a SparkContext without stopping
> PriceTest2Suite:
> - start a SparkContext without stopping *** FAILED ***
> org.apache.spark.SparkException: Only one SparkContext should be running in
> this JVM (see SPARK-2243).The currently running SparkContext was created at:
> org.apache.spark.PriceTest1Suite.$anonfun$new$1(PriceTest1Suite.scala:6)
> org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
> org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
> org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
> org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
> org.apache.spark.SparkFunSuite.failAfter(SparkFunSuite.scala:69)
> org.apache.spark.SparkFunSuite.$anonfun$test$2(SparkFunSuite.scala:155)
> org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
> org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
> org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
> org.scalatest.Transformer.apply(Transformer.scala:22)
> org.scalatest.Transformer.apply(Transformer.scala:20)
> org.scalatest.funsuite.AnyFunSuiteLike$$anon$1.apply(AnyFunSuiteLike.scala:226)
> org.apache.spark.SparkFunSuite.withFixture(SparkFunSuite.scala:227)
> org.scalatest.funsuite.AnyFunSuiteLike.invokeWithFixture$1(AnyFunSuiteLike.scala:224)
> org.scalatest.funsuite.AnyFunSuiteLike.$anonfun$runTest$1(AnyFunSuiteLike.scala:236)
> org.scalatest.SuperEngine.runTestImpl(Engine.scala:306)
> org.scalatest.funsuite.AnyFunSuiteLike.runTest(AnyFunSuiteLike.scala:236)
> org.scalatest.funsuite.AnyFunSuiteLike.runTest$(AnyFunSuiteLike.scala:218)
> org.apache.spark.SparkFunSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(SparkFunSuite.scala:69)
> at
> org.apache.spark.SparkContext$.$anonfun$assertNoOtherContextIsRunning$2(SparkContext.scala:3018)
> at scala.Option.foreach(Option.scala:437)
> at
> org.apache.spark.SparkContext$.assertNoOtherContextIsRunning(SparkContext.scala:3015)
> at
> org.apache.spark.SparkContext$.markPartiallyConstructed(SparkContext.scala:3106)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:101)
> at org.apache.spark.PriceTest2Suite.$anonfun$new$1(PriceTest2Suite.scala:6)
> at org.scalatest.enablers.Timed$$anon$1.timeoutAfter(Timed.scala:127)
> at org.scalatest.concurrent.TimeLimits$.failAfterImpl(TimeLimits.scala:282)
> at org.scalatest.concurrent.TimeLimits.failAfter(TimeLimits.scala:231)
> at org.scalatest.concurrent.TimeLimits.failAfter$(TimeLimits.scala:230)
> ...
> Run completed in 2 seconds, 411 milliseconds.
> Total number of tests run: 2
> Suites: completed 4, aborted 0
> Tests: succeeded 1, failed 1, canceled 0, ignored 0, pending 0
> *** 1 TEST FAILED ***
>
> {code}
> If any of these 2 tests clears the active SparkContext at the beginning, this
> error can be avoided. By adding the active clear in LocalSparkContext, we can
> prevent many developers from running into the same issue and save them much
> valuable time.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
