Github user gaborgsomogyi commented on a diff in the pull request: https://github.com/apache/spark/pull/20888#discussion_r183067778 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameRangeSuite.scala --- @@ -152,39 +154,53 @@ class DataFrameRangeSuite extends QueryTest with SharedSQLContext with Eventuall } test("Cancelling stage in a query with Range.") { - val listener = new SparkListener { - override def onJobStart(jobStart: SparkListenerJobStart): Unit = { - eventually(timeout(10.seconds), interval(1.millis)) { - assert(DataFrameRangeSuite.stageToKill > 0) + // Save and restore the value because SparkContext is shared + val savedInterruptOnCancel = sparkContext + .getLocalProperty(SparkContext.SPARK_JOB_INTERRUPT_ON_CANCEL) + + try { + sparkContext.setLocalProperty(SparkContext.SPARK_JOB_INTERRUPT_ON_CANCEL, "true") + + for (codegen <- Seq(true, false)) { + // This countdown latch used to make sure with all the stages cancelStage called in listener + val latch = new CountDownLatch(2) --- End diff -- I've just taken a look at the logs here: https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89602/artifact/sql/core/target/unit-tests.log and as I see the fixed test passed properly. ``` ===== FINISHED o.a.s.sql.DataFrameRangeSuite: 'Cancelling stage in a query with Range.' ===== ``` Hardcoding 2 is not really elegant I admit. To answer your question `SparkContext` is initialized in `TestSparkSession` this way: `SparkContext("local[2]", "test-sql-context", sparkConf.set("spark.sql.testkey", "true")`. Because of this and the fact that executors put to wait state `CountDownLatch(2)` is there. ``` sparkContext.range(0, 10000L, numSlices = slices).mapPartitions { x => x.synchronized { x.wait() } x }.toDF("id").agg(sum("id")).collect() ```
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org