Github user squito commented on the issue: https://github.com/apache/spark/pull/20888 I think you're right about killing the wrong stage, but I don't think its exactly what you've outlined. The original code doesn't try to kill a stage with ID == 0 -- instead its just waiting until that volatile is set to something > 0, and then proceeds. that seems to work fine, we do see that the stage does get canceled OK once. However, I think the problem is because the test [runs twice, with and without codegen](https://github.com/apache/spark/blob/4d37008c78d7d6b8f8a649b375ecc090700eca4f/sql/core/src/test/scala/org/apache/spark/sql/DataFrameRangeSuite.scala#L165). The first time, it'll always wait to till the stage Id is set, because of that `eventually { ... stageToKill > 0}`. however, on the second iteration, that `stageToKill` may still be > 0 based on the first iteration, not because its been set by the second iteration. So I think you just need to reset the value to -1 between iterations.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org