cloud-fan opened a new pull request, #57309: URL: https://github.com/apache/spark/pull/57309
Followup to https://github.com/apache/spark/pull/57096. ### What changes were proposed in this pull request? `AdaptiveQueryExecSuite`'s `SPARK-47148: AQE should avoid to submit shuffle job on cancellation` test registers a `slow_udf` that sleeps for 15s (raised from 3s in #57096 to deflake the test) and uses it inside scalar subqueries. Those scalar-subquery jobs on the `df`/`df3` shuffle stages are submitted asynchronously and run to completion (they are not the stages the test cancels), so each `slow_udf` invocation keeps a task busy for the full 15s while holding one of the test session's task slots (`local[2]`). Because the jobs are submitted asynchronously, a `slow_udf` task can even start running only after the test body has already returned. `QueryTest.withTempDir` runs a task drain after each test: ```scala protected override def withTempDir(f: File => Unit): Unit = { super.withTempDir { dir => f(dir) waitForTasksToFinish() // eventually(timeout(10.seconds)) { numRunningTasks == 0 } } } ``` So if another `withTempDir`-using test is scheduled within ~10s after `SPARK-47148` runs, that later test can see the leftover `slow_udf` tasks still occupying slots and fail its own drain with `1 was not equal to 0` — a failure that has nothing to do with the later test itself. This is a latent cross-test dependency: `SPARK-47148` leaks slot-holding tasks past its own scope, and the 15s sleep makes the leak long enough to outlive a sibling test's 10s drain window. This PR makes `SPARK-47148` wait for its own `slow_udf` scalar-subquery jobs to finish before returning, so they cannot bleed into a sibling test: ```scala } finally { spark.experimental.extraStrategies = Nil eventually(timeout(60.seconds), interval(500.milliseconds)) { assert(spark.sparkContext.statusTracker.getActiveJobIds().isEmpty, "slow_udf scalar-subquery jobs are still running") } } ``` It polls `getActiveJobIds` rather than `numRunningTasks`: when the `finally` runs, the subquery jobs are already *submitted* but their tasks may not have *started* yet, so a running-task count can race ahead and return early, whereas the job stays active until it completes. The 60s timeout comfortably outlasts the 15s sleep (the two subquery jobs serialize onto the 2 task slots, ~30s total). ### Why are the changes needed? `SPARK-47148` currently leaves long-running `slow_udf` tasks occupying task slots after it returns. With the 15s sleep this can flake any `withTempDir` test that happens to run shortly after it, via a task-drain timeout unrelated to that test's own logic. Confining the cleanup to the test that creates the leak removes this cross-test flakiness at its source. ### Does this PR introduce _any_ user-facing change? No. Test-only. ### How was this patch tested? `build/sbt "sql/testOnly org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite -- -z SPARK-47148"` — the test passes, taking ~30s as it now drains its own two 15s scalar-subquery jobs (per-test timeout is 20 minutes, so no meaningful cost). ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
