This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 982de1377e72 [SPARK-58018][SQL][TEST] Deflake AQE 'should avoid to
submit shuffle job on cancellation' test
982de1377e72 is described below
commit 982de1377e72ccca671fb92ddba8c64b5ab1d3a9
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Jul 8 08:58:29 2026 +0900
[SPARK-58018][SQL][TEST] Deflake AQE 'should avoid to submit shuffle job on
cancellation' test
### What changes were proposed in this pull request?
Test-only change to deflake `AdaptiveQueryExecSuite`'s `SPARK-47148: AQE
should avoid to submit shuffle job on cancellation`.
The test uses a `slow_udf` in scalar subqueries on the `df`/`df3` shuffle
stages to delay their submission, so that the `df2` coalesce stage (no
subquery) submits its shuffle job and fails with "coalesce test error" first,
while the two delayed stages are still sleeping and thus get cancelled before
submission. The test then asserts the failure chain contains "coalesce test
error" and that exactly the middle stage was materialized.
With a 3000ms sleep this ordering is timing-dependent: under a loaded CI
runner the coalesce stage's map task can be slow enough that a delayed stage is
submitted, or the coalesce stage is itself cancelled, before "coalesce test
error" is thrown — making the error disappear from the failure chain and
failing the assertion `errMsgList.exists(_.contains("coalesce test error"))`.
Increase the sleep to 15000ms so the coalesce stage reliably wins the race.
The per-test timeout is 20 minutes, so this adds no meaningful cost.
### Why are the changes needed?
The test is flaky on the `build_java21` lane (observed on branch-4.0),
failing with `...contains("coalesce test error") was false`.
### Does this PR introduce _any_ user-facing change?
No. Test-only.
### How was this patch tested?
Ran the SPARK-47148 AQE cancellation test repeatedly on GitHub Actions on a
fork.
**CI results**
- Failing (before): https://github.com/apache/spark/actions/runs/28778790780
- Passing (after, 5/5 green):
https://github.com/HyukjinKwon/spark/actions/runs/28905845288
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57096 from HyukjinKwon/ci-fix/aqe-cancellation-deflake.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit d91cf3124375f46dfc87bf451b013a79b360e2ec)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala | 11 ++++++++++-
1 file changed, 10 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
index 3e7d26f74bd4..188a28ff1c0e 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/adaptive/AdaptiveQueryExecSuite.scala
@@ -927,8 +927,17 @@ class AdaptiveQueryExecSuite
}
withUserDefinedFunction("slow_udf" -> true) {
+ // `slow_udf` sits in scalar subqueries on the `df`/`df3` shuffle stages
to delay their
+ // submission, so that the `df2` coalesce stage (which has no subquery)
submits its shuffle
+ // job and fails with "coalesce test error" first, while the two delayed
stages are still
+ // sleeping and thus get cancelled before submission. The test asserts
exactly that ordering.
+ // A short delay makes this timing-dependent: under a loaded CI runner
the coalesce stage's
+ // map task can be slow enough that a delayed stage is submitted, or the
coalesce stage is
+ // itself cancelled, before "coalesce test error" is thrown -- making
the error disappear from
+ // the failure chain and flaking the test. Use a generous sleep so the
coalesce stage reliably
+ // wins the race; the per-test timeout is 20 minutes, so this adds no
meaningful cost.
spark.udf.register("slow_udf", () => {
- Thread.sleep(3000)
+ Thread.sleep(15000)
1
})
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
