This is an automated email from the ASF dual-hosted git repository.
HyukjinKwon pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 3011bda97927 [SPARK-57654][SQL][TEST] Deflake
MetricsFailureInjectionSuite 'Force checksum mismatch aborts a downstream
ResultStage'
3011bda97927 is described below
commit 3011bda97927885cc728ea61d52a9af33801a59e
Author: Hyukjin Kwon <[email protected]>
AuthorDate: Wed Jun 24 19:42:20 2026 +0900
[SPARK-57654][SQL][TEST] Deflake MetricsFailureInjectionSuite 'Force
checksum mismatch aborts a downstream ResultStage'
### What changes were proposed in this pull request?
Group the abort-path test `MetricsFailureInjectionSuite."Force checksum
mismatch aborts a downstream ResultStage"` by the high-cardinality `id` column
instead of the 5-value `low_cardinality_col`, so every one of the 20 reducer
partitions reads the corrupted mapper-0.
### Why are the changes needed?
The test was flaky under Maven (~3/10 scheduled runs; it always passed on
SBT). Only ~5 of the 20 reducer partitions held mapper-0's few low-cardinality
keys, and the mapper-0 corruption is applied **asynchronously** after the first
result task succeeds (`RESULT_STAGE_DELAY=1`). The indeterminate-stage abort
therefore only fired if one of those few partitions happened to be scheduled
*after* the corruption landed — a scheduling race. Grouping by the
high-cardinality `id` makes every r [...]
### Does this PR introduce any user-facing change?
No, test only.
### How was this patch tested?
Ran the suite **20×** under Maven (the environment where it flaked) on a
fork — all 20 passed.
- ❌ Before (flaky, scheduled `Build / Maven (Scala 2.13, JDK 21)`):
https://github.com/apache/spark/actions/runs/28035705490
- ❌ Before (flaky, scheduled `Build / Maven (Scala 2.13, JDK 25)`):
https://github.com/apache/spark/actions/runs/28035606804
- ✅ After (this fix, MetricsFailureInjectionSuite ×20 under Maven, all
green): https://github.com/HyukjinKwon/spark/actions/runs/28066715792
### Was this patch authored or co-authored using generative AI tooling?
Yes, Generated-by: Claude Code
This pull request and its description were written by Isaac.
Closes #56724 from HyukjinKwon/SPARK-57654.
Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
(cherry picked from commit 0ba8d2afbb758f37dd4d8446ca055d596dd4cab9)
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../sql/execution/metric/MetricsFailureInjectionSuite.scala | 12 +++++++++++-
1 file changed, 11 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala
index b704628b13eb..005734d14713 100644
---
a/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala
+++
b/sql/core/src/test/scala/org/apache/spark/sql/execution/metric/MetricsFailureInjectionSuite.scala
@@ -630,6 +630,16 @@ class MetricsFailureInjectionSuite
// OSS Spark cannot roll back a partially-finished result stage, so the
job aborts. With
// the default RESULT_STAGE_DELAY=0 the result stage is corrupted before
any task
// dispatches and the rollback path does not abort.
+ //
+ // We group by the high-cardinality `id` column (not
`low_cardinality_col`) so that every
+ // one of the 20 reducer partitions reads data from the corrupted mapper
0. Otherwise only
+ // the handful of reducer partitions that happen to hold mapper-0's few
low-cardinality keys
+ // would observe the FetchFailed, and the abort would only fire when one
of those specific
+ // partitions happened to be scheduled after the (asynchronous) corruption
-- a scheduling
+ // race that made this test flaky under Maven. With `id`, every partition
depends on mapper
+ // 0, so once RESULT_STAGE_DELAY=1 has corrupted it (after the first
result task), local[2]
+ // dispatches the remaining result tasks afterwards and at least one is
guaranteed to hit
+ // the corrupted mapper, deterministically triggering the
indeterminate-stage abort.
withTable("test_table") {
setUpTestTable("test_table")
withSQLConf(SQLConf.SHUFFLE_PARTITIONS.key -> "20") {
@@ -638,7 +648,7 @@ class MetricsFailureInjectionSuite
config.Tests.INJECT_SHUFFLE_FETCH_FAILURES_RESULT_STAGE_DELAY.key
-> "1",
config.Tests.INJECT_SHUFFLE_FORCE_CHECKSUM_MISMATCH_ON_RECOMPUTE.key -> "true")
{
val df = spark.read.table("test_table")
- .groupBy("low_cardinality_col")
+ .groupBy("id")
.count()
val ex = intercept[SparkException] {
df.collect()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
