andygrove opened a new pull request, #4226:
URL: https://github.com/apache/datafusion-comet/pull/4226
## Which issue does this PR close?
Closes #4221.
## Rationale for this change
The Spark `StateStoreSuite.maintenance` test is intermittently flaky in CI
on the Spark 4.0.2 build, forcing repeated re-runs. Looking at the suite, all
27 tests target `StateStore.get/put/commit` directly via `SparkContext` — no
DataFrame queries, no SQL execution — so Comet does not exercise any of its
code paths and ignoring the suite under Comet does not lose meaningful coverage.
Separately, the `spark_sql_test.yml` matrix has two rows for Spark 4.0.2
(JDK 17 and JDK 21), but the job display name and fallback-log artifact name do
not include the JDK, making the two runs visually indistinguishable in the
GitHub Actions UI.
## What changes are included in this PR?
- `dev/diffs/4.0.2.diff` and `dev/diffs/4.1.1.diff`: override `test()` in
`StateStoreSuite` to reroute every test to `ignore(...)` when
`ENABLE_COMET=true`. The override lives only on `StateStoreSuite`, so
`RocksDBStateStoreSuite` (which shares the same base class) is unaffected. The
suite extends `SparkFunSuite` rather than `SQLTestUtils`, so the existing
`IgnoreCometSuite` trait could not be mixed in directly; the same logic is
inlined using `classic.SparkSession.isCometEnabled`.
- `.github/workflows/spark_sql_test.yml`: append `-jdk${{ matrix.config.java
}}` to both the matrix job display name and the fallback-log `upload-artifact`
name so the two Spark 4.0.2 rows render distinctly. The companion
`spark_sql_test_native_iceberg_compat.yml` workflow already includes the JDK in
its job name.
## How are these changes tested?
The diff regenerations were verified by resetting each Spark working tree to
its base tag, applying the regenerated diff, and confirming `git apply`
succeeds with no conflicts. CI on this PR will exercise the changes against the
full Spark SQL test suite for both JDK 17 and JDK 21 on 4.0.2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
