ErikBPF opened a new pull request, #6113:
URL: https://github.com/apache/datafusion-comet/pull/6113
Closes #6103.
## Rationale
The `iceberg-spark-extensions` job ran unsharded and was the longest job in
the
Iceberg workflow — 55m24s in a recent merge-queue run, against four shards
for the
core `iceberg-spark` task. It now uses the same shard machinery as the core
task.
## What changes
- `iceberg-spark-extensions` takes the shared shard matrix
(`fromJSON(needs.build-native.outputs.shard-matrix)`, `fail-fast: false`),
gains
the `/shard-${{ matrix.shard }}` name suffix, and passes the same Gradle
wiring as
the core task: `--init-script ../dev/ci/iceberg-test-shards.gradle`,
`-PcometShardTask=:iceberg-spark:iceberg-spark-extensions-...:test`,
`-PcometShardIndex`/`-PcometShardCount`.
- Each shard uploads its candidate inventory and test reports.
- A new `iceberg-spark-extensions-shard-coverage` job verifies the extensions
task's inventories (`check-iceberg-shards.py --manifests ... --task
...extensions...:test`),
mirroring the core coverage job.
- `dev/ci/check-ci-config.py` gains `check_iceberg_extensions_shards()` so
the job
cannot silently lose its shard wiring.
## Expected effect
~55m24s → projected ~15-20 min for the extensions job, roughly 35
queue-minutes
per run. This is the largest single saving identified in the CI review.
## How it was tested
- Guard is RED before the workflow change (four failures) and GREEN after:
`python3 dev/ci/check-ci-config.py` → `CI config checks passed`.
- `python3 dev/ci/test-iceberg-shards.py` → 15 tests OK.
- Not yet exercised end-to-end: per-shard balance and a non-empty extensions
`unshardedCandidates` inventory need one label run to confirm. The new
coverage
job fails if the union of shard candidates is not exactly the unsharded
inventory, so a bad split cannot merge silently.
## Constraint
`SHARD_COUNT = 4` in `dev/ci/check-iceberg-shards.py` is global; the
extensions
task reuses the same count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
