sunchao commented on PR #5459: URL: https://github.com/apache/datafusion-comet/pull/5459#issuecomment-5442375399
Updated in [913354ca0](https://github.com/apache/datafusion-comet/commit/913354ca0475770421f961f29aa78d6f1e94a2c0). Addressed the count and real-inventory validation gaps in this update. The workflow matrix and Gradle partition count now come from the same `SHARD_COUNT` definition. Neither uses `strategy.job-total`, so adding another matrix dimension cannot silently change ownership to nonexistent shard indices. Each real Iceberg worker now captures the unsharded Gradle candidate set with only our predicate temporarily disabled, restores the predicate, and records the selected set before running tests. A dependent job downloads the matching configuration's artifacts, requires every shard index, checks that the independently captured baselines agree, and verifies that the selected sets are disjoint and cover the baseline exactly once. It selects each shard's latest available attempt so failed-job reruns can retain earlier successful workers. There is no fixed release-specific class count. JUnit execution is still checked separately by the synthetic fixture, since candidate classes are not executed test cases. Validation: all 15 inventory-validator regressions pass; the updated Gradle 8.14.4/JDK 17 fixture verifies 10 candidate classes and 12 executed cases exactly once, including nested/parameterized/inherited/dynamic cases, upstream exclusions, another task's isolation, and failure propagation. Actionlint 1.7.12, Python syntax, Apache RAT, and `git diff --check` pass. The real Iceberg artifact check is wired into CI but awaits a complete hosted run; I have not represented it as a locally completed full Iceberg suite. I also made the cost estimate explicit in the description: using the previous 67 test minutes plus 11 setup/compile minutes, four workers are approximately **111 runner-minutes versus 78** per version, or **+33 minutes / +42%**, excluding the small inventory-check job. That assumes unchanged total test work and repeated setup, and is not a measured after result. Shards 2/3/4 completed in 26m54s, 28m08s, and 25m05s, but shard 1 did not finish, so those partial results cannot establish the full total or whole-CI critical-path improvement. Compilation sharing or fewer shards should be judged against a complete run. The balancing comment remains explicitly historical: it describes the Iceberg 1.11 run used to size the shards, not a current timing guarantee. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
