sunchao opened a new pull request, #5489: URL: https://github.com/apache/datafusion-comet/pull/5489
## Which issue does this PR close? No issue is closed. This follows up on #5422 and addresses the dependency-download failures seen on #5481. The separate libhdfs crash is covered by #5036. ## Rationale for this change #5422 added Maven distribution caching and bootstrap retries to the Java test action. Other Maven callers still bootstrap independently, and downloading Maven itself does not protect subsequent artifact downloads. On #5481, the [Delta build gate](https://github.com/apache/datafusion-comet/actions/runs/33018848971/job/98343877451) failed downloading a Maven plugin dependency with `Connection reset`, while a [Spark 4.1 SQL shard](https://github.com/apache/datafusion-comet/actions/runs/33018849422/job/98352428975) failed resolving a Guava dependency with HTTP 502 before its tests started. ## What changes are included in this PR? - Extract the existing Maven bootstrap/cache setup into a shared action and use it across CI. Preserve the existing cache keys and macOS cache exclusion, and configure bounded Maven Resolver retries for individual HTTP transfers without rerunning Maven goals. - Add bounded retries for dependency-only commands. Retry recognized transient download errors, preserve failure exit codes, and leave compilation, tests, permanent errors, and cancellation fatal. - Resolve Spark test dependencies before compilation, restore SBT/Coursier caches, and save successful downloads before later build or test failures. Only the full build job can publish the shared cache, so a shard cannot populate it with partial dependencies. - Preserve Delta gate Maven/Cargo logs, propagate effective-POM failures, and upload the logs when the gate fails. - Add regression checks to preflight and update change detection for the shared action and scripts. ## How are these changes tested? - 18 regression tests passed, including real Maven 3.9.6 requests against a local HTTP server for transient 429/502, exhausted 503 retries, and permanent 404 failures. Additional cases cover retry limits, exit codes, interrupted backoff, dependency-only Spark tasks, and Delta compilation failures and log retention. - Workflow and composite-action validation with actionlint, YAML parsing, and Bash syntax checks passed. - Existing suite/benchmark consistency checks and Apache RAT passed. The real Maven integration checks ran locally on JDK 21; CI preflight runs them on JDK 11. Full Spark/JVM/native suites and GitHub cache restore/save behavior still require hosted CI. This PR does not change the native crash fix in #5036. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
