sunchao opened a new pull request, #5976: URL: https://github.com/apache/datafusion-comet/pull/5976
## Which issue does this PR close? Part of #5830. Complements #5973 (large-cache write policy and Maven cache sharing) and #5841 (sharing a native build within a workflow run). ## Rationale for this change A JVM-only PR still waits for Cargo to produce the same native library. In [run 34971852034](https://github.com/apache/datafusion-comet/actions/runs/34971852034), the native job took 24m42s, including 23m28s of compilation, and produced a roughly 66 MiB library. Its logs also show `CARGO_HOME=/usr/local/cargo`, while the workflows cache `~/.cargo/registry` and `~/.cargo/git`. Reuse a compact native library when its build inputs match a successful main build. Correct the Cargo cache paths and snapshot keys before compilation, so generated Rust files cannot change the key between restore and save. The historical compile duration shows the opportunity; this PR does not yet establish a measured cache-hit speedup. ## What changes are included in this PR? - Add a shared Linux native-build action used by Linux PR CI, reusable Spark SQL/Iceberg workflows, and manual writer tests. It stages the same `native/target/ci/libcomet.so` and preserves downstream artifacts and tests. - Key the compact library on tracked native/protobuf/build inputs, resolved compiler tools, JDK headers/library, package versions, Cargo configuration, profile, architecture, and supported build environment. Use the existing portable `x86-64-v3` flags. Generated files and unrelated Spark edits do not invalidate it; CI definition changes conservatively do. Unsupported build overrides fail before producing a key. - Require an exact cache-service hit plus a matching manifest and SHA-256 before skipping Cargo. Missing or invalid entries run `cargo build --locked --profile ci`; an incremental target-cache hit alone never skips compilation. - Let only pushes to main publish these caches. Main pushes always compile to keep the incremental cache warm, even when a compact library already exists. Compact entries have no prefix fallback. - Cache the effective Cargo home and reuse the same pre-build key for restore/save in both CI and debug builds. Remove the unused Cargo registry cache from JVM workers, which explicitly skip native compilation. Maven-cache consolidation remains in #5973. - Add CI routing, preflight regression checks, and operational documentation. ## How are these changes tested? - 31 focused tests pass: 13 build-identity tests, 13 library validation/staging tests, and 5 tests of the actual action conditions and shell blocks. Coverage includes changed inputs, generated-file stability, malformed/partial/corrupt entries, locked-build fallback, failed builds, and main-only publication. Cargo and remote cache operations are stubbed in the workflow tests. - In the public `amd64/rust:latest` container with a real JDK 17 and required tools, verified stable keys across repeated calls, generated Rust files, and an unrelated Spark edit; a protobuf edit invalidated source/library keys. Verified `/usr/local/cargo` is the reported cache home. All 31 tests also pass there. - CI configuration/routing, suite inventory, benchmark-runner, Iceberg-runner, and PR-label checks pass, along with actionlint, README formatting, and whitespace checks. Full native/Spark execution and the GitHub cache-service path require hosted CI. The new namespace starts cold; cross-run library reuse can be measured after a successful main push populates it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
