pingzh opened a new issue, #5490: URL: https://github.com/apache/datafusion-comet/issues/5490
Linux Spark CI currently starts from generic Rust images and installs additional build tools in each job. Repeated setup and dependency downloads add overhead and expose builds to transient network failures before tests run. See the [Spark SQL workflow](https://github.com/apache/datafusion-comet/blob/main/.github/workflows/spark_sql_test_reusable.yml) and [setup action](https://github.com/apache/datafusion-comet/blob/main/.github/actions/setup-builder/action.yaml). We could improve reliability and reduce setup time by publishing dedicated CI images containing the required toolchains and third-party dependencies. Proposed scope: - Preinstall pinned versions of Rust, JDK, clang, protoc, the Maven wrapper distribution, and required Rust components. - Prepopulate Maven dependencies/plugins and SBT’s Coursier/Ivy caches. Configure cache paths explicitly so they remain accessible under GitHub Actions’ container mounts and runtime home directory. - Start with one Linux Spark/JDK combination, then expand based on measured results. - Publish images only from trusted `main` builds. Rebuild when toolchain or dependency manifests change, plus periodically for security updates. - Pin workflows to image digests and retain the previous working image for rollback. - Exclude Comet binaries and locally built/SNAPSHOT artifacts from the image. Every PR must build its own code; artifacts can still be shared between jobs within that run. - Retain bounded download retries for new dependencies and cache misses, complementing the work in #5489. Before expanding adoption, validate that a fresh container can run a representative build without external network access once the required sources are available. Compare image-pull time, setup time, total CI duration, and download-related failures against the current workflow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
