FelixYBW commented on issue #12288:
URL: https://github.com/apache/gluten/issues/12288#issuecomment-6374765633
Usage breakdown of one typical PR, #13410 (the daily Velox bump, head
`5aee7e272`), taken from every run on `branch:tagging-2026_10_22` across all
attempts.
### Per workflow
| Workflow | Jobs run | Job-min | Native builds | Native-build min | Jobs
installing packages at runtime | Raw OS image jobs |
|---|---|---|---|---|---|---|
| `velox_backend_x86` (3 attempts) | 49 | 1,899 | 3 | 334 | 45 | 8 |
| `velox_backend_arm` (2 attempts) | 5 | 355 | 3 (1 killed) | 338 | 4 | 1 |
| `velox_backend_enhanced` | 4 | 240 | 1 | 111 | 3 | 0 |
| `velox_kafka` | 2 | 128 | 1 | 118 | 2 | 0 |
| `delta_spark_ut` | 3 | 115 | 1 | 111 | 2 | 0 |
| `iceberg_spark_ut` | 16 | 396 | 2 | 96 | 2 | 0 |
| `check_license` | 1 | 0 | 0 | 0 | 1 | 0 |
| **Total** | **80** | **3,134** | **11** | **~1,108 (35%)** | **59** |
**9** |
### Native builds (11)
| Workflow | Job | Image | Time |
|---|---|---|---|
| x86 | `build-native-lib-centos-8` (static) | `vcpkg-almalinux-8-gcc13` |
117m |
| x86 | `cpp-test-udf-test` (shared + tests) | `centos-9-jdk8` | 140m |
| x86 | `build-cudf-centos-9` | `centos-9-jdk17-cuda13.1-cudf` | 76m |
| ARM | `build-native-lib-centos-9`, attempt 1 | `vcpkg-centos-9` | 39m,
killed (runner shutdown) |
| ARM | `build-native-lib-centos-9`, attempt 2 | `vcpkg-centos-9` | 133m |
| ARM | `cpp-test-udf-test` (shared + tests) | `centos-9-jdk8` | 166m |
| enhanced | static, enhanced features | `vcpkg-almalinux-8-gcc13` | 111m |
| kafka | static, Kafka enabled | `vcpkg-almalinux-8-gcc13` | 118m |
| delta | static (`ci-velox-buildstatic-centos-8.sh`) |
`vcpkg-almalinux-8-gcc13` | 111m |
| iceberg | head (`ci-velox-buildstatic-centos-8.sh`) |
`vcpkg-almalinux-8-gcc13` | 93m |
| iceberg | base revision | `vcpkg-almalinux-8-gcc13` | 3m |
- **Duplicate builds:** x86, Delta and Iceberg-head run the same static
build script on the same commit (117 + 111 + 93 min), so two of them (~204 min)
are duplicates. #13421 pauses enhanced and makes Kafka and Iceberg reuse the
x86 static artifact, which saves ~322 min on a PR like this one. Delta could
reuse it the same way.
- **Why native builds take ~2h:** the ccache key is
`hashFiles('ep/build-velox/src/**')`, which changes on every Velox bump PR.
Those PRs therefore miss the stash and build Velox mostly cold.
### Runtime environment setup
59 of the 80 jobs install packages at runtime (`yum`/`dnf`/`apt-get`/`pip`),
mostly on top of the pre-built `apache/gluten` images: JDK 17 into
`centos-8-jdk8`, `yum install java-17` even in `*-jdk17` images, python39 +
pyspark/pandas/pyarrow, and gh/jq/bison. 9 jobs start from raw OS images:
`centos:8` (EOL) ×4, `ubuntu:22.04` ×4, and `quay.io/centos/centos:stream9` ×1
(ARM TPC).
These runtime installs caused 3 of the 4 failures on this PR, and each
needed a rerun:
| Failure | Cause |
|---|---|
| x86 `spark-test-spark35 (2)` | EPEL 8 mirror errors during `yum install
java-17-openjdk-devel` |
| x86 `spark-test-spark35-scala213 (2)` | same |
| Delta `build-gluten-bundle` | EPEL 9 mirror timeouts during `yum install` |
| ARM `build-native-lib-centos-9` | runner shutdown signal. The rerun
rebuilt native (133m) and pushed the PR's finish out by ~24h |
### Proposed follow-ups
- [ ] Reuse the x86 static native build in Kafka/Iceberg and pause enhanced:
#13421
- [ ] Reuse the x86 static native build in `delta_spark_ut`
- [ ] Bake JDKs, Python + pyspark venvs, `procps-ng` and CI tools
(gh/jq/bison/clang-tools-extra) into the `apache/gluten` images, then remove
every runtime install step from PR jobs and replace the raw
`centos:8`/`ubuntu:22.04`/`stream9` containers with pre-built images. This
needs two PRs: Dockerfiles first, then the workflows once the images are
published.
- [ ] Improve ccache hit rate on Velox bump PRs, e.g. fall back to the
previous Velox version's stash key
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
