This is an automated email from the ASF dual-hosted git repository.
FelixYBW pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new 4e5d44684e [VL][CI] Reuse the x86 static build for Iceberg and pause
enhanced on PRs (#13421)
4e5d44684e is described below
commit 4e5d44684e1856f029026e99761abaddc5169f77
Author: BInwei Yang <[email protected]>
AuthorDate: Mon Oct 26 19:40:13 2026 -0700
[VL][CI] Reuse the x86 static build for Iceberg and pause enhanced on PRs
(#13421)
* [VL][CI] Reuse the x86 static native build for Kafka and Iceberg, pause
enhanced on PRs
- velox_backend_enhanced.yml: run on workflow_dispatch only.
- velox_kafka.yml: removed. Its Spark 3.5 job moves to
velox_backend_x86.yml on
top of build-native-lib-centos-8. Its native Kafka tests already run in
cpp-test-udf-test, whose shared build enables Kafka and runs ctest.
- iceberg_spark_ut.yml: called from velox_backend_x86.yml with the static
native
artifact for the head revision; still runnable via workflow_dispatch.
baseline.py accepts results recorded by either workflow.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
* [MINOR][VL] Share the Spark 3.5 CI change condition
Generated-by: Codex GPT-6
* [VL][CI] Keep Kafka and Iceberg CI independent
Retain independent Kafka and Spark 3.5 Iceberg workflows and their native
builds. Limit this PR to manual enhanced CI and document its availability for
urgent patches.
Generated-by: Codex GPT-6
* [VL][CI] Restore Iceberg static native build reuse
Run Iceberg alongside Spark and TPC jobs after the shared x86 static native
build. Keep Kafka independent, restore baseline provenance support, and include
Iceberg-only harness changes in detection.
Generated-by: Codex GPT-6
---------
Co-authored-by: Claude Opus 5.5 <[email protected]>
---
.github/workflows/iceberg_spark_ut.yml | 46 +++++-----------------
.github/workflows/iceberg_upstream_revision.yml | 9 ++++-
.../workflows/util/iceberg-upstream/baseline.py | 8 +++-
.github/workflows/velox_backend_enhanced.yml | 32 +++++----------
.github/workflows/velox_backend_x86.yml | 33 +++++++++++-----
docs/backends/velox/developers/velox-backend-CI.md | 27 +++++++++----
.../test/python/test_baseline.py | 5 +++
7 files changed, 83 insertions(+), 77 deletions(-)
diff --git a/.github/workflows/iceberg_spark_ut.yml
b/.github/workflows/iceberg_spark_ut.yml
index 47d63d0393..f7344d58e1 100644
--- a/.github/workflows/iceberg_spark_ut.yml
+++ b/.github/workflows/iceberg_spark_ut.yml
@@ -15,43 +15,14 @@
name: Iceberg Spark UT (Gluten)
+# Pull requests run this from velox_backend_x86.yml, which passes the static
native libraries
+# it already built for the head revision. Manual runs build them here.
on:
- pull_request:
- paths:
- - '.github/workflows/iceberg_spark_ut.yml'
- - '.github/workflows/iceberg_upstream_revision.yml'
- - '.github/workflows/velox_native_build.yml'
- - '.github/workflows/util/iceberg-upstream/**'
- - 'backends-velox/src-iceberg*/**'
- - 'gluten-iceberg/**'
- - 'gluten-ut/src-iceberg-upstream/**'
- # Shared production code and dependencies used by Iceberg on Spark 3.5.
- - 'pom.xml'
- - 'backends-velox/pom.xml'
- - 'backends-velox/src/main/**'
- - 'gluten-core/pom.xml'
- - 'gluten-core/src/main/**'
- - 'gluten-substrait/pom.xml'
- - 'gluten-substrait/src/main/**'
- - 'gluten-arrow/pom.xml'
- - 'gluten-arrow/src/main/**'
- - 'shims/pom.xml'
- - 'shims/common/pom.xml'
- - 'shims/common/src/main/**'
- - 'shims/spark35/pom.xml'
- - 'shims/spark35/src/main/**'
- - 'build/mvn'
- - 'dev/gluten-build-info.sh'
- # Native code and the build inputs used by velox_native_build.yml.
- - 'cpp/**'
- - 'ep/build-velox/**'
- - 'dev/ci-velox-buildstatic-centos-8.sh'
- - 'dev/builddeps-veloxbe.sh'
- - 'dev/build-helper-functions.sh'
- - 'dev/vcpkg/**'
- # Documentation-only changes do not affect this runtime.
- - '!**/*.md'
- - '!**/*.rst'
+ workflow_call:
+ inputs:
+ native_artifact:
+ type: string
+ default: ''
workflow_dispatch:
permissions:
@@ -73,7 +44,7 @@ env:
# Five class shards avoid repeating class fixtures on multiple runners.
# Keep the default job timeout so cold builds can finish; measure elapsed time
separately.
-# Independent entry point keeps these shards outside the main backend
workflow's concurrency.
+# The head revision reuses the static native build of velox_backend_x86.yml
when called from it.
jobs:
prepare:
runs-on: ubuntu-22.04
@@ -134,6 +105,7 @@ jobs:
label: head
native_image: ${{ needs.prepare.outputs.native_image }}
test_image: ${{ needs.prepare.outputs.test_image }}
+ native_artifact: ${{ inputs.native_artifact }}
iceberg-base-test:
needs: prepare
diff --git a/.github/workflows/iceberg_upstream_revision.yml
b/.github/workflows/iceberg_upstream_revision.yml
index e0a0bb1b97..ca71e0efab 100644
--- a/.github/workflows/iceberg_upstream_revision.yml
+++ b/.github/workflows/iceberg_upstream_revision.yml
@@ -30,6 +30,11 @@ on:
test_image:
required: true
type: string
+ native_artifact:
+ # Native libraries already built for this revision in the calling run.
When empty,
+ # they are built here.
+ type: string
+ default: ''
env:
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
@@ -41,6 +46,7 @@ env:
jobs:
native:
+ if: inputs.native_artifact == ''
uses: ./.github/workflows/velox_native_build.yml
with:
revision: ${{ inputs.revision }}
@@ -49,6 +55,7 @@ jobs:
build:
needs: native
+ if: ${{ !cancelled() && (needs.native.result == 'success' ||
needs.native.result == 'skipped') }}
runs-on: ubuntu-22.04
container: ${{ inputs.test_image }}
env:
@@ -73,7 +80,7 @@ jobs:
- name: Download native libraries
uses: actions/download-artifact@v8
with:
- name: iceberg-native-${{ inputs.label }}
+ name: ${{ inputs.native_artifact || format('iceberg-native-{0}',
inputs.label) }}
path: cpp/build/releases/
- name: Build Gluten once for all shards
shell: bash
diff --git a/.github/workflows/util/iceberg-upstream/baseline.py
b/.github/workflows/util/iceberg-upstream/baseline.py
index 804ab1a05a..b17080c1e4 100644
--- a/.github/workflows/util/iceberg-upstream/baseline.py
+++ b/.github/workflows/util/iceberg-upstream/baseline.py
@@ -36,6 +36,12 @@ from summarize import write_reports
NATIVE_IMAGE = "apache/gluten:vcpkg-almalinux-8-gcc13"
TEST_IMAGE = "apache/gluten:centos-9-jdk17"
+# Pull requests run the Iceberg workflow from the x86 backend workflow; manual
runs and
+# results saved before that change come from the Iceberg workflow itself.
+WORKFLOW_PATHS = (
+ ".github/workflows/iceberg_spark_ut.yml",
+ ".github/workflows/velox_backend_x86.yml",
+)
HARNESS_PATHS = (
".github/workflows/iceberg_spark_ut.yml",
".github/workflows/iceberg_upstream_revision.yml",
@@ -242,7 +248,7 @@ def verify_source(api, artifact, identity, key, tree,
branch):
run = api.request(f"actions/runs/{run_id}/attempts/{attempt}")
if (
run["repository"]["full_name"] != api.repository
- or run["path"] != ".github/workflows/iceberg_spark_ut.yml"
+ or run["path"] not in WORKFLOW_PATHS
or run["status"] != "completed"
or run["conclusion"] not in ("success", "failure")
):
diff --git a/.github/workflows/velox_backend_enhanced.yml
b/.github/workflows/velox_backend_enhanced.yml
index 6520430bf8..00d5da022b 100644
--- a/.github/workflows/velox_backend_enhanced.yml
+++ b/.github/workflows/velox_backend_enhanced.yml
@@ -15,29 +15,10 @@
name: Velox Backend (Enhanced)
+# No pending patches remain in the enhanced branch. Pause automatic PR runs to
save CI
+# capacity, and keep manual runs available for urgent patches when needed.
on:
- pull_request:
- paths:
- - '.github/workflows/velox_backend_enhanced.yml'
- - '.github/workflows/util/parallel-ut/**'
- - 'pom.xml'
- - 'backends-velox/**'
- - 'gluten-uniffle/**'
- - 'gluten-celeborn/**'
- - 'gluten-core/**'
- - 'gluten-substrait/**'
- - 'gluten-arrow/**'
- - 'gluten-delta/**'
- - 'gluten-iceberg/**'
- - 'gluten-hudi/**'
- - 'gluten-ut/**'
- - 'package/**'
- - 'shims/**'
- - 'tools/gluten-it/**'
- - 'ep/build-velox/**'
- - 'cpp/**'
- - 'dev/**'
- - 'build/mvn'
+ workflow_dispatch:
env:
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
@@ -75,6 +56,13 @@ jobs:
- name: Detect changed paths
id: filter
run: |
+ if [ "${{ github.event_name }}" != "pull_request" ]; then
+ # No PR diff to inspect on manual runs: run everything.
+ for flag in cpp java shims35 shims40; do
+ echo "$flag=true" >> $GITHUB_OUTPUT
+ done
+ exit 0
+ fi
BASE=${{ github.event.pull_request.base.sha }}
HEAD=${{ github.sha }}
changed=$(git diff --name-only "$BASE" "$HEAD")
diff --git a/.github/workflows/velox_backend_x86.yml
b/.github/workflows/velox_backend_x86.yml
index 40ca515c0c..ed20516c7c 100644
--- a/.github/workflows/velox_backend_x86.yml
+++ b/.github/workflows/velox_backend_x86.yml
@@ -20,6 +20,10 @@ on:
paths:
- '.github/workflows/velox_backend_x86.yml'
- '.github/workflows/util/parallel-ut/**'
+ - '.github/workflows/iceberg_spark_ut.yml'
+ - '.github/workflows/iceberg_upstream_revision.yml'
+ - '.github/workflows/velox_native_build.yml'
+ - '.github/workflows/util/iceberg-upstream/**'
- 'pom.xml'
- 'backends-velox/**'
- 'gluten-uniffle/**'
@@ -107,6 +111,10 @@ jobs:
shims41: ${{ steps.filter.outputs.shims41 }}
shims42: ${{ steps.filter.outputs.shims42 }}
tools_it: ${{ steps.filter.outputs.tools_it }}
+ spark35_needed: >-
+ ${{ steps.filter.outputs.java == 'true' ||
+ steps.filter.outputs.shims35 == 'true' ||
+ steps.filter.outputs.cpp == 'true' }}
steps:
- uses: actions/checkout@v7
with:
@@ -146,7 +154,7 @@ jobs:
match() { echo "$changed" | grep -qE "$1" && echo true || echo
false; }
echo "cpp=$(match '^(cpp/|ep/build-velox/|dev/)')"
>> $GITHUB_OUTPUT
- echo "java=$(match
'^(\.github/workflows/|pom\.xml|backends-velox/|gluten-(uniffle|celeborn|ras|core|substrait|arrow|delta|iceberg|hudi|paimon)/|gluten-ut/(common/|test/|pom\.xml)|package/|build/mvn)')"
>> $GITHUB_OUTPUT
+ echo "java=$(match
'^(\.github/workflows/|pom\.xml|backends-velox/|gluten-(uniffle|celeborn|ras|core|substrait|arrow|delta|iceberg|hudi|paimon)/|gluten-ut/(common/|test/|src-iceberg-upstream/|pom\.xml)|package/|build/mvn)')"
>> $GITHUB_OUTPUT
echo "shims34=$(match
'^(shims/(spark34|common)/|gluten-ut/spark34/)')" >>
$GITHUB_OUTPUT
echo "shims35=$(match
'^(shims/(spark35|common)/|gluten-ut/spark35/)')" >>
$GITHUB_OUTPUT
echo "shims40=$(match
'^(shims/(spark40|common)/|gluten-ut/spark40/)')" >>
$GITHUB_OUTPUT
@@ -865,10 +873,7 @@ jobs:
spark-test-spark35:
needs: [detect-changes, build-native-lib-centos-8]
- if: >-
- needs.detect-changes.outputs.java == 'true' ||
- needs.detect-changes.outputs.shims35 == 'true' ||
- needs.detect-changes.outputs.cpp == 'true'
+ if: needs.detect-changes.outputs.spark35_needed == 'true'
runs-on: ubuntu-22.04
strategy:
fail-fast: false
@@ -1004,10 +1009,7 @@ jobs:
spark-test-spark35-slow:
needs: [detect-changes, build-native-lib-centos-8]
- if: >-
- needs.detect-changes.outputs.java == 'true' ||
- needs.detect-changes.outputs.shims35 == 'true' ||
- needs.detect-changes.outputs.cpp == 'true'
+ if: needs.detect-changes.outputs.spark35_needed == 'true'
runs-on: ubuntu-22.04
strategy:
fail-fast: false
@@ -1181,6 +1183,19 @@ jobs:
name: ${{ github.job }}-test-log
path: "**/target/*.log"
+ # Reuse the head revision's static native libraries. Iceberg runs alongside
the
+ # Spark/TPC jobs once the shared build completes, without waiting for their
results.
+ iceberg-spark-ut:
+ needs: [detect-changes, build-native-lib-centos-8]
+ if: needs.detect-changes.outputs.spark35_needed == 'true'
+ permissions:
+ contents: read
+ actions: read
+ pull-requests: read
+ uses: ./.github/workflows/iceberg_spark_ut.yml
+ with:
+ native_artifact: velox-native-lib-centos-8-${{github.sha}}
+
build-cudf-centos-9:
needs: detect-changes
if: needs.detect-changes.outputs.cpp == 'true'
diff --git a/docs/backends/velox/developers/velox-backend-CI.md
b/docs/backends/velox/developers/velox-backend-CI.md
index 36f16a37c6..b4882e175f 100644
--- a/docs/backends/velox/developers/velox-backend-CI.md
+++ b/docs/backends/velox/developers/velox-backend-CI.md
@@ -57,15 +57,28 @@ See
[.github/workflows/util/delta-spark-ut/README.md](https://github.com/apache/
for the gate, the flaky-test quarantine and baseline bootstrapping.
Open follow-ups are tracked in
[#12743](https://github.com/apache/gluten/issues/12743).
-## Iceberg Spark UT
+## Enhanced Backend CI
+
+`velox_backend_enhanced.yml` runs only through `workflow_dispatch`. There are
no pending
+patches in the enhanced branch, so automatic pull-request runs are paused to
save CI
+capacity. The workflow remains available for urgent patches and manual
validation; manual
+runs enable all native and Spark test jobs.
-`iceberg_spark_ut.yml` runs upstream Iceberg query/write tests on Spark 3.5.
PR runs are
-limited to Iceberg sources and the harness, native/Velox code, shared
production code in
-the Velox backend, core, Substrait and Arrow modules, Spark 3.5/common shims,
and their
-Maven/native-build inputs.
+Kafka CI keeps its independent pull-request workflow and Kafka-enabled native
build.
+
+## Iceberg Spark UT
-`iceberg_upstream_revision.yml` builds each revision's native and JVM
artifacts once and
-shares them across five class shards, with one Spark fork per runner.
+`iceberg_spark_ut.yml` runs upstream Iceberg query/write tests on Spark 3.5.
On pull requests
+and nightly runs, `velox_backend_x86.yml` calls it after
`build-native-lib-centos-8` and passes
+the head revision's static native libraries. It runs for Java, Spark
3.5/common shim or native
+changes. Once the shared native artifact is ready, Iceberg runs alongside the
Spark and TPC
+jobs; it does not wait for those jobs to finish. Its five class shards also
run in parallel,
+with one Spark fork per runner.
+
+`iceberg_upstream_revision.yml` skips the head native build when an artifact
is supplied,
+then builds the JVM artifacts once for all shards. Manual `workflow_dispatch`
runs build
+native libraries themselves. The base revision also builds its own native
libraries when
+no saved baseline is available, because it tests a different commit.
### Results and gate
diff --git a/gluten-ut/src-iceberg-upstream/test/python/test_baseline.py
b/gluten-ut/src-iceberg-upstream/test/python/test_baseline.py
index cb75fdcc20..cf5032b429 100644
--- a/gluten-ut/src-iceberg-upstream/test/python/test_baseline.py
+++ b/gluten-ut/src-iceberg-upstream/test/python/test_baseline.py
@@ -340,6 +340,11 @@ class BaselineTest(unittest.TestCase):
):
source.verify()
+ def test_result_from_the_x86_backend_workflow_is_a_baseline(self):
+ source = Source()
+ source.run["path"] = ".github/workflows/velox_backend_x86.yml"
+ source.verify()
+
def
test_manual_main_result_can_seed_baseline_without_a_passing_comparison(self):
source = Source()
source.run.update(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
