This is an automated email from the ASF dual-hosted git repository.
LuciferYang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/gluten.git
The following commit(s) were added to refs/heads/main by this push:
new f6e4123d29 [GLUTEN-13541][VL][CI] Run the Velox x86, ARM and Kafka
pipelines in contributors' forks (#13535)
f6e4123d29 is described below
commit f6e4123d29a801620520122365e0ebfa8397214d
Author: YangJie <[email protected]>
AuthorDate: Fri Nov 13 10:33:58 2026 -0500
[GLUTEN-13541][VL][CI] Run the Velox x86, ARM and Kafka pipelines in
contributors' forks (#13535)
## What changes are proposed in this pull request?
Fork PR runs of the Velox x86, ARM and Kafka workflows used 223,000 of
apache/gluten's 292,000 runner-minutes in the week of 2026-10-27, over the
ASF's 250,000 weekly limit. This PR runs them in the contributor's fork on
push, as Spark does.
- A fork run tests the branch squash-merged onto the current `main`. Fork
PRs to `main` no longer run these workflows here.
- Fork builds restore apache/gluten's `main` ccache. With `CCACHE_BASEDIR`,
forks with other names hit it too.
- Fork runs of the Iceberg UT read baselines from apache/gluten.
- `update_fork_build_status.yml` mirrors each fork run as a PR check every
5 minutes, without `pull_request_target`. A check fails on its run's first
failed job and lists failed jobs.
Open fork PRs run again once rebased onto `main` and pushed. Fork PRs won't
get #13611's "Report test results" check, since fork runs don't trigger
`workflow_run`. Contributor steps are in `CONTRIBUTING.md`, background and
follow-ups in #13541.
## How was this patch tested?
Fork runs of the latest commit passed:
[x86](https://github.com/LuciferYang/gluten/actions/runs/43160683425),
[ARM](https://github.com/LuciferYang/gluten/actions/runs/43160683016),
[Kafka](https://github.com/LuciferYang/gluten/actions/runs/43160683029). The
Velox jobs on this PR show as skipped. After merge, fork PRs still list those
skipped jobs, plus `... in fork` checks with their fork runs' results. Those
checks can't show up before merge, because `update_fork_build_status.yml` ru
[...]
## Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: Claude Opus 5.5
---
.github/actions/restore-upstream-ccache/action.yml | 88 +++++++
.github/actions/squash-merge-upstream/action.yml | 81 +++++++
.github/workflows/iceberg_spark_ut.yml | 21 +-
.github/workflows/iceberg_upstream_revision.yml | 8 +-
.github/workflows/update_fork_build_status.yml | 52 +++++
.../workflows/util/iceberg-upstream/baseline.py | 5 +-
.github/workflows/util/update-fork-build-status.js | 252 +++++++++++++++++++++
.github/workflows/velox_backend_arm.yml | 95 +++++++-
.github/workflows/velox_backend_x86.yml | 156 ++++++++++++-
.github/workflows/velox_kafka.yml | 65 +++++-
CONTRIBUTING.md | 11 +
dev/ci-velox-buildshared-centos-9.sh | 4 +
...velox-buildstatic-centos-8-enhanced-features.sh | 4 +
dev/ci-velox-buildstatic-centos-8.sh | 4 +
dev/ci-velox-buildstatic-centos-9.sh | 4 +
15 files changed, 824 insertions(+), 26 deletions(-)
diff --git a/.github/actions/restore-upstream-ccache/action.yml
b/.github/actions/restore-upstream-ccache/action.yml
new file mode 100644
index 0000000000..ddd025f990
--- /dev/null
+++ b/.github/actions/restore-upstream-ccache/action.yml
@@ -0,0 +1,88 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Restore ccache from apache/gluten main
+description: >-
+ Download the ccache stash that apache/infrastructure-actions/stash/save
stored
+ for `key` on apache/gluten main. stash/restore only searches the repository
the
+ run belongs to, so a fork run uses this after missing its own stash. Without
a
+ stash for `key`, the newest stash for the same key prefix is used, as
+ actions/cache restore-keys would.
+inputs:
+ key:
+ description: The key the stash was saved with, ending in a hashFiles()
hash.
+ required: true
+ path:
+ description: The directory to restore into.
+ required: true
+outputs:
+ hit:
+ description: "'true' if a stash was downloaded."
+ value: ${{ steps.download.outcome == 'success' }}
+runs:
+ using: composite
+ steps:
+ - name: Find the stash
+ id: find
+ uses: actions/github-script@v9
+ env:
+ KEY: ${{ inputs.key }}
+ with:
+ script: |
+ const repo = { owner: 'apache', repo: 'gluten' }
+ // stash/save appends the branch name to the key. Only stashes saved
by
+ // apache/gluten's own runs on main count.
+ const usable = (a) => !a.expired && a.name.endsWith('-main')
+ && a.workflow_run.head_branch == 'main'
+ && a.workflow_run.head_repository_id ==
a.workflow_run.repository_id
+ const exact = process.env.KEY + '-main'
+ const { data } = await github.rest.actions.listArtifactsForRepo({
+ ...repo, name: exact, per_page: 10
+ })
+ let stash = data.artifacts.find(usable)
+ if (!stash) {
+ // The key changed in this branch, or main changed it and the cache
+ // workflow hasn't saved the new stash yet: take the newest one it
saved.
+ const prefix = process.env.KEY.replace(/[0-9a-f]{64}$/, '')
+ const { data: runs } = await github.rest.actions.listWorkflowRuns({
+ ...repo, workflow_id: 'velox_backend_cache.yml', branch: 'main',
+ status: 'success', per_page: 5
+ })
+ for (const run of runs.workflow_runs) {
+ const { data: arts } = await
github.rest.actions.listWorkflowRunArtifacts({
+ ...repo, run_id: run.id, per_page: 100
+ })
+ stash = arts.artifacts.find(a => usable(a) &&
a.name.startsWith(prefix))
+ if (stash) break
+ }
+ }
+ if (stash) {
+ console.log('Restoring ' + stash.name + ' from apache/gluten run '
+ stash.workflow_run.id)
+ core.setOutput('name', stash.name)
+ core.setOutput('run_id', stash.workflow_run.id)
+ } else {
+ console.log('No stash for ' + exact + ' on apache/gluten main')
+ }
+ - name: Download the stash
+ id: download
+ if: steps.find.outputs.run_id != ''
+ continue-on-error: true
+ uses: actions/download-artifact@v8
+ with:
+ name: ${{ steps.find.outputs.name }}
+ repository: apache/gluten
+ run-id: ${{ steps.find.outputs.run_id }}
+ github-token: ${{ github.token }}
+ path: ${{ inputs.path }}
diff --git a/.github/actions/squash-merge-upstream/action.yml
b/.github/actions/squash-merge-upstream/action.yml
new file mode 100644
index 0000000000..ccdf39b24a
--- /dev/null
+++ b/.github/actions/squash-merge-upstream/action.yml
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Squash-merge onto apache/gluten main
+description: >-
+ In a fork, squash-merge the commit under test onto the current apache/gluten
+ main, which is what a pull request merge ref would hold. If anything in
+ `paths` changed, push the result to a refs/gluten-ci/ ref in the fork, so the
+ workflow's other jobs can check it out by commit. Use it after
+ actions/checkout with fetch-depth 0, in a job with `contents: write`.
+inputs:
+ paths:
+ description: >-
+ Extended regex for the files the workflow covers. A `paths:` filter on
push
+ diffs against the branch's previous push, not against main, so fork runs
+ apply the list here.
+ required: true
+ paths-ignore:
+ description: Extended regex for files to leave out before matching `paths`.
+ default: '^$'
+outputs:
+ base_sha:
+ description: The apache/gluten main commit the branch was merged onto.
+ value: ${{ steps.merge.outputs.base_sha }}
+ revision:
+ description: The merged commit.
+ value: ${{ steps.merge.outputs.revision }}
+ relevant:
+ description: "'false' if nothing in `paths` changed."
+ value: ${{ steps.merge.outputs.relevant }}
+runs:
+ using: composite
+ steps:
+ - name: Squash-merge the commit under test
+ id: merge
+ shell: bash
+ env:
+ PATHS: ${{ inputs.paths }}
+ PATHS_IGNORE: ${{ inputs.paths-ignore }}
+ run: |
+ git config user.name 'github-actions[bot]'
+ git config user.email
'41898282+github-actions[bot]@users.noreply.github.com'
+ git fetch --no-tags https://github.com/apache/gluten.git main
+ base_sha=$(git rev-parse FETCH_HEAD)
+ git checkout --quiet --detach "$base_sha"
+ if ! git merge --squash --no-commit "$GITHUB_SHA"; then
+ echo "::error::$GITHUB_REF_NAME does not merge cleanly into
apache/gluten main ($base_sha). Rebase it and push again."
+ exit 1
+ fi
+ git commit --quiet --allow-empty -m "Squash-merge $GITHUB_SHA onto
$base_sha"
+ echo "base_sha=$base_sha" >> "$GITHUB_OUTPUT"
+ echo "revision=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+ # Not `grep -q`: it can exit before the upstream grep is done writing,
and under
+ # pipefail the SIGPIPE would read as no match.
+ if git diff --name-only "$base_sha" HEAD | grep -vE "$PATHS_IGNORE" |
grep -E "$PATHS" > /dev/null; then
+ echo "relevant=true" >> "$GITHUB_OUTPUT"
+ else
+ echo "Nothing this workflow covers changed -- skipping."
+ echo "relevant=false" >> "$GITHUB_OUTPUT"
+ fi
+ # One ref per workflow and branch, overwritten by the next run. Slashes in
the
+ # branch name are replaced so that branches `a` and `a/b` don't collide.
+ - name: Publish the merged commit in the fork
+ if: steps.merge.outputs.relevant == 'true'
+ shell: bash
+ run: |
+ workflow=${GITHUB_WORKFLOW_REF%%@*}
+ workflow=${workflow##*/}
+ git push --force origin
"HEAD:refs/gluten-ci/${workflow%.*}/${GITHUB_REF_NAME//\//-}"
diff --git a/.github/workflows/iceberg_spark_ut.yml
b/.github/workflows/iceberg_spark_ut.yml
index 6f8c8ec1eb..b3f4a25cee 100644
--- a/.github/workflows/iceberg_spark_ut.yml
+++ b/.github/workflows/iceberg_spark_ut.yml
@@ -23,6 +23,14 @@ on:
native_artifact:
type: string
default: ''
+ # A fork push passes the commit it squash-merged onto apache/gluten
main, and
+ # that main commit. Empty: the event's commit and pull request base.
+ revision:
+ type: string
+ default: ''
+ base_revision:
+ type: string
+ default: ''
workflow_dispatch:
permissions:
@@ -60,11 +68,12 @@ jobs:
steps:
- uses: actions/checkout@v7
with:
+ ref: ${{ inputs.revision }}
fetch-depth: 0
- name: Identify the merged baseline and runtimes
id: baseline
env:
- BASE_SHA: ${{ github.event.pull_request.base.sha }}
+ BASE_SHA: ${{ inputs.base_revision ||
github.event.pull_request.base.sha }}
run: |
python3 .github/workflows/util/iceberg-upstream/baseline.py prepare \
--revision "${BASE_SHA:-$(git rev-parse HEAD^)}" >>
"$GITHUB_OUTPUT"
@@ -77,6 +86,8 @@ jobs:
continue-on-error: true
env:
GH_TOKEN: ${{ github.token }}
+ # Baselines are saved by apache/gluten runs; a fork reads them from
there.
+ ICEBERG_BASELINE_REPOSITORY: apache/gluten
BASELINE_KEY: ${{ steps.baseline.outputs.baseline_key }}
BASE_SHA: ${{ steps.baseline.outputs.base_sha }}
BASE_BRANCH: ${{ github.event.pull_request.base.ref ||
github.event.repository.default_branch }}
@@ -101,7 +112,8 @@ jobs:
needs: prepare
uses: ./.github/workflows/iceberg_upstream_revision.yml
with:
- revision: ${{ github.sha }}
+ revision: ${{ inputs.revision || github.sha }}
+ harness_revision: ${{ inputs.revision }}
label: head
native_image: ${{ needs.prepare.outputs.native_image }}
test_image: ${{ needs.prepare.outputs.test_image }}
@@ -113,6 +125,7 @@ jobs:
uses: ./.github/workflows/iceberg_upstream_revision.yml
with:
revision: ${{ needs.prepare.outputs.base_sha }}
+ harness_revision: ${{ inputs.revision }}
label: base
native_image: ${{ needs.prepare.outputs.native_image }}
test_image: ${{ needs.prepare.outputs.test_image }}
@@ -125,6 +138,7 @@ jobs:
steps:
- uses: actions/checkout@v7
with:
+ ref: ${{ inputs.revision }}
fetch-depth: 0
- name: Require complete test runs
env:
@@ -153,9 +167,10 @@ jobs:
if: always() && steps.head.outcome == 'success' &&
needs.iceberg-upstream-test.result == 'success'
env:
HEAD_KEY: ${{ needs.prepare.outputs.head_key }}
+ HEAD_SHA: ${{ inputs.revision || github.sha }}
run: |
python3 .github/workflows/util/iceberg-upstream/baseline.py record \
- --directory iceberg-head --key "$HEAD_KEY" --revision "$GITHUB_SHA"
+ --directory iceberg-head --key "$HEAD_KEY" --revision "$HEAD_SHA"
- name: Save the result for reuse after merge
if: always() && steps.head-identity.outcome == 'success'
uses: actions/upload-artifact@v7
diff --git a/.github/workflows/iceberg_upstream_revision.yml
b/.github/workflows/iceberg_upstream_revision.yml
index 4dc933ab52..c82c7d626e 100644
--- a/.github/workflows/iceberg_upstream_revision.yml
+++ b/.github/workflows/iceberg_upstream_revision.yml
@@ -30,6 +30,10 @@ on:
test_image:
required: true
type: string
+ # The commit holding the comparison harness. Empty: the event's commit.
+ harness_revision:
+ type: string
+ default: ''
native_artifact:
# Native libraries already built for this revision in the calling run.
When empty,
# they are built here.
@@ -68,7 +72,7 @@ jobs:
if: inputs.label == 'base'
uses: actions/checkout@v7
with:
- ref: ${{ github.sha }}
+ ref: ${{ inputs.harness_revision || github.sha }}
path: iceberg-harness
- name: Apply the current harness to the base production revision
if: inputs.label == 'base'
@@ -121,7 +125,7 @@ jobs:
if: inputs.label == 'base'
uses: actions/checkout@v7
with:
- ref: ${{ github.sha }}
+ ref: ${{ inputs.harness_revision || github.sha }}
path: iceberg-harness
- name: Apply the current harness to the base production revision
if: inputs.label == 'base'
diff --git a/.github/workflows/update_fork_build_status.yml
b/.github/workflows/update_fork_build_status.yml
new file mode 100644
index 0000000000..a1f13a31cb
--- /dev/null
+++ b/.github/workflows/update_fork_build_status.yml
@@ -0,0 +1,52 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# A pull request from a fork runs velox_backend_x86.yml, velox_backend_arm.yml
and
+# velox_kafka.yml in the fork, on push. This workflow reports those runs as
checks on
+# the pull request, using util/update-fork-build-status.js. It only reads run
metadata,
+# so it never runs pull request code and needs no pull_request_target. Adapted
from
+# apache/spark's update_build_status.yml (SPARK-60578).
+name: Update fork build status
+
+on:
+ schedule:
+ # Every 5 minutes, off the top of the hour, when GitHub delays scheduled
runs most.
+ - cron: '2-57/5 * * * *'
+ workflow_dispatch:
+
+permissions: {}
+
+concurrency:
+ group: ${{ github.workflow }}
+ cancel-in-progress: false
+
+jobs:
+ update:
+ if: github.repository == 'apache/gluten'
+ runs-on: ubuntu-slim
+ permissions:
+ actions: read
+ checks: write
+ contents: read
+ steps:
+ - uses: actions/checkout@v7
+ with:
+ sparse-checkout: .github/workflows/util/update-fork-build-status.js
+ sparse-checkout-cone-mode: false
+ - uses: actions/github-script@v9
+ with:
+ script: |
+ const script =
require(`${process.env.GITHUB_WORKSPACE}/.github/workflows/util/update-fork-build-status.js`)
+ await script({ github, context, core })
diff --git a/.github/workflows/util/iceberg-upstream/baseline.py
b/.github/workflows/util/iceberg-upstream/baseline.py
index b17080c1e4..da64292aee 100644
--- a/.github/workflows/util/iceberg-upstream/baseline.py
+++ b/.github/workflows/util/iceberg-upstream/baseline.py
@@ -374,7 +374,10 @@ if __name__ == "__main__":
args.key,
args.revision,
args.branch,
- GitHub(os.environ["GITHUB_REPOSITORY"]),
+ GitHub(
+ os.environ.get("ICEBERG_BASELINE_REPOSITORY")
+ or os.environ["GITHUB_REPOSITORY"]
+ ),
)
except (ValueError, KeyError, OSError) as error:
args.directory.mkdir(parents=True, exist_ok=True)
diff --git a/.github/workflows/util/update-fork-build-status.js
b/.github/workflows/util/update-fork-build-status.js
new file mode 100644
index 0000000000..204c08d158
--- /dev/null
+++ b/.github/workflows/util/update-fork-build-status.js
@@ -0,0 +1,252 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+// Run by update_fork_build_status.yml. For each open pull request from a
fork, reports
+// the fork's push run of each workflow below as the check named next to it.
+
+const WORKFLOWS = [
+ { file: 'velox_backend_x86.yml', check: 'Velox Backend (x86) in fork' },
+ { file: 'velox_backend_arm.yml', check: 'Velox Backend (ARM) in fork' },
+ { file: 'velox_kafka.yml', check: 'Velox Kafka in fork' },
+];
+
+// Check runs accept fewer conclusions than a workflow run reports; anything
else
+// means the run produced no result.
+const CONCLUSIONS = ['success', 'failure', 'neutral', 'cancelled',
'timed_out', 'action_required', 'skipped'];
+
+// Rate limiting, which GitHub reports as 429 or as 403 with rate-limit
headers.
+function isRateLimited(e) {
+ if (e.status == 429) return true;
+ if (e.status != 403) return false;
+ const h = (e.response && e.response.headers) || {};
+ return !!h['retry-after'] || h['x-ratelimit-remaining'] === '0' || /rate
limit/i.test(e.message || '');
+}
+
+// 5xx and network errors are retried on a later pass.
+function isTransient(e) {
+ return !e.status || e.status >= 500;
+}
+
+const runPath = (w) => '.github/workflows/' + w.file;
+
+// Job conclusions that make the run fail. No job in these workflows sets
+// continue-on-error, so a check can turn red before the run ends.
+const FAILED = ['failure', 'timed_out'];
+
+// What a check records in its output text about its fork run, or null.
+function recordOf(check) {
+ if (check.conclusion == 'action_required') return null;
+ try {
+ const p = JSON.parse(check.output.text);
+ return p && p.run_id ? p : null;
+ } catch (e) {
+ return null;
+ }
+}
+
+// The fork run attempt a check points at, as "run_id/attempt", or null. Checks
+// written before attempts were recorded count as attempt 1.
+function attemptOf(check) {
+ const p = recordOf(check);
+ return p ? `${p.run_id}/${p.run_attempt || 1}` : null;
+}
+
+// The check status (and conclusion) for a run's current state and its failed
jobs.
+function stateOf(run, failedJobs) {
+ if (failedJobs.length) return { status: 'completed', conclusion: 'failure' };
+ if (run.status == 'completed') {
+ return { status: 'completed', conclusion:
CONCLUSIONS.includes(run.conclusion) ? run.conclusion : 'failure' };
+ }
+ return { status: run.status == 'in_progress' ? 'in_progress' : 'queued' };
+}
+
+function noRunSummary(w, sha) {
+ return [
+ `${w.file} runs in your fork on each push to a branch other than main. No
run was found for ${sha}.`,
+ '',
+ '1. Enable GitHub Actions in the Actions tab of your fork, then push
again. See [Managing GitHub Actions settings for a
repository](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository).',
+ '2. If your branch predates fork runs, rebase it on the latest
apache/gluten main and push again.',
+ "3. If this pull request comes from your fork's main branch, push it to
another branch and open the pull request from there.",
+ ].join('\n');
+}
+
+module.exports = async ({ github, context, core }) => {
+ const repo = { owner: context.repo.owner, repo: context.repo.repo };
+
+ const listChecks = (sha) =>
+ github.paginate('GET /repos/{owner}/{repo}/commits/{ref}/check-runs', {
...repo, ref: sha, filter: 'all', per_page: 100 });
+
+ // The fork's push runs for the pull request's head commit. Re-polled until
every
+ // workflow in `wanted` has one, because a run can take a few seconds to
register.
+ const listForkRuns = async (pr, wanted, attempts) => {
+ let runs = [];
+ for (let i = 0; i < attempts; i++) {
+ try {
+ const { data } = await github.request('GET
/repos/{owner}/{repo}/actions/runs', {
+ owner: pr.head.repo.owner.login,
+ repo: pr.head.repo.name,
+ event: 'push',
+ branch: pr.head.ref,
+ head_sha: pr.head.sha,
+ per_page: 100,
+ });
+ runs = data.workflow_runs.filter((r) => r.head_sha == pr.head.sha);
+ } catch (e) {
+ if (e.status == 404) return [];
+ throw e;
+ }
+ if (wanted.every((w) => runs.some((r) => r.path == runPath(w)))) break;
+ if (i < attempts - 1) await new Promise((r) => setTimeout(r, 3000));
+ }
+ return runs;
+ };
+
+ // Jobs of the run's latest attempt that failed. Read only while the run is
going or
+ // when it didn't pass.
+ const failedJobsOf = async (pr, run) => {
+ if (run.status == 'completed' && ['success', 'skipped',
'neutral'].includes(run.conclusion)) return [];
+ const jobs = await github.paginate('GET
/repos/{owner}/{repo}/actions/runs/{run_id}/jobs', {
+ owner: pr.head.repo.owner.login,
+ repo: pr.head.repo.name,
+ run_id: run.id,
+ filter: 'latest',
+ per_page: 100,
+ });
+ return jobs.filter((j) => FAILED.includes(j.conclusion));
+ };
+
+ const checkFor = async (pr, run) => {
+ const failedJobs = await failedJobsOf(pr, run);
+ const done = run.status == 'completed';
+ const count = failedJobs.length == 1 ? '1 job failed' :
`${failedJobs.length} jobs failed`;
+ const lines = [
+ `[See the run in ${pr.head.repo.full_name}](${run.html_url}). ` +
+ 'Rerun failed jobs there if the failure is unrelated to this pull
request.',
+ ];
+ if (failedJobs.length) {
+ lines.push('', done ? 'Failed jobs:' : 'Failed jobs so far (the run is
still going):');
+ for (const j of failedJobs) lines.push(`- [${j.name}](${j.html_url})`);
+ }
+ return {
+ ...stateOf(run, failedJobs),
+ output: {
+ title: failedJobs.length ? count + (done ? '' : ', run still going') :
'Fork run',
+ summary: lines.join('\n'),
+ // `done` stops further reads of a check whose run has ended.
+ text: JSON.stringify({
+ owner: pr.head.repo.owner.login,
+ repo: pr.head.repo.name,
+ run_id: run.id,
+ run_attempt: run.run_attempt,
+ done,
+ }),
+ },
+ };
+ };
+
+ const syncCheck = async (pr, w, check, run) => {
+ if (recordOf(check).done) return;
+ const next = await checkFor(pr, run);
+ const same = (k) => (check.output[k] || '') == next.output[k];
+ if (
+ check.status == next.status &&
+ check.conclusion == (next.conclusion || null) &&
+ same('title') &&
+ same('summary') &&
+ same('text')
+ ) {
+ return;
+ }
+ console.log(`#${pr.number} ${w.file}: ${next.status} ${next.conclusion ||
''} ${next.output.title}`);
+ // An update without `output` clears its text, which is how the next pass
finds this check.
+ return github.request('PATCH
/repos/{owner}/{repo}/check-runs/{check_run_id}', {
+ ...repo,
+ check_run_id: check.id,
+ details_url: run.html_url,
+ ...next,
+ });
+ };
+
+ const createRunCheck = async (pr, w, run) => {
+ console.log(`#${pr.number} ${w.file}: tracking ${run.html_url}`);
+ return github.rest.checks.create({
+ ...repo,
+ name: w.check,
+ head_sha: pr.head.sha,
+ details_url: run.html_url,
+ ...(await checkFor(pr, run)),
+ });
+ };
+
+ const createNoRunCheck = (pr, w) => {
+ console.log(`#${pr.number} ${w.file}: no fork run`);
+ return github.rest.checks.create({
+ ...repo,
+ name: w.check,
+ head_sha: pr.head.sha,
+ status: 'completed',
+ conclusion: 'action_required',
+ output: { title: 'No run found in your fork', summary: noRunSummary(w,
pr.head.sha) },
+ });
+ };
+
+ const failed = [];
+ let rateLimited = false;
+ for await (const { data: prs } of
github.paginate.iterator(github.rest.pulls.list, {
+ ...repo,
+ state: 'open',
+ base: 'main',
+ per_page: 100,
+ })) {
+ for (const pr of prs) {
+ // Branches of this repository (Dependabot) run upstream; skip deleted
forks.
+ if (!pr.head.repo || pr.head.repo.full_name ==
`${repo.owner}/${repo.repo}`) continue;
+ try {
+ const checks = await listChecks(pr.head.sha);
+ const checksOf = (w) => checks.filter((c) => c.name == w.check);
+ // Only a workflow without any check yet can have a run that hasn't
registered.
+ const fresh = WORKFLOWS.filter((w) => checksOf(w).length == 0);
+ const runs = await listForkRuns(pr, fresh, fresh.length ? 3 : 1);
+ for (const w of WORKFLOWS) {
+ const run = runs.filter((r) => r.path == runPath(w)).sort((a, b) =>
b.id - a.id)[0];
+ if (run) {
+ // A newer run or attempt than any check points at (a rerun, or a
push back to
+ // an earlier commit) gets a check of its own, so a rerun shows as
in progress
+ // instead of keeping the previous attempt's result.
+ const check = checksOf(w).find((c) => attemptOf(c) ==
`${run.id}/${run.run_attempt}`);
+ await (check ? syncCheck(pr, w, check, run) : createRunCheck(pr,
w, run));
+ } else if (!checksOf(w).some((c) => c.conclusion ==
'action_required')) {
+ await createNoRunCheck(pr, w);
+ }
+ }
+ } catch (e) {
+ // Stop the pass instead of sending more requests; the next pass picks
up here.
+ if (isRateLimited(e)) {
+ console.log(`#${pr.number}: rate limited, stopping this pass`);
+ rateLimited = true;
+ break;
+ }
+ if (isTransient(e)) continue;
+ console.error(`#${pr.number}: ${e.message}`);
+ failed.push(pr.number);
+ }
+ }
+ if (rateLimited) break;
+ }
+ // Keep a recurring error visible instead of hiding it in a green run.
+ if (failed.length) core.setFailed('Failed to sync #' + failed.join(', #'));
+};
diff --git a/.github/workflows/velox_backend_arm.yml
b/.github/workflows/velox_backend_arm.yml
index abf7114891..7af5b956cc 100644
--- a/.github/workflows/velox_backend_arm.yml
+++ b/.github/workflows/velox_backend_arm.yml
@@ -19,6 +19,7 @@ on:
pull_request:
paths:
- '.github/workflows/velox_backend_arm.yml'
+ - '.github/actions/**'
- 'pom.xml'
- 'backends-velox/**'
- 'gluten-uniffle/**'
@@ -40,6 +41,13 @@ on:
- '!cpp/bolt.CMakeLists.cmake'
- 'dev/**'
- 'build/mvn'
+ # A pull request from a fork is tested in the fork: each push to a branch
+ # other than main runs this workflow there (see detect-changes). In
+ # apache/gluten a push run skips every job.
+ push:
+ branches-ignore:
+ - main
+ workflow_dispatch:
env:
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION: true
@@ -48,42 +56,99 @@ env:
CCACHE_DIR: "${{ github.workspace }}/.ccache"
concurrency:
- group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
+ # head_ref is empty on push. Keying push runs by branch lets a new push
cancel the previous run.
+ group: >-
+ ${{ github.repository }}-${{ github.head_ref || (github.event_name ==
'push' && github.ref) ||
+ github.sha }}-${{ github.workflow }}
cancel-in-progress: true
jobs:
+ # Every other job chains off this one, so its `if` decides whether the
workflow runs:
+ # push and workflow_dispatch only in forks, where the run tests a fork pull
request;
+ # pull_request only in apache/gluten and only from its own branches
(Dependabot).
+ # In a fork it squash-merges the commit under test onto apache/gluten main
and
+ # publishes the result, which the jobs below check out by `revision`. It is
the
+ # only job that can write, and none of its steps run in apache/gluten.
+ merge-upstream:
+ if: >-
+ (github.repository_owner != 'apache' &&
+ (github.event_name == 'push' || github.event_name ==
'workflow_dispatch')) ||
+ (github.event_name == 'pull_request' && github.repository_owner ==
'apache' &&
+ github.event.pull_request.head.repo.full_name == github.repository)
+ runs-on: ubuntu-24.04
+ permissions:
+ contents: write
+ outputs:
+ relevant: ${{ steps.merge.outputs.relevant }}
+ revision: ${{ steps.merge.outputs.revision }}
+ base_sha: ${{ steps.merge.outputs.base_sha }}
+ steps:
+ - uses: actions/checkout@v7
+ if: github.repository_owner != 'apache'
+ with:
+ fetch-depth: 0
+ - name: Squash-merge onto apache/gluten main
+ id: merge
+ if: github.repository_owner != 'apache'
+ uses: ./.github/actions/squash-merge-upstream
+ with:
+ # The `paths:` list above. Keep the two in sync.
+ paths:
'^(\.github/workflows/velox_backend_arm\.yml$|\.github/actions/|pom\.xml$|build/mvn$|(backends-velox|gluten-(uniffle|celeborn|core|substrait|arrow|delta|iceberg|hudi|paimon|ut)|package|shims|tools/gluten-it|ep/build-velox|cpp|dev)/)'
+ paths-ignore: '^cpp/bolt/|^cpp/bolt\.CMakeLists\.cmake$'
+
# Detect which parts of the codebase were modified so downstream jobs can
# skip work that is irrelevant to the change.
detect-changes:
+ needs: merge-upstream
runs-on: ubuntu-24.04
outputs:
+ # false only on a fork run that changes nothing in the `paths:` list
above.
+ relevant: ${{ needs.merge-upstream.outputs.relevant }}
+ # The commit the other jobs check out: the merged commit on a fork run,
otherwise
+ # empty, which keeps actions/checkout's default.
+ revision: ${{ needs.merge-upstream.outputs.revision }}
cpp: ${{ steps.filter.outputs.cpp }}
java: ${{ steps.filter.outputs.java }}
tools_it: ${{ steps.filter.outputs.tools_it }}
steps:
- uses: actions/checkout@v7
with:
+ ref: ${{ needs.merge-upstream.outputs.revision }}
fetch-depth: 0
- name: Detect changed paths
id: filter
run: |
- BASE=${{ github.event.pull_request.base.sha }}
- HEAD=${{ github.sha }}
+ if [ "${{ needs.merge-upstream.outputs.relevant }}" = "false" ]; then
+ for flag in cpp java tools_it; do
+ echo "$flag=false" >> $GITHUB_OUTPUT
+ done
+ exit 0
+ fi
+ if [ -n "${{ needs.merge-upstream.outputs.base_sha }}" ]; then
+ BASE=${{ needs.merge-upstream.outputs.base_sha }}
+ HEAD=$(git rev-parse HEAD)
+ else
+ BASE=${{ github.event.pull_request.base.sha }}
+ HEAD=${{ github.sha }}
+ fi
changed=$(git diff --name-only "$BASE" "$HEAD")
echo "$changed"
match() { echo "$changed" | grep -qE "$1" && echo true || echo
false; }
echo "cpp=$(match '^(cpp/|ep/build-velox/|dev/)')"
>> $GITHUB_OUTPUT
- echo "java=$(match
'^(\.github/workflows/|pom\.xml|backends-velox/|gluten-(uniffle|celeborn|ras|core|substrait|arrow|delta|iceberg|hudi|paimon)/|gluten-ut/(common/|test/|pom\.xml)|package/|build/mvn)')"
>> $GITHUB_OUTPUT
+ echo "java=$(match
'^(\.github/(workflows|actions)/|pom\.xml|backends-velox/|gluten-(uniffle|celeborn|ras|core|substrait|arrow|delta|iceberg|hudi|paimon)/|gluten-ut/(common/|test/|pom\.xml)|package/|build/mvn)')"
>> $GITHUB_OUTPUT
echo "tools_it=$(match '^tools/gluten-it/')"
>> $GITHUB_OUTPUT
build-native-lib-centos-9:
needs: detect-changes
+ if: needs.detect-changes.outputs.relevant != 'false'
runs-on: ubuntu-24.04-arm
container: apache/gluten:vcpkg-centos-9
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Install Stash restore dependencies # the stash action needs gh
and jq, absent from this container
run: |
case "$(uname -m)" in
@@ -102,8 +167,15 @@ jobs:
with:
path: '${{ env.CCACHE_DIR }}'
key: ccache-centos9-release-default-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
+ - name: Get Ccache from apache/gluten main (fork run on Stash miss)
+ id: ccache-upstream
+ if: github.repository_owner != 'apache' &&
steps.ccache-stash.outputs.stash-hit != 'true'
+ uses: ./.github/actions/restore-upstream-ccache
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-centos9-release-default-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
- name: Get Ccache from actions/cache (fallback on Stash miss)
- if: steps.ccache-stash.outputs.stash-hit != 'true'
+ if: steps.ccache-stash.outputs.stash-hit != 'true' &&
steps.ccache-upstream.outputs.hit != 'true'
uses: actions/cache/restore@v4
with:
path: '${{ env.CCACHE_DIR }}'
@@ -147,6 +219,8 @@ jobs:
container: ${{ matrix.os }}
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Native Artifacts
uses: actions/download-artifact@v8
with:
@@ -185,6 +259,8 @@ jobs:
container: apache/gluten:centos-9-jdk8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Install Stash restore dependencies # the stash action needs gh
and jq, absent from this container
run: |
case "$(uname -m)" in
@@ -203,8 +279,15 @@ jobs:
with:
path: '${{ env.CCACHE_DIR }}'
key: ccache-centos9-release-shared-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
+ - name: Get Ccache from apache/gluten main (fork run on Stash miss)
+ id: ccache-upstream
+ if: github.repository_owner != 'apache' &&
steps.ccache-stash.outputs.stash-hit != 'true'
+ uses: ./.github/actions/restore-upstream-ccache
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-centos9-release-shared-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
- name: Get Ccache from actions/cache (fallback on Stash miss)
- if: steps.ccache-stash.outputs.stash-hit != 'true'
+ if: steps.ccache-stash.outputs.stash-hit != 'true' &&
steps.ccache-upstream.outputs.hit != 'true'
uses: actions/cache/restore@v4
with:
path: '${{ env.CCACHE_DIR }}'
diff --git a/.github/workflows/velox_backend_x86.yml
b/.github/workflows/velox_backend_x86.yml
index 5a4a22fcd2..c7a9ecc288 100644
--- a/.github/workflows/velox_backend_x86.yml
+++ b/.github/workflows/velox_backend_x86.yml
@@ -24,6 +24,7 @@ on:
- '.github/workflows/iceberg_upstream_revision.yml'
- '.github/workflows/velox_native_build.yml'
- '.github/workflows/util/iceberg-upstream/**'
+ - '.github/actions/**'
- 'pom.xml'
- 'backends-velox/**'
- 'gluten-uniffle/**'
@@ -45,6 +46,12 @@ on:
- '!cpp/bolt.CMakeLists.cmake'
- 'dev/**'
- 'build/mvn'
+ # A pull request from a fork is tested in the fork: each push to a branch
+ # other than main runs this workflow there (see detect-changes). In
+ # apache/gluten a push run skips every job.
+ push:
+ branches-ignore:
+ - main
# Nightly full run. PR runs trim the TPC matrices to representative JDKs and
# skip the OOM / random-kill chaos jobs unless native code changed (see
# detect-changes' `full_run` output); this scheduled run restores the full
@@ -73,10 +80,51 @@ env:
UT_PARALLELISM: 3
concurrency:
- group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
+ # head_ref is empty on push. Keying push runs by branch lets a new push
cancel the previous run.
+ group: >-
+ ${{ github.repository }}-${{ github.head_ref || (github.event_name ==
'push' && github.ref) ||
+ github.sha }}-${{ github.workflow }}
cancel-in-progress: true
jobs:
+ # Every other job chains off this one, so its `if` decides whether the
workflow runs:
+ # • push / workflow_dispatch in a fork: the run tests a fork pull request.
+ # • pull_request: only in apache/gluten and only from its own branches
+ # (Dependabot); a fork's pull request is tested by the push run in the
fork.
+ # • schedule: only in apache/gluten. A fork with Actions enabled would
+ # otherwise burn ~1800 runner minutes nightly on a signal nobody reads.
+ # In a fork it squash-merges the commit under test onto apache/gluten main
and
+ # publishes the result, which the jobs below check out by `revision`. It is
the
+ # only job that can write, and none of its steps run in apache/gluten.
+ merge-upstream:
+ if: >-
+ (github.repository_owner != 'apache' &&
+ (github.event_name == 'push' || github.event_name ==
'workflow_dispatch')) ||
+ (github.event_name == 'pull_request' && github.repository_owner ==
'apache' &&
+ github.event.pull_request.head.repo.full_name == github.repository) ||
+ (github.event_name == 'schedule' && github.repository_owner == 'apache')
||
+ (github.event_name == 'workflow_dispatch' && github.repository_owner ==
'apache')
+ runs-on: ubuntu-24.04
+ permissions:
+ contents: write
+ outputs:
+ relevant: ${{ steps.merge.outputs.relevant }}
+ revision: ${{ steps.merge.outputs.revision }}
+ base_sha: ${{ steps.merge.outputs.base_sha }}
+ steps:
+ - uses: actions/checkout@v7
+ if: github.repository_owner != 'apache'
+ with:
+ fetch-depth: 0
+ - name: Squash-merge onto apache/gluten main
+ id: merge
+ if: github.repository_owner != 'apache'
+ uses: ./.github/actions/squash-merge-upstream
+ with:
+ # The `paths:` list above. Keep the two in sync.
+ paths:
'^(\.github/workflows/(velox_backend_x86|iceberg_spark_ut|iceberg_upstream_revision|velox_native_build)\.yml$|\.github/workflows/util/(parallel-ut|iceberg-upstream)/|\.github/actions/|pom\.xml$|build/mvn$|(backends-velox|gluten-(uniffle|celeborn|core|substrait|arrow|delta|iceberg|hudi|paimon|ut)|package|shims|tools/gluten-it|ep/build-velox|cpp|dev)/)'
+ paths-ignore: '^cpp/bolt/|^cpp/bolt\.CMakeLists\.cmake$'
+
# Detect which parts of the codebase were modified so downstream jobs can
# skip work that is irrelevant to the change (e.g. a Spark-3.4-shim-only
change does
# not need to run spark-test-spark35/40/41; C++ changes still run
spark-test-*
@@ -98,12 +146,15 @@ jobs:
# widen their matrices to the extended combos that PR runs skip (alternative
# JDKs, older Celeborn, chaos tests on java-only changes).
detect-changes:
- # Every job chains off this one, so the fork guard here skips the whole
- # workflow: a fork with Actions enabled would otherwise burn ~1800 runner
- # minutes nightly on a signal nobody reads.
- if: github.event_name != 'schedule' || github.repository_owner == 'apache'
+ needs: merge-upstream
runs-on: ubuntu-24.04
outputs:
+ # false only on a fork run that changes nothing in the `paths:` list
above.
+ relevant: ${{ needs.merge-upstream.outputs.relevant }}
+ # The commit the other jobs check out: the merged commit on a fork run,
otherwise
+ # empty, which keeps actions/checkout's default.
+ revision: ${{ needs.merge-upstream.outputs.revision }}
+ base_sha: ${{ needs.merge-upstream.outputs.base_sha }}
full_run: ${{ steps.filter.outputs.full_run }}
cpp: ${{ steps.filter.outputs.cpp }}
java: ${{ steps.filter.outputs.java }}
@@ -121,12 +172,15 @@ jobs:
steps:
- uses: actions/checkout@v7
with:
+ ref: ${{ needs.merge-upstream.outputs.revision }}
fetch-depth: 0
- name: Detect changed paths
id: filter
run: |
event="${{ github.event_name }}"
- if [ "$event" = "schedule" ] || [ "$event" = "workflow_dispatch" ];
then
+ # A fork run, including a manual one, diffs its merged commit
against main.
+ if [ -z "${{ needs.merge-upstream.outputs.base_sha }}" ] &&
+ { [ "$event" = "schedule" ] || [ "$event" = "workflow_dispatch"
]; }; then
# schedule / workflow_dispatch: no PR diff to inspect.
# For scheduled runs, skip entirely when no commits landed in the
# last 24 hours — avoids burning runner minutes on idle days.
@@ -149,15 +203,26 @@ jobs:
exit 0
fi
echo "full_run=false" >> $GITHUB_OUTPUT
- BASE=${{ github.event.pull_request.base.sha }}
- HEAD=${{ github.sha }}
+ if [ "${{ needs.merge-upstream.outputs.relevant }}" = "false" ]; then
+ for flag in cpp java udf shims34 shims35 shims40 shims41 shims42
tools_it; do
+ echo "$flag=false" >> $GITHUB_OUTPUT
+ done
+ exit 0
+ fi
+ if [ -n "${{ needs.merge-upstream.outputs.base_sha }}" ]; then
+ BASE=${{ needs.merge-upstream.outputs.base_sha }}
+ HEAD=$(git rev-parse HEAD)
+ else
+ BASE=${{ github.event.pull_request.base.sha }}
+ HEAD=${{ github.sha }}
+ fi
changed=$(git diff --name-only "$BASE" "$HEAD")
echo "$changed"
match() { echo "$changed" | grep -qE "$1" && echo true || echo
false; }
echo "cpp=$(match '^(cpp/|ep/build-velox/|dev/)')"
>> $GITHUB_OUTPUT
- echo "java=$(match
'^(\.github/workflows/|pom\.xml|backends-velox/|gluten-(uniffle|celeborn|ras|core|substrait|arrow|delta|iceberg|hudi|paimon)/|gluten-ut/(common/|test/|src-iceberg-upstream/|pom\.xml)|package/|build/mvn)')"
>> $GITHUB_OUTPUT
+ echo "java=$(match
'^(\.github/(workflows|actions)/|pom\.xml|backends-velox/|gluten-(uniffle|celeborn|ras|core|substrait|arrow|delta|iceberg|hudi|paimon)/|gluten-ut/(common/|test/|src-iceberg-upstream/|pom\.xml)|package/|build/mvn)')"
>> $GITHUB_OUTPUT
echo "udf=$(match
'^(backends-velox/src/test/scala/org/apache/gluten/expression/VeloxUdfSuite\.scala|\.github/workflows/velox_backend_x86\.yml)$')"
>> $GITHUB_OUTPUT
echo "shims34=$(match
'^(shims/(spark34|common)/|gluten-ut/spark34/)')" >>
$GITHUB_OUTPUT
echo "shims35=$(match
'^(shims/(spark35|common)/|gluten-ut/spark35/)')" >>
$GITHUB_OUTPUT
@@ -168,6 +233,7 @@ jobs:
build-native-lib-centos-8:
needs: detect-changes
+ if: needs.detect-changes.outputs.relevant != 'false'
runs-on: ubuntu-24.04
container: apache/gluten:vcpkg-almalinux-8-gcc13
steps:
@@ -178,14 +244,23 @@ jobs:
dnf config-manager --add-repo
https://cli.github.com/packages/rpm/gh-cli.repo
dnf install -y gh --setopt=gpgcheck=0
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Get Ccache from Apache Stash
id: ccache-stash
uses:
apache/infrastructure-actions/stash/restore@69afc125e535c4c41e7f1b7470f583087e0f344b
with:
path: '${{ env.CCACHE_DIR }}'
key: ccache-almalinux8-release-default-${{
hashFiles('ep/build-velox/src/**') }}
+ - name: Get Ccache from apache/gluten main (fork run on Stash miss)
+ id: ccache-upstream
+ if: github.repository_owner != 'apache' &&
steps.ccache-stash.outputs.stash-hit != 'true'
+ uses: ./.github/actions/restore-upstream-ccache
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-almalinux8-release-default-${{
hashFiles('ep/build-velox/src/**') }}
- name: Get Ccache from actions/cache (fallback on Stash miss)
- if: steps.ccache-stash.outputs.stash-hit != 'true'
+ if: steps.ccache-stash.outputs.stash-hit != 'true' &&
steps.ccache-upstream.outputs.hit != 'true'
uses: actions/cache/restore@v4
with:
path: '${{ env.CCACHE_DIR }}'
@@ -264,6 +339,8 @@ jobs:
container: ${{ matrix.os }}
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Native Artifacts
uses: actions/download-artifact@v8
with:
@@ -394,6 +471,8 @@ jobs:
container: ${{ matrix.os }}
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Native Artifacts
uses: actions/download-artifact@v8
with:
@@ -474,6 +553,8 @@ jobs:
sudo docker image prune --all --force > /dev/null
df -h
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Native Artifacts
uses: actions/download-artifact@v8
with:
@@ -592,6 +673,8 @@ jobs:
sudo docker image prune --all --force > /dev/null
df -h
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Native Artifacts
uses: actions/download-artifact@v8
with:
@@ -634,6 +717,8 @@ jobs:
container: apache/gluten:centos-8-jdk8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Native Artifacts
uses: actions/download-artifact@v8
with:
@@ -691,6 +776,8 @@ jobs:
container: centos:8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Native Artifacts
uses: actions/download-artifact@v8
with:
@@ -766,6 +853,8 @@ jobs:
container: apache/gluten:centos-8-jdk8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -837,6 +926,8 @@ jobs:
container: apache/gluten:centos-8-jdk8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -888,6 +979,8 @@ jobs:
container: apache/gluten:centos-8-jdk8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -961,6 +1054,8 @@ jobs:
container: apache/gluten:centos-8-jdk8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -1024,6 +1119,8 @@ jobs:
container: apache/gluten:centos-8-jdk8
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -1069,6 +1166,7 @@ jobs:
steps:
- uses: actions/checkout@v7
with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
# 'check.py tidy commit' diffs HEAD^..HEAD, which a depth-1 clone
lacks: the
# diff comes back empty and the check passes without inspecting
anything.
fetch-depth: 2
@@ -1090,8 +1188,15 @@ jobs:
with:
path: '${{ env.CCACHE_DIR }}'
key: ccache-centos9-release-shared-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
+ - name: Get Ccache from apache/gluten main (fork run on Stash miss)
+ id: ccache-upstream
+ if: github.repository_owner != 'apache' &&
steps.ccache-stash.outputs.stash-hit != 'true'
+ uses: ./.github/actions/restore-upstream-ccache
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-centos9-release-shared-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
- name: Get Ccache from actions/cache (fallback on Stash miss)
- if: steps.ccache-stash.outputs.stash-hit != 'true'
+ if: steps.ccache-stash.outputs.stash-hit != 'true' &&
steps.ccache-upstream.outputs.hit != 'true'
uses: actions/cache/restore@v4
with:
path: '${{ env.CCACHE_DIR }}'
@@ -1207,6 +1312,8 @@ jobs:
uses: ./.github/workflows/iceberg_spark_ut.yml
with:
native_artifact: velox-native-lib-centos-8-${{github.sha}}
+ revision: ${{ needs.detect-changes.outputs.revision }}
+ base_revision: ${{ needs.detect-changes.outputs.base_sha }}
build-cudf-centos-9:
needs: detect-changes
@@ -1221,14 +1328,23 @@ jobs:
- run: df -h | sort -k 5 -nr # check disk space for debug
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Get Ccache from Apache Stash
id: ccache-stash
uses:
apache/infrastructure-actions/stash/restore@69afc125e535c4c41e7f1b7470f583087e0f344b
with:
path: '${{ env.CCACHE_DIR }}'
key: ccache-centos9-cudf-release-shared-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
+ - name: Get Ccache from apache/gluten main (fork run on Stash miss)
+ id: ccache-upstream
+ if: github.repository_owner != 'apache' &&
steps.ccache-stash.outputs.stash-hit != 'true'
+ uses: ./.github/actions/restore-upstream-ccache
+ with:
+ path: '${{ env.CCACHE_DIR }}'
+ key: ccache-centos9-cudf-release-shared-${{runner.arch}}-${{
hashFiles('ep/build-velox/src/**') }}
- name: Get Ccache from actions/cache (fallback on Stash miss)
- if: steps.ccache-stash.outputs.stash-hit != 'true'
+ if: steps.ccache-stash.outputs.stash-hit != 'true' &&
steps.ccache-upstream.outputs.hit != 'true'
uses: actions/cache/restore@v4
with:
path: '${{ env.CCACHE_DIR }}'
@@ -1277,6 +1393,8 @@ jobs:
container: apache/gluten:centos-8-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Build with fast-build profile (Spark 4.0, Java 17)
run: |
cd $GITHUB_WORKSPACE/
@@ -1292,6 +1410,8 @@ jobs:
container: apache/gluten:centos-8-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Build Spark 4.2 Velox (Java 17, Scala 2.13)
run: |
cd $GITHUB_WORKSPACE/
@@ -1313,6 +1433,8 @@ jobs:
container: apache/gluten:centos-8-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -1379,6 +1501,8 @@ jobs:
container: apache/gluten:centos-8-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -1436,6 +1560,8 @@ jobs:
container: apache/gluten:centos-9-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -1508,6 +1634,8 @@ jobs:
container: apache/gluten:centos-9-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -1565,6 +1693,8 @@ jobs:
container: apache/gluten:centos-9-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
@@ -1640,6 +1770,8 @@ jobs:
container: apache/gluten:centos-9-jdk17
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.detect-changes.outputs.revision }}
- name: Download All Artifacts
uses: actions/download-artifact@v8
with:
diff --git a/.github/workflows/velox_kafka.yml
b/.github/workflows/velox_kafka.yml
index 0acab52607..e65bd4310c 100644
--- a/.github/workflows/velox_kafka.yml
+++ b/.github/workflows/velox_kafka.yml
@@ -19,6 +19,7 @@ on:
pull_request:
paths:
- '.github/workflows/velox_kafka.yml'
+ - '.github/actions/**'
- 'pom.xml'
- 'backends-velox/**'
- 'gluten-kafka/**'
@@ -32,6 +33,12 @@ on:
- 'ep/build-velox/**'
- 'dev/**'
- 'build/mvn'
+ # A pull request from a fork is tested in the fork: each push to a branch
+ # other than main runs this workflow there (see merge-upstream). In
+ # apache/gluten a push run skips every job.
+ push:
+ branches-ignore:
+ - main
workflow_dispatch:
permissions:
@@ -39,7 +46,10 @@ permissions:
actions: read
concurrency:
- group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{
github.workflow }}
+ # head_ref is empty on push. Keying push runs by branch lets a new push
cancel the previous run.
+ group: >-
+ ${{ github.repository }}-${{ github.head_ref || (github.event_name ==
'push' && github.ref) ||
+ github.sha }}-${{ github.workflow }}
cancel-in-progress: true
env:
@@ -47,12 +57,52 @@ env:
PROMPT_ALWAYS_RESPOND: 'y'
jobs:
+ # Every other job chains off this one, so its `if` decides whether the
workflow runs:
+ # push and workflow_dispatch in forks, where the run tests a fork pull
request;
+ # pull_request only in apache/gluten and only from its own branches
(Dependabot);
+ # workflow_dispatch in apache/gluten.
+ # In a fork it squash-merges the commit under test onto apache/gluten main
and
+ # publishes the result, which the jobs below check out by `revision`. It is
the
+ # only job that can write, and none of its steps run in apache/gluten.
+ merge-upstream:
+ if: >-
+ (github.event_name == 'push' && github.repository_owner != 'apache') ||
+ (github.event_name == 'pull_request' && github.repository_owner ==
'apache' &&
+ github.event.pull_request.head.repo.full_name == github.repository) ||
+ github.event_name == 'workflow_dispatch'
+ runs-on: ubuntu-24.04
+ permissions:
+ contents: write
+ outputs:
+ # false only on a fork run that changes nothing in the `paths:` list
above.
+ relevant: ${{ steps.merge.outputs.relevant }}
+ # The commit the other jobs check out: the merged commit on a fork run,
otherwise
+ # empty, which keeps actions/checkout's default.
+ revision: ${{ steps.merge.outputs.revision }}
+ steps:
+ - uses: actions/checkout@v7
+ if: github.repository_owner != 'apache'
+ with:
+ fetch-depth: 0
+ - name: Squash-merge onto apache/gluten main
+ id: merge
+ if: github.repository_owner != 'apache'
+ uses: ./.github/actions/squash-merge-upstream
+ with:
+ # The `paths:` list above. Keep the two in sync.
+ paths:
'^(\.github/workflows/velox_kafka\.yml$|\.github/actions/|pom\.xml$|build/mvn$|(backends-velox|gluten-(kafka|core|substrait|ut)|shims|cpp|ep/build-velox|dev)/)'
+ paths-ignore: '^cpp/bolt/|^cpp/bolt\.CMakeLists\.cmake$'
+
build-native:
+ needs: merge-upstream
+ if: needs.merge-upstream.outputs.relevant != 'false'
runs-on: ubuntu-24.04
container: apache/gluten:vcpkg-almalinux-8-gcc13
timeout-minutes: 180
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.merge-upstream.outputs.revision }}
- name: Install build tools
run: |
dnf install -y jq bison tzdata
@@ -66,8 +116,15 @@ jobs:
with:
path: ${{ env.CCACHE_DIR }}
key: ccache-almalinux8-release-default-${{
hashFiles('ep/build-velox/src/**') }}
+ - name: Get Ccache from apache/gluten main (fork run on Stash miss)
+ id: ccache-upstream
+ if: github.repository_owner != 'apache' &&
steps.ccache-stash.outputs.stash-hit != 'true'
+ uses: ./.github/actions/restore-upstream-ccache
+ with:
+ path: ${{ env.CCACHE_DIR }}
+ key: ccache-almalinux8-release-default-${{
hashFiles('ep/build-velox/src/**') }}
- name: Get Ccache from actions/cache (fallback on Stash miss)
- if: steps.ccache-stash.outputs.stash-hit != 'true'
+ if: steps.ccache-stash.outputs.stash-hit != 'true' &&
steps.ccache-upstream.outputs.hit != 'true'
uses: actions/cache/restore@v4
with:
path: ${{ env.CCACHE_DIR }}
@@ -79,6 +136,8 @@ jobs:
run: |
source /opt/rh/gcc-toolset-13/enable
export NUM_THREADS=2 CCACHE_MAXSIZE=1G
+ # As in dev/ci-velox-build*.sh, which build the stash restored above.
+ export CCACHE_BASEDIR=$GITHUB_WORKSPACE
./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --enable_kafka=ON \
--build_arrow=OFF --build_tests=ON --build_benchmarks=OFF
--build_examples=OFF \
--enable_s3=OFF --enable_gcs=OFF --enable_hdfs=OFF
--enable_abfs=OFF
@@ -127,6 +186,8 @@ jobs:
--health-retries 12
steps:
- uses: actions/checkout@v7
+ with:
+ ref: ${{ needs.merge-upstream.outputs.revision }}
- name: Setup java
run: |
sudo apt-get update
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 294afc15be..412b1a5716 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -76,6 +76,15 @@ To re-trigger CH CI, please post the below comment on PR
page:
##### Velox Backend CI
To check CI failure for Velox backend, please go into the GitHub action page
from your PR page.
+For a pull request from a fork, the Velox x86, ARM and Kafka workflows
(`velox_backend_x86.yml`, `velox_backend_arm.yml` and `velox_kafka.yml`) run in
your fork each time you push the PR branch, not in apache/gluten. To make that
work:
+
+- Enable GitHub Actions in the Actions tab of your fork.
+- Open the pull request from a branch other than your fork's `main`.
+
+Each run tests your branch merged onto the current apache/gluten `main`, but
it runs the workflow files from your branch. After a CI change lands on `main`,
merge or rebase onto `main` and push to pick it up.
+
+The `Velox Backend (x86) in fork`, `Velox Backend (ARM) in fork` and `Velox
Kafka in fork` checks on your pull request link to the runs in your fork. They
are updated every 5 minutes, and a check fails as soon as a job in its run
fails. To rerun failed jobs, rerun them in your fork. If you can't, a committer
can push your branch to their own fork and link that run on the pull request.
+
To see the perf. impact on Velox backend, you can comment `/Benchmark Velox`
on PR page to trigger a pretest. The benchmark
(currently TPC-H) result will be posted after completed.
@@ -83,6 +92,8 @@ To see the perf. impact on Velox backend, you can comment
`/Benchmark Velox` on
Please ensure no CI failure is reported for your PR.
+The fork runs of a PR use the workflow files from its branch, so review
changes under `.github/` with that in mind.
+
A PR may need several iterations in the community review process. Please
@reviewer once your PR is ready for next round of review.
For critical code change, merging the PR requires at least two committers'
approval.
diff --git a/dev/ci-velox-buildshared-centos-9.sh
b/dev/ci-velox-buildshared-centos-9.sh
index 02fcd50a88..4ba1d91eb9 100755
--- a/dev/ci-velox-buildshared-centos-9.sh
+++ b/dev/ci-velox-buildshared-centos-9.sh
@@ -25,6 +25,10 @@ if [ "$(uname -m)" = "aarch64" ]; then
fi
+# Rewrite absolute paths under the checkout to relative ones, so a ccache
saved by
+# a checkout at another path still hits.
+export CCACHE_BASEDIR=$(cd "$(dirname "$BASH_SOURCE")/.."; pwd)
+
export VELOX_BUILD_SHARED=ON
./dev/builddeps-veloxbe.sh --run_setup_script=OFF --build_arrow=OFF
--build_tests=ON \
diff --git a/dev/ci-velox-buildstatic-centos-8-enhanced-features.sh
b/dev/ci-velox-buildstatic-centos-8-enhanced-features.sh
index 176a13bd18..3d211c8c77 100755
--- a/dev/ci-velox-buildstatic-centos-8-enhanced-features.sh
+++ b/dev/ci-velox-buildstatic-centos-8-enhanced-features.sh
@@ -24,6 +24,10 @@ if [ "$(uname -m)" = "aarch64" ]; then
export VCPKG_FORCE_SYSTEM_BINARIES=1;
fi
+# Rewrite absolute paths under the checkout to relative ones, so a ccache
saved by
+# a checkout at another path still hits.
+export CCACHE_BASEDIR=$(cd "$(dirname "$BASH_SOURCE")/.."; pwd)
+
export NUM_THREADS=2
./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF
--build_tests=ON --build_benchmarks=ON \
--build_examples=OFF --enable_s3=ON --enable_gcs=ON
--enable_hdfs=ON --enable_abfs=ON --enable_enhanced_features=ON
diff --git a/dev/ci-velox-buildstatic-centos-8.sh
b/dev/ci-velox-buildstatic-centos-8.sh
index f5cd4ed884..c82ce63260 100644
--- a/dev/ci-velox-buildstatic-centos-8.sh
+++ b/dev/ci-velox-buildstatic-centos-8.sh
@@ -24,6 +24,10 @@ if [ "$(uname -m)" = "aarch64" ]; then
export VCPKG_FORCE_SYSTEM_BINARIES=1;
fi
+# Rewrite absolute paths under the checkout to relative ones, so a ccache
saved by
+# a checkout at another path still hits.
+export CCACHE_BASEDIR=$(cd "$(dirname "$BASH_SOURCE")/.."; pwd)
+
export NUM_THREADS=2
./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF
--build_tests=OFF --build_benchmarks=OFF \
--build_examples=OFF --enable_s3=ON --enable_gcs=ON
--enable_hdfs=ON --enable_abfs=ON
diff --git a/dev/ci-velox-buildstatic-centos-9.sh
b/dev/ci-velox-buildstatic-centos-9.sh
index 15b9767e71..fa58cba77c 100755
--- a/dev/ci-velox-buildstatic-centos-9.sh
+++ b/dev/ci-velox-buildstatic-centos-9.sh
@@ -23,6 +23,10 @@ if [ "$(uname -m)" = "aarch64" ]; then
export VCPKG_FORCE_SYSTEM_BINARIES=1;
fi
+# Rewrite absolute paths under the checkout to relative ones, so a ccache
saved by
+# a checkout at another path still hits.
+export CCACHE_BASEDIR=$(cd "$(dirname "$BASH_SOURCE")/.."; pwd)
+
export NUM_THREADS=2
./dev/builddeps-veloxbe.sh --enable_vcpkg=ON --build_arrow=OFF
--build_tests=OFF --build_benchmarks=OFF \
--build_examples=OFF --enable_s3=ON --enable_gcs=ON
--enable_hdfs=ON --enable_abfs=ON
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]