This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/doris-thirdparty.git
The following commit(s) were added to refs/heads/main by this push:
new 7adcd59775c [ci] Unblock Docker on the Linux build, cache ccache, and
stop macOS from stalling (#410)
7adcd59775c is described below
commit 7adcd59775c1af5263f969bd1f793c7505a904bd
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Aug 17 17:29:08 2026 +0800
[ci] Unblock Docker on the Linux build, cache ccache, and stop macOS from
stalling (#410)
Run 31988123966 took 5h01m end to end. Where it went, and what this changes.
**The Docker image waited 1h39m for nothing.** `update-docker` only consumes
`doris-thirdparty-prebuilt-linux-x86_64.tar.xz`, which was published at
05:19,
but as `needs: build` it depended on the whole matrix and did not start
until
06:58 - held up by macOS x86_64, which finished at 06:56. The build job is
now a
reusable workflow called once per platform, so `update-docker` can depend on
`build-linux-x86_64` alone. Splitting it this way is also what makes the two
changes below expressible per platform.
**Every macOS build sat idle for exactly 10 minutes.** thrift's configure
finds
the dotnet the runner images ship, so `make` runs `dotnet build -c Release`
for
`lib/netstd`. The build itself finishes in seconds, and then the compiler
server
it leaves behind holds the stdout it inherited for its full keep-alive while
nothing happens - 600.0s on macos-14, 598.8s on macos-15-intel, 0.01s on
Linux.
Set `UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` so no
server is
left running.
**macOS x86_64 costs 4h22m41s of a 5h01m run** - against 1h27m54s for Linux
arm64 - because that runner is roughly 2x slower than the M1 one at
compiling and
about 8x slower at spawning processes, which is what a tree of 70+ autotools
packages does all day. curl's 576 configure probes take 21s on macos-14 and
178s
there. It already runs with `continue-on-error`, so build it only once its
published archive is older than `vars.MACOS_X86_MAX_AGE_DAYS` (default 7)
rather
than on every thirdparty change. `workflow_dispatch` with `force_build`
still
builds it unconditionally.
**Nothing was cached between runs.** ccache is installed on the runners but
nothing routed through it and nothing persisted it, so a thirdparty change
that
touches one package still rebuilt all ~80. Set
`ENABLE_THIRDPARTY_CCACHE=ON` and
carry `$CCACHE_DIR` across runs with `actions/cache`. The cache is saved
even
when the build fails, which is what makes the `max_attempts=3` retry in the
prerelease job affordable - today a flaky download mirror can cost three
full
builds. This needs apache/doris to honour `ENABLE_THIRDPARTY_CCACHE`; until
that
lands it is an unused variable and the cache stays empty.
`success` and `failure` now list every build job. `success` had to move to
`!cancelled() && !failure()`, because a skipped `build-macos-x86_64` would
otherwise skip it too and leave the release note stuck at BUILDING - which
is the
state that stops all future scheduled runs.
Co-authored-by: morningman <[email protected]>
---
.github/workflows/build-target.yml | 243 ++++++++++++++++++++++++++++
.github/workflows/build.yml | 321 ++++++++++---------------------------
2 files changed, 325 insertions(+), 239 deletions(-)
diff --git a/.github/workflows/build-target.yml
b/.github/workflows/build-target.yml
new file mode 100644
index 00000000000..192c94085f3
--- /dev/null
+++ b/.github/workflows/build-target.yml
@@ -0,0 +1,243 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# Builds the third party tree for one target platform and uploads the archive
to the
+# `automation` release. build.yml calls this once per platform.
+#
+# This is a reusable workflow rather than a matrix so that each platform is a
separate
+# job in the caller and can be depended on individually: `update-docker` only
consumes
+# the Linux x86_64 archive, but as one matrix job it had to wait for every
platform.
+# The package lists and the toolchain are derived from RUNNER_OS /
RUNNER_ARCH, which
+# is what the per-platform matrix entries used to carry.
+
+name: Build target
+
+on:
+ workflow_call:
+ inputs:
+ target:
+ description: "Display name of the target platform, e.g. Linux-arm64"
+ required: true
+ type: string
+ runs_on:
+ description: "Runner label to build on"
+ required: true
+ type: string
+ thirdparty_commit_hash:
+ description: "apache/doris commit that last touched thirdparty/,
stamped into the archive"
+ required: true
+ type: string
+ tolerate_failure:
+ description: "Report success even if the build fails, so this platform
cannot hold up a release"
+ required: false
+ default: false
+ type: boolean
+
+jobs:
+ build:
+ name: ${{ inputs.target }}
+ runs-on: ${{ inputs.runs_on }}
+ permissions:
+ contents: write
+ env:
+ GH_REPO: ${{ github.repository }}
+ GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ CMAKE_VERSION: 3.25.3
+ # Every run of this job is a cold build of the whole tree, and one
changed package
+ # in apache/doris rebuilds all ~80 of them. Compile the cmake-based
packages
+ # through ccache and carry the cache across runs so a rebuild costs what
actually
+ # changed. This is also what makes the bounded retry in the prerelease
job
+ # affordable: today a flaky download mirror can cost three full builds.
+ # Needs apache/doris to honour ENABLE_THIRDPARTY_CCACHE; until it does,
this is
+ # an unused variable and the cache stays empty.
+ ENABLE_THIRDPARTY_CCACHE: 'ON'
+ CCACHE_COMPRESS: '1'
+ # The whole repository shares a 10 GB Actions cache budget and four
platforms save
+ # into it, so keep each one inside a quarter of that.
+ CCACHE_MAXSIZE: '2G'
+ # thrift's configure finds the dotnet the runner images ship, so `make`
runs
+ # `dotnet build -c Release` for lib/netstd. On macOS the build itself
takes
+ # seconds, and then the compiler server it leaves behind holds the
stdout it
+ # inherited for its full 10 minute keep-alive while nothing happens:
600.0s on
+ # macos-14 and 598.8s on macos-15-intel in run 31988123966. Do not start
a shared
+ # compiler server, and do not keep MSBuild worker nodes around either.
+ UseSharedCompilation: 'false'
+ MSBUILDDISABLENODEREUSE: '1'
+ DOTNET_CLI_TELEMETRY_OPTOUT: '1'
+ DOTNET_NOLOGO: '1'
+ steps:
+ - name: Prepare ccache directory
+ run: |
+ ccache_dir="${RUNNER_TEMP}/ccache"
+ mkdir -p "${ccache_dir}"
+ echo "CCACHE_DIR=${ccache_dir}" >>"${GITHUB_ENV}"
+
+ - name: Restore ccache
+ uses: actions/cache/restore@v4
+ with:
+ path: ${{ env.CCACHE_DIR }}
+ # A key that is fresh every run, so a build always saves a new
entry, plus a
+ # prefix match that picks up the newest previous one.
+ key: ccache-${{ inputs.target }}-${{ github.run_id }}
+ restore-keys: |
+ ccache-${{ inputs.target }}-
+
+ - name: Checkout easimon/maximize-build-space
+ if: ${{ inputs.target == 'Linux' }}
+ run: |
+ git clone -b v7 https://github.com/easimon/maximize-build-space
+
+ #- name: Maximize build space
+ # if: ${{ inputs.target == 'Linux' }}
+ # uses: ./maximize-build-space
+ # with:
+ # root-reserve-mb: 4096
+ # swap-size-mb: 8192
+ # remove-dotnet: 'true'
+ # remove-android: 'true'
+ # remove-haskell: 'true'
+ # remove-codeql: 'true'
+ # remove-docker-images: 'true'
+
+ - name: Checkout
+ uses: actions/checkout@v5
+ with:
+ repository: 'apache/doris'
+
+ - name: Download
+ run: |
+ cd thirdparty
+ curl -L https://github.com/${{ github.repository
}}/releases/download/automation/doris-thirdparty-source.tgz \
+ -o doris-thirdparty-source.tgz
+ tar -zxvf doris-thirdparty-source.tgz
+
+ # The gcc.gnu.org git endpoint occasionally rate-limits hosted
runners and
+ # leaves this header empty in the cached source archive. Repair only
an
+ # invalid copy, and verify the mirror content before replacing it.
+ tsan_header="src/tsan_interface_atomic.h"
+ expected_md5="d72679bea167d6a513d959f5abd149dc"
+ actual_md5="$(openssl dgst -md5 "${tsan_header}" | awk '{ print $NF
}')"
+ if [[ "${actual_md5}" != "${expected_md5}" ]]; then
+ tsan_header_tmp="${tsan_header}.tmp"
+ trap 'rm -f "${tsan_header_tmp}"' EXIT
+ curl --fail --location --retry 3 \
+
https://raw.githubusercontent.com/gcc-mirror/gcc/releases/gcc-7/libsanitizer/include/sanitizer/tsan_interface_atomic.h
\
+ --output "${tsan_header_tmp}"
+ actual_md5="$(openssl dgst -md5 "${tsan_header_tmp}" | awk '{
print $NF }')"
+ [[ "${actual_md5}" == "${expected_md5}" ]]
+ mv "${tsan_header_tmp}" "${tsan_header}"
+ trap - EXIT
+ fi
+
+ - name: Prepare for ${{ inputs.runs_on }}
+ run: |
+ # Doris's lance-c build prefers Rust 1.91.0 when it is installed.
+ # Pin it here so hosted-runner toolchain updates do not break locked
Rust dependencies.
+ rustup toolchain install 1.91.0 --profile minimal
+ rustup run 1.91.0 rustc --version
+
+ if [[ "${RUNNER_OS}" == 'macOS' ]]; then
+ # Install packages except cmake
+ brew install \
+ 'm4' 'automake' 'autoconf' 'libtool' 'pkg-config' 'texinfo'
'coreutils' \
+ 'gnu-getopt' 'python@3' 'ninja' 'ccache' 'bison' 'byacc'
'gettext' 'wget' \
+ 'pcre' 'openjdk@11' 'maven' 'node' 'llvm@20' || true
+ # Arrow 24 requires CMake 3.25 or newer.
+ brew unlink cmake || true
+ wget
"https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-macos-universal.tar.gz";
+ tar -xzf "cmake-${CMAKE_VERSION}-macos-universal.tar.gz"
+ sudo cp -r
"cmake-${CMAKE_VERSION}-macos-universal/CMake.app/Contents/"* /usr/local/
+ cmake --version
+ else
+ export DEFAULT_DIR='/opt/doris'
+ export PATH="${DEFAULT_DIR}/ldb-toolchain/bin:${PATH}"
+
+ sudo apt update
+ sudo apt-cache policy cmake
+ sudo DEBIAN_FRONTEND=noninteractive apt install --yes \
+ 'build-essential' 'automake' 'autoconf' 'libtool-bin'
'pkg-config' \
+ 'cmake=3.22.1-1ubuntu1.22.04.2' 'ninja-build' 'ccache'
'python-is-python3' \
+ 'bison' 'byacc' 'flex' 'binutils-dev' 'libiberty-dev' 'curl'
'git' 'zip' \
+ 'unzip' 'autopoint' 'openjdk-8-jdk' 'openjdk-8-jdk-headless'
'maven'
+
+ if [[ "${RUNNER_ARCH}" == 'ARM64' ]]; then
+ toolchain='ldb_toolchain_gen_aarch64.sh'
+ else
+ toolchain='ldb_toolchain_gen.sh'
+ fi
+ mkdir -p "${DEFAULT_DIR}"
+ wget
"https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.25/${toolchain}";
\
+ -q -O /tmp/ldb_toolchain_gen.sh
+ bash /tmp/ldb_toolchain_gen.sh "${DEFAULT_DIR}/ldb-toolchain"
+ fi
+
+ - name: Build and Upload
+ continue-on-error: ${{ inputs.tolerate_failure }}
+ run: |
+ if [[ "${RUNNER_OS}" == 'Linux' ]]; then
+ export DEFAULT_DIR='/opt/doris'
+ export PATH="${DEFAULT_DIR}/ldb-toolchain/bin:${PATH}"
+ export PATH="$(find /usr/lib/jvm/java-8-openjdk* -maxdepth 1 -type
d -name 'bin'):${PATH}"
+ export JAVA_HOME="$(find /usr/lib/jvm/java-8-openjdk* -maxdepth 0)"
+ export DORIS_TOOLCHAIN=clang
+ else
+ export MACOSX_DEPLOYMENT_TARGET=12.0
+ fi
+
+ export CMAKE_POLICY_VERSION_MINIMUM=3.10
+ export CUSTOM_CMAKE="/usr/local/bin/cmake"
+ cd thirdparty
+ # Diagnostics only - build-thirdparty.sh is what insists on ccache
being here.
+ ccache --zero-stats || true
+ ./build-thirdparty.sh -j "$(nproc)"
+ ccache --show-stats || true
+
+ thirdparty_commit_hash="${{ inputs.thirdparty_commit_hash }}"
+ if [[ -z "${thirdparty_commit_hash}" ]]; then
+ echo 'Thirdparty commit hash is empty' >&2
+ exit 1
+ fi
+ echo "${thirdparty_commit_hash}" >installed/_doris_thirdparty_commit_
+
+ kernel="$(uname -s | awk '{print tolower($0)}')"
+ arch="$(uname -m)"
+ archive="doris-thirdparty-prebuilt-${kernel}-${arch}.tar.xz"
+ rm -f "${archive}"
+ tar -cf - installed | xz -z -T0 - >"${archive}"
+
+ assets=("${archive}")
+ if [[ "${kernel}" == 'linux' && "${arch}" == 'aarch64' ]]; then
+ # Preserve the existing Linux ARM64 URL used by Doris release jobs.
+ cp "${archive}" doris-thirdparty-prebuild-arm64.tar.xz
+ assets+=(doris-thirdparty-prebuild-arm64.tar.xz)
+ fi
+ # Drop the old assets first: `gh release upload --clobber`
intermittently
+ # exits 1 with "release not found" while deleting the asset it is
+ # replacing, even though the upload itself went through.
+ for asset in "${assets[@]}"; do
+ gh release delete-asset automation "${asset}" --yes || true
+ done
+ gh release upload --clobber automation "${assets[@]}"
+
+ # Saved even when the build failed: what it did manage to compile is
exactly what
+ # the retry should not have to compile again.
+ - name: Save ccache
+ if: always()
+ uses: actions/cache/save@v4
+ with:
+ path: ${{ env.CCACHE_DIR }}
+ key: ccache-${{ inputs.target }}-${{ github.run_id }}
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index a221a7d4eb1..e682e2a8849 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -44,10 +44,12 @@ jobs:
env:
GH_REPO: ${{ github.repository }}
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+ MACOS_X86_MAX_AGE_DAYS: ${{ vars.MACOS_X86_MAX_AGE_DAYS || '7' }}
permissions:
contents: write
outputs:
should_release: ${{ steps.check_diff.outputs.should_release }}
+ should_build_macos_x86: ${{
steps.check_diff.outputs.should_build_macos_x86 }}
thirdparty_commit_hash: ${{
steps.check_diff.outputs.thirdparty_commit_hash }}
doris_version: ${{ steps.check_diff.outputs.doris_version }}
attempt: ${{ steps.check_diff.outputs.attempt }}
@@ -107,6 +109,26 @@ jobs:
should_release=true
fi
+ # The macOS x86_64 leg is by far the slowest of the four - 4h22m41s
of the
+ # 5h01m run 31988123966 took, against 1h27m54s for Linux arm64 - and
it is
+ # already allowed to fail without holding up a release. Rebuild it
only once
+ # its published archive has gone stale rather than on every
thirdparty change.
+ should_build_macos_x86="${should_release}"
+ if [[ "${should_build_macos_x86}" == 'true' ]]; then
+ macos_x86_asset='doris-thirdparty-prebuilt-darwin-x86_64.tar.xz'
+ asset_updated_at="$(gh release view automation --json assets \
+ --jq ".assets[] | select(.name == \"${macos_x86_asset}\") |
.updatedAt")" || asset_updated_at=''
+ if [[ -n "${asset_updated_at}" ]]; then
+ age_days="$(( ($(date -u +%s) - $(date -u -d
"${asset_updated_at}" +%s)) / 86400 ))"
+ echo "macOS x86_64 archive is ${age_days} day(s) old, rebuild
threshold is ${MACOS_X86_MAX_AGE_DAYS} day(s)"
+ if [[ "${age_days}" -lt "${MACOS_X86_MAX_AGE_DAYS}" ]]; then
+ should_build_macos_x86=false
+ fi
+ else
+ echo "No macOS x86_64 archive published yet, building it"
+ fi
+ fi
+
if "${should_release}"; then
echo -ne "Update Time: *$(date)*\nDoris Version:
*${current_version}*\nStatus: *BUILDING*\nAttempts: *${attempt}*"
>release_note.md
else
@@ -118,6 +140,7 @@ jobs:
gh release edit -F release_note.md automation
echo "should_release=${should_release}" >> $GITHUB_OUTPUT
+ echo "should_build_macos_x86=${should_build_macos_x86}" >>
$GITHUB_OUTPUT
echo "thirdparty_commit_hash=${thirdparty_commit_hash}" >>
$GITHUB_OUTPUT
echo "doris_version=${current_version}" >> $GITHUB_OUTPUT
echo "attempt=${attempt}" >> $GITHUB_OUTPUT
@@ -136,252 +159,67 @@ jobs:
gh release delete-asset automation doris-thirdparty-source.tgz --yes
|| true
gh release upload --clobber automation doris-thirdparty-source.tgz
- build:
+ # One job per platform instead of one matrix job, so that `update-docker`
can depend
+ # on the Linux x86_64 archive alone. As a matrix it had to wait for all
four: in run
+ # 31988123966 the archive it consumes was published at 05:19 and the Docker
job did
+ # not start until 06:58, held up by macOS x86_64.
+ build-linux-x86_64:
name: Build
needs: prerelease
if: needs.prerelease.outputs.should_release == 'true' ||
(github.event_name == 'workflow_dispatch' && github.event.inputs.force_build ==
'true')
- strategy:
- fail-fast: false
- matrix:
- config:
- - name: macOS-x86_64
- os: macos-15-intel
- continue-on-error: true
- packages: >-
- 'm4'
- 'automake'
- 'autoconf'
- 'libtool'
- 'pkg-config'
- 'texinfo'
- 'coreutils'
- 'gnu-getopt'
- 'python@3'
- 'ninja'
- 'ccache'
- 'bison'
- 'byacc'
- 'gettext'
- 'wget'
- 'pcre'
- 'openjdk@11'
- 'maven'
- 'node'
- 'llvm@20'
-
- - name: macOS-arm64
- os: macos-14
- packages: >-
- 'm4'
- 'automake'
- 'autoconf'
- 'libtool'
- 'pkg-config'
- 'texinfo'
- 'coreutils'
- 'gnu-getopt'
- 'python@3'
- 'ninja'
- 'ccache'
- 'bison'
- 'byacc'
- 'gettext'
- 'wget'
- 'pcre'
- 'openjdk@11'
- 'maven'
- 'node'
- 'llvm@20'
-
- - name: Linux
- os: ubuntu-22.04
- toolchain: ldb_toolchain_gen.sh
- packages: >-
- 'build-essential'
- 'automake'
- 'autoconf'
- 'libtool-bin'
- 'pkg-config'
- 'cmake=3.22.1-1ubuntu1.22.04.2'
- 'ninja-build'
- 'ccache'
- 'python-is-python3'
- 'bison'
- 'byacc'
- 'flex'
- 'binutils-dev'
- 'libiberty-dev'
- 'curl'
- 'git'
- 'zip'
- 'unzip'
- 'autopoint'
- 'openjdk-8-jdk'
- 'openjdk-8-jdk-headless'
- 'maven'
-
- - name: Linux-arm64
- os: ubuntu-22.04-arm
- toolchain: ldb_toolchain_gen_aarch64.sh
- packages: >-
- 'build-essential'
- 'automake'
- 'autoconf'
- 'libtool-bin'
- 'pkg-config'
- 'cmake=3.22.1-1ubuntu1.22.04.2'
- 'ninja-build'
- 'ccache'
- 'python-is-python3'
- 'bison'
- 'byacc'
- 'flex'
- 'binutils-dev'
- 'libiberty-dev'
- 'curl'
- 'git'
- 'zip'
- 'unzip'
- 'autopoint'
- 'openjdk-8-jdk'
- 'openjdk-8-jdk-headless'
- 'maven'
-
- runs-on: ${{ matrix.config.os }}
- continue-on-error: ${{ matrix.config.continue-on-error == true }}
- env:
- GH_REPO: ${{ github.repository }}
- GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- CMAKE_VERSION: 3.25.3
- steps:
- - name: Checkout easimon/maximize-build-space
- if: ${{ matrix.config.name == 'Linux' }}
- run: |
- git clone -b v7 https://github.com/easimon/maximize-build-space
-
- #- name: Maximize build space
- # if: ${{ matrix.config.name == 'Linux' }}
- # uses: ./maximize-build-space
- # with:
- # root-reserve-mb: 4096
- # swap-size-mb: 8192
- # remove-dotnet: 'true'
- # remove-android: 'true'
- # remove-haskell: 'true'
- # remove-codeql: 'true'
- # remove-docker-images: 'true'
-
- - name: Checkout
- uses: actions/checkout@v5
- with:
- repository: 'apache/doris'
-
- - name: Download
- run: |
- cd thirdparty
- curl -L https://github.com/${{ github.repository
}}/releases/download/automation/doris-thirdparty-source.tgz \
- -o doris-thirdparty-source.tgz
- tar -zxvf doris-thirdparty-source.tgz
-
- # The gcc.gnu.org git endpoint occasionally rate-limits hosted
runners and
- # leaves this header empty in the cached source archive. Repair only
an
- # invalid copy, and verify the mirror content before replacing it.
- tsan_header="src/tsan_interface_atomic.h"
- expected_md5="d72679bea167d6a513d959f5abd149dc"
- actual_md5="$(openssl dgst -md5 "${tsan_header}" | awk '{ print $NF
}')"
- if [[ "${actual_md5}" != "${expected_md5}" ]]; then
- tsan_header_tmp="${tsan_header}.tmp"
- trap 'rm -f "${tsan_header_tmp}"' EXIT
- curl --fail --location --retry 3 \
-
https://raw.githubusercontent.com/gcc-mirror/gcc/releases/gcc-7/libsanitizer/include/sanitizer/tsan_interface_atomic.h
\
- --output "${tsan_header_tmp}"
- actual_md5="$(openssl dgst -md5 "${tsan_header_tmp}" | awk '{
print $NF }')"
- [[ "${actual_md5}" == "${expected_md5}" ]]
- mv "${tsan_header_tmp}" "${tsan_header}"
- trap - EXIT
- fi
-
- - name: Prepare for ${{ matrix.config.os }}
- run: |
- # Doris's lance-c build prefers Rust 1.91.0 when it is installed.
- # Pin it here so hosted-runner toolchain updates do not break locked
Rust dependencies.
- rustup toolchain install 1.91.0 --profile minimal
- rustup run 1.91.0 rustc --version
-
- if [[ "${{ matrix.config.name }}" =~ macOS-* ]]; then
- # Install packages except cmake
- brew install ${{ matrix.config.packages }} || true
- # Arrow 24 requires CMake 3.25 or newer.
- brew unlink cmake || true
- wget
"https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-macos-universal.tar.gz";
- tar -xzf "cmake-${CMAKE_VERSION}-macos-universal.tar.gz"
- sudo cp -r
"cmake-${CMAKE_VERSION}-macos-universal/CMake.app/Contents/"* /usr/local/
- cmake --version
- else
- export DEFAULT_DIR='/opt/doris'
- export PATH="${DEFAULT_DIR}/ldb-toolchain/bin:${PATH}"
-
- sudo apt update
- sudo apt-cache policy cmake
- sudo DEBIAN_FRONTEND=noninteractive apt install --yes ${{
matrix.config.packages }}
-
- mkdir -p "${DEFAULT_DIR}"
- wget
"https://github.com/amosbird/ldb_toolchain_gen/releases/download/v0.25/${{
matrix.config.toolchain }}" \
- -q -O /tmp/ldb_toolchain_gen.sh
- bash /tmp/ldb_toolchain_gen.sh "${DEFAULT_DIR}/ldb-toolchain"
- fi
-
- - name: Build and Upload
- run: |
- if [[ "${{ matrix.config.name }}" == Linux* ]]; then
- export DEFAULT_DIR='/opt/doris'
- export PATH="${DEFAULT_DIR}/ldb-toolchain/bin:${PATH}"
- export PATH="$(find /usr/lib/jvm/java-8-openjdk* -maxdepth 1 -type
d -name 'bin'):${PATH}"
- export JAVA_HOME="$(find /usr/lib/jvm/java-8-openjdk* -maxdepth 0)"
- export DORIS_TOOLCHAIN=clang
- else
- export MACOSX_DEPLOYMENT_TARGET=12.0
- fi
-
- export CMAKE_POLICY_VERSION_MINIMUM=3.10
- export CUSTOM_CMAKE="/usr/local/bin/cmake"
- cd thirdparty
- ./build-thirdparty.sh -j "$(nproc)"
-
- thirdparty_commit_hash="${{
needs.prerelease.outputs.thirdparty_commit_hash }}"
- if [[ -z "${thirdparty_commit_hash}" ]]; then
- echo 'Thirdparty commit hash is empty' >&2
- exit 1
- fi
- echo "${thirdparty_commit_hash}" >installed/_doris_thirdparty_commit_
+ permissions:
+ contents: write
+ uses: ./.github/workflows/build-target.yml
+ with:
+ target: Linux
+ runs_on: ubuntu-22.04
+ thirdparty_commit_hash: ${{
needs.prerelease.outputs.thirdparty_commit_hash }}
- kernel="$(uname -s | awk '{print tolower($0)}')"
- arch="$(uname -m)"
- archive="doris-thirdparty-prebuilt-${kernel}-${arch}.tar.xz"
- rm -f "${archive}"
- tar -cf - installed | xz -z -T0 - >"${archive}"
+ build-linux-arm64:
+ name: Build
+ needs: prerelease
+ if: needs.prerelease.outputs.should_release == 'true' ||
(github.event_name == 'workflow_dispatch' && github.event.inputs.force_build ==
'true')
+ permissions:
+ contents: write
+ uses: ./.github/workflows/build-target.yml
+ with:
+ target: Linux-arm64
+ runs_on: ubuntu-22.04-arm
+ thirdparty_commit_hash: ${{
needs.prerelease.outputs.thirdparty_commit_hash }}
- assets=("${archive}")
- if [[ "${kernel}" == 'linux' && "${arch}" == 'aarch64' ]]; then
- # Preserve the existing Linux ARM64 URL used by Doris release jobs.
- cp "${archive}" doris-thirdparty-prebuild-arm64.tar.xz
- assets+=(doris-thirdparty-prebuild-arm64.tar.xz)
- fi
- # Drop the old assets first: `gh release upload --clobber`
intermittently
- # exits 1 with "release not found" while deleting the asset it is
- # replacing, even though the upload itself went through.
- for asset in "${assets[@]}"; do
- gh release delete-asset automation "${asset}" --yes || true
- done
- gh release upload --clobber automation "${assets[@]}"
+ build-macos-arm64:
+ name: Build
+ needs: prerelease
+ if: needs.prerelease.outputs.should_release == 'true' ||
(github.event_name == 'workflow_dispatch' && github.event.inputs.force_build ==
'true')
+ permissions:
+ contents: write
+ uses: ./.github/workflows/build-target.yml
+ with:
+ target: macOS-arm64
+ runs_on: macos-14
+ thirdparty_commit_hash: ${{
needs.prerelease.outputs.thirdparty_commit_hash }}
+
+ # Skipped while its published archive is still fresh enough - see the
prerelease job.
+ build-macos-x86_64:
+ name: Build
+ needs: prerelease
+ if: needs.prerelease.outputs.should_build_macos_x86 == 'true' ||
(github.event_name == 'workflow_dispatch' && github.event.inputs.force_build ==
'true')
+ permissions:
+ contents: write
+ uses: ./.github/workflows/build-target.yml
+ with:
+ target: macOS-x86_64
+ runs_on: macos-15-intel
+ tolerate_failure: true
+ thirdparty_commit_hash: ${{
needs.prerelease.outputs.thirdparty_commit_hash }}
update-docker:
name: Update Docker Image
- needs: [prerelease, build]
- # macOS failures must not block Docker when the current Linux x86_64
package is available.
+ needs: [prerelease, build-linux-x86_64]
+ # Only the Linux x86_64 archive goes into the image, so nothing else is
waited for.
if: |
always() &&
- needs.build.result != 'cancelled' &&
+ needs.build-linux-x86_64.result != 'cancelled' &&
(needs.prerelease.outputs.should_release == 'true' || github.event_name
== 'workflow_dispatch')
runs-on: ubuntu-latest
env:
@@ -527,8 +365,13 @@ jobs:
success:
name: Success
- needs: [prerelease, build, update-docker]
- if: needs.prerelease.outputs.should_release == 'true' ||
(github.event_name == 'workflow_dispatch' && github.event.inputs.force_build ==
'true')
+ needs: [prerelease, build-linux-x86_64, build-linux-arm64,
build-macos-arm64, build-macos-x86_64, update-docker]
+ # `!cancelled() && !failure()` rather than the default: build-macos-x86_64
is
+ # skipped on most runs, and a skipped dependency would otherwise skip this
job and
+ # leave the release note stuck at BUILDING forever.
+ if: |
+ !cancelled() && !failure() &&
+ (needs.prerelease.outputs.should_release == 'true' || (github.event_name
== 'workflow_dispatch' && github.event.inputs.force_build == 'true'))
runs-on: ubuntu-latest
env:
GH_REPO: ${{ github.repository }}
@@ -549,7 +392,7 @@ jobs:
failure:
name: Failure
- needs: [prerelease, build, update-docker]
+ needs: [prerelease, build-linux-x86_64, build-linux-arm64,
build-macos-arm64, build-macos-x86_64, update-docker]
# Only report on a build that actually ran. If prerelease itself failed
there
# is no version to record, and writing an empty one is exactly what used to
# make the next scheduled run rebuild from scratch, forever.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
