This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new a82cfa64658 [chore](thirdparty) build only hadoop 3.4.2.4, build azure
everywhere and slim it down, and add ccache (#66842)
a82cfa64658 is described below
commit a82cfa64658a6ace2af6da81af2a13fd7cc3a4db
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Aug 19 09:02:59 2026 +0800
[chore](thirdparty) build only hadoop 3.4.2.4, build azure everywhere and
slim it down, and add ccache (#66842)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: apache/doris-thirdparty#410
Problem Summary:
Third-party build changes, all measured against
[apache/doris-thirdparty run
31988123966](https://github.com/apache/doris-thirdparty/actions/runs/31988123966).
Supersedes #66837 and #66839, which are now closed.
**First half of a two-PR split.** This PR only changes what the
third-party build
produces (`thirdparty/**` and `.github/workflows/build-thirdparty.yml`).
The consumer
side - `cloud/`, `be/src/io/hdfs_builder.cpp`, `build.sh`,
`run-be-ut.sh`,
`run-cloud-ut.sh`, the thirdparty lifecycle test and the `env.sh` azure
default - is
in #66908, which has to land **after** this one and after the prebuilt
archives have
been rebuilt from it. See "Merge order" at the bottom.
---
## 1. Only build hadoop-libs 3.4.2.4
The tree carried two libhdfs builds: `build_hadoop_libs` built the
**3.3.6.6** fork
into `installed/{include,lib}/hadoop_hdfs/`, `build_hadoop_libs_3_4`
built the
**3.4.2.4** fork into `.../hadoop_hdfs_3_4/`, and on Linux both ran -
11m38s of the
x86_64 build and 10m00s of the arm64 one, for a fork only the cloud
module still
consumed.
The two prefixes were also mixed up on the consumer side:
`be/src/io/fs/hdfs.h` and
`be/CMakeLists.txt` were on 3.4.2.4, while `be/src/io/hdfs_builder.cpp`
and all of
`cloud/` were on 3.3.6.6. Both headers share the `LIBHDFS_HDFS_H`
include guard, so
`hdfs_builder.cpp` compiled against the 3.3.6.6 header while BE linked
the 3.4.2.4
archive. Straightening that out is #66908.
3.4.2.4 stays where it is, under `hadoop_hdfs_3_4/`, so a build-env
image from this
change can still compile older Doris branches. The 3.3.6.6 build and its
`hadoop_hdfs/` prefix are gone, and `hadoop_libs_3_4` now runs on Linux
and macOS
alike.
## 2. Build azure everywhere, and stop building what Doris does not link
**It was only ever built on x86_64 Linux.** `env.sh` forced
`DISABLE_BUILD_AZURE=ON`
on aarch64 and macOS, and `build_azure` skipped `Darwin` a second time,
so BE and the
cloud meta-service lost `+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` there.
Nothing in the
recipe was genuinely x86_64-Linux-only, just two details: `-ldl` fails
on Apple (there
is no libdl; those symbols are in libSystem), and vcpkg ships no
prebuilt cmake/ninja/
curl for aarch64 Linux so it needs `VCPKG_FORCE_SYSTEM_BINARIES=1`. The
triplet is now
derived from `uname` rather than left to vcpkg's host detection, and
`build_azure` no
longer skips Darwin.
`DISABLE_BUILD_AZURE` was answering two questions at once: whether the
third-party
tree *carries* azure, and whether BE and the cloud meta-service *link*
it. The archive
has to grow the libraries before anything can link them, so the two are
split here -
the third-party build reads `DISABLE_THIRDPARTY_BUILD_AZURE` and leaves
`env.sh`'s
switch to the consumers, whose default still skips azure on aarch64 and
macOS until
#66908.
**It was the most expensive package in the tree** - 26m06s of the 2h43m
x86_64 Linux
build - almost none of it for Doris:
| phase | time |
|---|---|
| `vcpkg install` | 22m11s |
| - protobuf 5.29.3 | 12m29s |
| - opentelemetry-cpp | 4m35s |
| - abseil | 1m58s |
| - openssl 1.1.1n | 1m38s |
| - curl / libxml2 / zlib / uAMQP / utf8-range | ~1m30s |
| azure SDK configure + build | ~3m55s |
protobuf, abseil and utf8-range were pulled in **only** because
`vcpkg.json` declared
`opentelemetry-cpp` unconditionally - roughly 19 of the 26 minutes.
Doris does not use
azure's OpenTelemetry tracing. On top of that vcpkg builds every port
twice
(`Building x64-linux-dbg` then `-rel`) and Doris links only the release
halves, and the
SDK builds appconfiguration, attestation, eventhubs, keyvault, tables,
template, uAMQP,
storage-files-datalake, storage-files-shares and storage-queues, while
`be/cmake/thirdparty.cmake` links exactly four targets: `azure-core`,
`azure-identity`,
`azure-storage-blobs`, `azure-storage-common`.
So the patch drops `opentelemetry-cpp` and the uAMQP C libraries from
`vcpkg.json`,
flips `DISABLE_AMQP` / `DISABLE_AZURE_CORE_OPENTELEMETRY` to `ON` and
trims the
sub-projects to the four Doris links; `build_azure` writes an overlay
triplet that sets
`VCPKG_BUILD_TYPE release`. What is left for vcpkg to build is curl,
libxml2, openssl
1.1.1n and zlib, release only.
## 3. ccache for the third-party build
`ENABLE_THIRDPARTY_CCACHE=ON` (off by default) exports
`CMAKE_C_COMPILER_LAUNCHER` /
`CMAKE_CXX_COMPILER_LAUNCHER`, which CMake initialises from the
environment, so no
cmake invocation in the script changes. It deliberately does not prefix
`CC`/`CXX`:
CMake splits `"ccache clang"` into the compiler plus a
`CMAKE_<LANG>_FLAGS` entry, and
that leaks into whatever a package exports. Autotools packages are left
alone.
`.github/workflows/build-thirdparty.yml` turns it on and carries the
cache with the
`ccache-action` this repository already vendors. Today a change to one
package
recompiles all ~80, at `-j 2` on two of the three jobs.
`max-size` is **1G** per job so the three together stay inside what is
left of the
repository's 10 GB Actions cache budget next to the 5G `BE-UT-macOS`
cache. Note that a
cache saved by a pull request run is only visible to that same pull
request, so this
pays off across pushes to one branch; sharing it across pull requests
would need this
workflow to run on master, the way `be-ut-mac.yml` uses a schedule for
exactly that
reason. Happy to drop this hunk if the cache budget is too tight.
## 4. Retry a failed vcpkg download instead of losing the azure build
All three jobs of run 32032837067 built 73 of the 74 packages and then
failed on the
74th, azure, in the same place: vcpkg downloading
`github.com/madler/zlib` - HTTP 429
on Linux and macOS, 500 on macOS-arm64, while the Actions cache service
was also
answering "Our services aren't available right now". azure is the only
package that
fetches its own sources (everything else goes through
`download-thirdparty.sh`, which
has a mirror and a fallback), its own retry is three attempts inside one
second, and it
is the last package built - so each job threw away a finished tree over
one file. And
the exposure just tripled now that azure builds on all three platforms.
So the configure is retried with a backoff long enough for a rate-limit
window to pass
(2, 4, 6, 8 minutes) and **only** when the failure was a download - a
port that will not
compile still stops on the first attempt. Ports that did build come back
from vcpkg's
binary cache, and `VCPKG_DOWNLOADS` now points outside the directory
`build_azure` wipes
on entry. `thirdparty/test/azure-vcpkg-retry-test.sh` drives
`build_azure` against a
stubbed cmake and covers both halves, in seconds, in the script-test
job.
## Also: the macOS jobs stalled for 10 minutes each
Carried over from apache/doris-thirdparty#410. thrift's configure finds
the dotnet the
runner images ship, so `make` runs `dotnet build -c Release` for
`lib/netstd`. The build
finishes in seconds and then the compiler server it leaves behind holds
the stdout it
inherited for its full keep-alive while nothing happens:
| runner | "Build succeeded" -> "Making install in compiler/cpp" |
|---|---|
| macos-14 | **600.03s** |
| macos-15-intel | **598.79s** |
| ubuntu-22.04 | 0.01s |
`UseSharedCompilation=false` and `MSBUILDDISABLENODEREUSE=1` stop the
server from being
started. Both macOS jobs in this workflow pay this today.
The other two fixes in apache/doris-thirdparty#410 do not apply here:
there is no Docker
job to decouple, and no macOS x86_64 job to build less often.
## Merge order
1. **This PR.** Its own CI is unaffected by either half of the split:
`env.sh` is
untouched, so BE still links no azure on macOS, and `build.sh` still
agrees with the
lifecycle test on the `hadoop_hdfs/` sentinel.
2. The apache/doris-thirdparty automation rebuilds the prebuilt
archives. It triggers on
`^thirdparty/` and takes ~3h; only then do the darwin-arm64 and
linux-aarch64
archives carry azure. The aarch64 build-env image has to be rebuilt too.
3. **#66908.**
Between 1 and 3, a Linux tree that picks up a freshly built third-party
archive will not
find `installed/lib/hadoop_hdfs/native/libhdfs.a`, which is still what
`build.sh`'s
`LAST_THIRDPARTY_LIB` looks for, and will start a full third-party
rebuild. Nothing in
CI consumes such an archive for a BE or cloud build - `be-ut-mac.yml` is
on Darwin,
where the sentinel is `libbrotlienc.a`, and `build-extension.yml` only
takes `thrift`
out of the Linux archive - but that is the window #66908 closes.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
This PR touches `thirdparty/**`, so its own `Build Third Party
Libraries` run
builds the whole tree on ubuntu-22.04, macos-15 and macos-14 - which is
exactly
where the hadoop change, the azure slimming and, for the first time,
azure on
macOS arm64 get exercised. **aarch64 Linux is the one path this workflow
does not
cover**; that is only built in apache/doris-thirdparty.
The azure patch was verified to apply cleanly to a pristine
`azure-sdk-for-cpp-azure-core_1.16.0` tarball, and the resulting
`vcpkg.json`
parses and declares only `curl`, `libxml2`, `openssl`, `wil`.
`actionlint` is
clean on the workflow, and the four `thirdparty/test/*.sh` scripts the
script-test
job runs pass.
---
.github/workflows/build-thirdparty.yml | 95 ++++++++-
thirdparty/CHANGELOG.md | 13 ++
thirdparty/build-thirdparty.sh | 219 ++++++++++++++++-----
.../azure-sdk-for-cpp-azure-core_1.16.0.patch | 155 +++++++++++++--
thirdparty/test/azure-vcpkg-retry-test.sh | 167 ++++++++++++++++
thirdparty/vars.sh | 7 -
6 files changed, 581 insertions(+), 75 deletions(-)
diff --git a/.github/workflows/build-thirdparty.yml
b/.github/workflows/build-thirdparty.yml
index 1fe25b781e8..e4f8bd4056c 100644
--- a/.github/workflows/build-thirdparty.yml
+++ b/.github/workflows/build-thirdparty.yml
@@ -80,8 +80,8 @@ jobs:
- name: Test focused thirdparty lifecycle
run: thirdparty/test/arrow-paimon-lifecycle-test.sh
- download_script_test:
- name: Download Script Test
+ script_test:
+ name: Thirdparty Script Test
needs: changes
if: ${{ needs.changes.outputs.thirdparty_changes == 'true' }}
runs-on: ubuntu-22.04
@@ -98,11 +98,20 @@ jobs:
thirdparty/test/juicefs-default-mirror-test.sh
thirdparty/test/download-thirdparty-md5-test.sh
+ # azure is the only package that fetches its own sources, and it is the
+ # last one built, so a download that fails there costs the whole tree.
+ # This drives build_azure against a stubbed cmake, in seconds.
+ - name: Test build-thirdparty.sh
+ run: |
+ thirdparty/test/azure-vcpkg-retry-test.sh
+
build_linux:
name: Build Third Party Libraries (Linux)
needs: changes
if: ${{ needs.changes.outputs.thirdparty_changes == 'true' }}
runs-on: ubuntu-22.04
+ env:
+ ENABLE_THIRDPARTY_CCACHE: 'ON'
steps:
- name: Checkout easimon/maximize-build-space
run: |
@@ -123,6 +132,24 @@ jobs:
- name: Checkout ${{ github.ref }}
uses: actions/checkout@v4
+ # ccache-action is vendored as a submodule and the plain checkout above
does not
+ # fetch it. Initialise only this one; `submodules: recursive` would also
clone
+ # contrib/clucene, which a third party build does not need.
+ - name: Checkout ccache-action
+ run: git submodule update --init .github/actions/ccache-action
+
+ # This build is cold every time, so changing one package recompiles all
~80 of
+ # them. build-thirdparty.sh routes the cmake-based packages through
ccache when
+ # ENABLE_THIRDPARTY_CCACHE is on. Note that a cache saved by a pull
request run
+ # is only visible to that same pull request, so this pays off across
pushes to
+ # one branch; sharing it across pull requests needs a run on master.
+ - name: Ccache ${{ github.ref }}
+ uses: ./.github/actions/ccache-action
+ with:
+ key: THIRDPARTY-Linux
+ max-size: "1G"
+ restore-keys: THIRDPARTY-Linux-
+
- name: Download
run: |
cd thirdparty
@@ -197,10 +224,42 @@ jobs:
needs: changes
if: ${{ needs.changes.outputs.thirdparty_changes == 'true' }}
runs-on: macos-15
+ env:
+ ENABLE_THIRDPARTY_CCACHE: 'ON'
+ # thrift's configure finds the dotnet these images ship, so `make` runs
+ # `dotnet build -c Release` for lib/netstd. The build itself takes
seconds, and
+ # then the compiler server it leaves behind holds the stdout it
inherited for its
+ # full 10 minute keep-alive while nothing happens. Measured in
+ # apache/doris-thirdparty run 31988123966: 600.03s on macos-14 and
598.79s on
+ # macos-15-intel between "Build succeeded" and "Making install in
compiler/cpp",
+ # against 0.01s on Linux. Do not start a shared compiler server, and do
not keep
+ # MSBuild worker nodes around either.
+ UseSharedCompilation: 'false'
+ MSBUILDDISABLENODEREUSE: '1'
+ DOTNET_CLI_TELEMETRY_OPTOUT: '1'
+ DOTNET_NOLOGO: '1'
steps:
- name: Checkout ${{ github.ref }}
uses: actions/checkout@v4
+ # ccache-action is vendored as a submodule and the plain checkout above
does not
+ # fetch it. Initialise only this one; `submodules: recursive` would also
clone
+ # contrib/clucene, which a third party build does not need.
+ - name: Checkout ccache-action
+ run: git submodule update --init .github/actions/ccache-action
+
+ # This build is cold every time, so changing one package recompiles all
~80 of
+ # them. build-thirdparty.sh routes the cmake-based packages through
ccache when
+ # ENABLE_THIRDPARTY_CCACHE is on. Note that a cache saved by a pull
request run
+ # is only visible to that same pull request, so this pays off across
pushes to
+ # one branch; sharing it across pull requests needs a run on master.
+ - name: Ccache ${{ github.ref }}
+ uses: ./.github/actions/ccache-action
+ with:
+ key: THIRDPARTY-macOS
+ max-size: "1G"
+ restore-keys: THIRDPARTY-macOS-
+
- name: Download
run: |
cd thirdparty
@@ -262,10 +321,42 @@ jobs:
needs: changes
if: ${{ needs.changes.outputs.thirdparty_changes == 'true' }}
runs-on: macos-14
+ env:
+ ENABLE_THIRDPARTY_CCACHE: 'ON'
+ # thrift's configure finds the dotnet these images ship, so `make` runs
+ # `dotnet build -c Release` for lib/netstd. The build itself takes
seconds, and
+ # then the compiler server it leaves behind holds the stdout it
inherited for its
+ # full 10 minute keep-alive while nothing happens. Measured in
+ # apache/doris-thirdparty run 31988123966: 600.03s on macos-14 and
598.79s on
+ # macos-15-intel between "Build succeeded" and "Making install in
compiler/cpp",
+ # against 0.01s on Linux. Do not start a shared compiler server, and do
not keep
+ # MSBuild worker nodes around either.
+ UseSharedCompilation: 'false'
+ MSBUILDDISABLENODEREUSE: '1'
+ DOTNET_CLI_TELEMETRY_OPTOUT: '1'
+ DOTNET_NOLOGO: '1'
steps:
- name: Checkout ${{ github.ref }}
uses: actions/checkout@v4
+ # ccache-action is vendored as a submodule and the plain checkout above
does not
+ # fetch it. Initialise only this one; `submodules: recursive` would also
clone
+ # contrib/clucene, which a third party build does not need.
+ - name: Checkout ccache-action
+ run: git submodule update --init .github/actions/ccache-action
+
+ # This build is cold every time, so changing one package recompiles all
~80 of
+ # them. build-thirdparty.sh routes the cmake-based packages through
ccache when
+ # ENABLE_THIRDPARTY_CCACHE is on. Note that a cache saved by a pull
request run
+ # is only visible to that same pull request, so this pays off across
pushes to
+ # one branch; sharing it across pull requests needs a run on master.
+ - name: Ccache ${{ github.ref }}
+ uses: ./.github/actions/ccache-action
+ with:
+ key: THIRDPARTY-macOS-arm64
+ max-size: "1G"
+ restore-keys: THIRDPARTY-macOS-arm64-
+
- name: Download
run: |
cd thirdparty
diff --git a/thirdparty/CHANGELOG.md b/thirdparty/CHANGELOG.md
index 8c049ea51bb..777789b1307 100644
--- a/thirdparty/CHANGELOG.md
+++ b/thirdparty/CHANGELOG.md
@@ -2,6 +2,19 @@
This file contains version of the third-party dependency libraries in the
build-env image. The docker build-env image is apache/doris, and the tag is
`build-env-${version}`
+## 20260817
+
+- Removed: hadoop-libs 3.3.6.6, along with the
`installed/{include,lib}/hadoop_hdfs/`
+ prefix it produced. hadoop-libs 3.4.2.4 under `hadoop_hdfs_3_4/` is the only
libhdfs
+ built now.
+- Modified: azure-core 1.16.0 is now built on aarch64 and macOS as well, and
only the
+ parts Doris links (azure-core, azure-identity, azure-storage-common,
+ azure-storage-blobs) are built. Its vcpkg dependency closure no longer
contains
+ opentelemetry-cpp, protobuf, abseil, utf8-range, uAMQP, and vcpkg builds the
+ remaining ports release-only. `DISABLE_THIRDPARTY_BUILD_AZURE=ON` skips the
package
+ during a third-party build; `DISABLE_BUILD_AZURE` now only decides whether
BE and
+ the cloud meta-service link it.
+
## 20260816
- Modified: hadoop-libs 3.4.2.3 -> 3.4.2.4
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 6f0676e565a..8d4b4b4ada8 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -56,6 +56,26 @@ if [[ -f "${DORIS_HOME}/env.sh" ]]; then
export DO_NOT_CHECK_JAVA_ENV=
fi
+# Optional ccache for the cmake-based packages. A full third-party build is
cold every
+# time, so a warm ccache turns a rebuild triggered by one changed package into
a few
+# minutes instead of hours - that is what this is for, and CI is where it pays
off.
+#
+# CMake initialises CMAKE_<LANG>_COMPILER_LAUNCHER from the environment
variables of the
+# same name, so this needs no change to the cmake invocations below. It is
also why this
+# does not go through CC/CXX: "ccache <compiler>" would land the compiler name
in
+# CMAKE_<LANG>_FLAGS and leak into whatever the package exports. Autotools
packages are
+# deliberately left alone. Off by default, since prefixing the compiler
changes how every
+# package configures itself.
+if [[ "${ENABLE_THIRDPARTY_CCACHE:-OFF}" == "ON" ]]; then
+ if ! command -v ccache &>/dev/null; then
+ echo "ENABLE_THIRDPARTY_CCACHE=ON, but ccache is not in PATH" >&2
+ exit 1
+ fi
+ export CMAKE_C_COMPILER_LAUNCHER='ccache'
+ export CMAKE_CXX_COMPILER_LAUNCHER='ccache'
+ echo "ccache is enabled for the cmake-based third-party packages"
+fi
+
# Check args
usage() {
echo "
@@ -64,6 +84,10 @@ Usage: $0 [options...] [packages...]
-j <num> build thirdparty parallel
--clean clean the extracted data
--continue <package> continue to build the remaining packages (starts
from the specified package)
+
+ Environment variables:
+ ENABLE_THIRDPARTY_CCACHE=ON compile the cmake-based packages
through ccache
+ DISABLE_THIRDPARTY_BUILD_AZURE=ON skip the azure-sdk-for-cpp package
"
exit 1
}
@@ -134,7 +158,13 @@ if [[ "${HELP}" -eq 1 ]]; then
usage
fi
-if [[ "$(echo "${DISABLE_BUILD_AZURE}" | tr '[:lower:]' '[:upper:]')" == "ON"
]]; then
+# Whether the third-party tree carries azure and whether Doris links it are two
+# separate questions. env.sh's DISABLE_BUILD_AZURE answers the second one for
BE and
+# the cloud meta-service, and it still defaults to ON on aarch64 and macOS,
where the
+# published prebuilt archives predate azure. Those archives have to grow the
libraries
+# before anything can link them, so the build does not read that switch;
+# DISABLE_THIRDPARTY_BUILD_AZURE opts the build itself out.
+if [[ "$(echo "${DISABLE_THIRDPARTY_BUILD_AZURE}" | tr '[:lower:]'
'[:upper:]')" == "ON" ]]; then
BUILD_AZURE='OFF'
fi
@@ -1904,24 +1934,6 @@ build_fast_float() {
cp -r ./include/fast_float "${TP_INSTALL_DIR}/include/"
}
-# hadoop_libs
-build_hadoop_libs() {
- check_if_source_exist "${HADOOP_LIBS_SOURCE}"
- cd "${TP_SOURCE_DIR}/${HADOOP_LIBS_SOURCE}"
- echo "THIRDPARTY_INSTALLED=${TP_INSTALL_DIR}" >env.sh
- ./build.sh
-
- rm -rf "${TP_INSTALL_DIR}/include/hadoop_hdfs/"
- rm -rf "${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
- mkdir -p "${TP_INSTALL_DIR}/include/hadoop_hdfs/"
- mkdir -p "${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
- cp -r ./hadoop-dist/target/hadoop-libhdfs-3.3.6/*
"${TP_INSTALL_DIR}/lib/hadoop_hdfs/"
- cp -r ./hadoop-dist/target/hadoop-libhdfs-3.3.6/include/hdfs.h
"${TP_INSTALL_DIR}/include/hadoop_hdfs/"
- rm -rf "${TP_INSTALL_DIR}/lib/hadoop_hdfs/native/*.a"
- find ./hadoop-dist/target/hadoop-3.3.6/lib/native/ -type f ! -name '*.a'
-exec cp {} "${TP_INSTALL_DIR}/lib/hadoop_hdfs/native/" \;
- find ./hadoop-dist/target/hadoop-3.3.6/lib/native/ -type l -exec cp -P {}
"${TP_INSTALL_DIR}/lib/hadoop_hdfs/native/" \;
-}
-
# hadoop_libs_3_4
build_hadoop_libs_3_4() {
check_if_source_exist "${HADOOP_LIBS_3_4_SOURCE}"
@@ -2043,30 +2055,150 @@ build_base64() {
# azure blob storage
build_azure() {
- if [[ "${BUILD_AZURE}" == "OFF" || "$(uname -s)" == 'Darwin' ]]; then
+ if [[ "${BUILD_AZURE}" == "OFF" ]]; then
echo "Skip build azure"
+ return
+ fi
+
+ check_if_source_exist "${AZURE_SOURCE}"
+ cd "${TP_SOURCE_DIR}/${AZURE_SOURCE}"
+ azure_dir="$(pwd)"
+
+ rm -rf "${BUILD_DIR}"
+ mkdir -p "${BUILD_DIR}"
+ cd "${BUILD_DIR}"
+
+ # We need use openssl 1.1.1n, which is already carried in
vcpkg-custom-ports
+ AZURE_PORTS="vcpkg-custom-ports"
+ AZURE_MANIFEST_DIR="."
+
+ local azure_machine_type
+ local vcpkg_arch
+ azure_machine_type="$(uname -m)"
+ case "${azure_machine_type}" in
+ aarch64 | arm64)
+ vcpkg_arch='arm64'
+ ;;
+ x86_64 | amd64)
+ vcpkg_arch='x64'
+ ;;
+ *)
+ echo "azure: unsupported machine type ${azure_machine_type}" >&2
+ exit 1
+ ;;
+ esac
+
+ # vcpkg builds every port twice, debug and release, and installs both.
Doris only
+ # ever links the release halves, and VCPKG_BUILD_TYPE - the only supported
way to
+ # ask for release only - can be set from a triplet file, so shadow the
built-in
+ # triplet with our own. Naming the file after the built-in triplet is what
makes
+ # the overlay take precedence.
+ local vcpkg_triplet
+ local vcpkg_triplet_dir="${PWD}/doris-vcpkg-triplets"
+ mkdir -p "${vcpkg_triplet_dir}"
+ if [[ "${KERNEL}" == 'Darwin' ]]; then
+ local vcpkg_osx_arch='x86_64'
+ if [[ "${vcpkg_arch}" == 'arm64' ]]; then vcpkg_osx_arch='arm64'; fi
+ vcpkg_triplet="${vcpkg_arch}-osx"
+ cat >"${vcpkg_triplet_dir}/${vcpkg_triplet}.cmake" <<EOF
+set(VCPKG_TARGET_ARCHITECTURE ${vcpkg_arch})
+set(VCPKG_CRT_LINKAGE dynamic)
+set(VCPKG_LIBRARY_LINKAGE static)
+set(VCPKG_CMAKE_SYSTEM_NAME Darwin)
+set(VCPKG_OSX_ARCHITECTURES ${vcpkg_osx_arch})
+set(VCPKG_BUILD_TYPE release)
+EOF
+ if [[ -n "${MACOSX_DEPLOYMENT_TARGET}" ]]; then
+ echo "set(VCPKG_OSX_DEPLOYMENT_TARGET
\"${MACOSX_DEPLOYMENT_TARGET}\")" \
+ >>"${vcpkg_triplet_dir}/${vcpkg_triplet}.cmake"
+ fi
else
- check_if_source_exist "${AZURE_SOURCE}"
- cd "${TP_SOURCE_DIR}/${AZURE_SOURCE}"
- azure_dir=$(pwd)
+ vcpkg_triplet="${vcpkg_arch}-linux"
+ cat >"${vcpkg_triplet_dir}/${vcpkg_triplet}.cmake" <<EOF
+set(VCPKG_TARGET_ARCHITECTURE ${vcpkg_arch})
+set(VCPKG_CRT_LINKAGE dynamic)
+set(VCPKG_LIBRARY_LINKAGE static)
+set(VCPKG_CMAKE_SYSTEM_NAME Linux)
+set(VCPKG_BUILD_TYPE release)
+EOF
+ fi
+
+ # vcpkg ships no prebuilt cmake/ninja/curl for aarch64 Linux, so it has to
reuse
+ # the ones already on PATH.
+ if [[ "${vcpkg_arch}" == 'arm64' && "${KERNEL}" != 'Darwin' ]]; then
+ export VCPKG_FORCE_SYSTEM_BINARIES=1
+ fi
+
+ # libcrypto.a needs dlopen/dlsym/dlclose/dlerror, and with clang
find_library may
+ # not turn up libdl, so ask for it explicitly. Apple has no libdl - those
symbols
+ # live in libSystem - and -ldl would fail the link there.
+ local azure_link_flags=()
+ if [[ "${KERNEL}" != 'Darwin' ]]; then
+ azure_link_flags=(-DCMAKE_EXE_LINKER_FLAGS="-ldl"
-DCMAKE_SHARED_LINKER_FLAGS="-ldl")
+ fi
+
+ # vcpkg fetches the sources of curl, libxml2, openssl and zlib from their
upstream
+ # hosts while cmake configures, and none of that goes through
download-thirdparty.sh
+ # and its mirror. Keep those tarballs outside "${BUILD_DIR}", which was
wiped above,
+ # so a retry - or a later run in the same tree - only fetches what the
attempt
+ # before it missed. "thirdparty/src*" is already gitignored.
+ VCPKG_DOWNLOADS="${TP_SOURCE_DIR}/vcpkg-downloads"
+ export VCPKG_DOWNLOADS
+ mkdir -p "${VCPKG_DOWNLOADS}"
+
+ # vcpkg's own retry is three attempts inside one second, which rides out
nothing:
+ # in apache/doris run 32032837067 all three jobs died on
+ # github.com/madler/zlib/archive/v1.3.1.tar.gz answering 429 (500 on macOS
arm64),
+ # and because azure is the last package this script builds, each of them
threw away
+ # a finished tree over one file. So wait out a rate limit window instead.
Only a
+ # download is worth waiting on - a port that will not compile, or a bad
option,
+ # fails identically every time - so the configure is retried on nothing
else, and
+ # the ports that did build come back from vcpkg's binary cache, which
lives outside
+ # this directory. cmake is piped through tee rather than redirected so
that a
+ # vcpkg install that takes twenty minutes still shows progress; this
script runs
+ # under `set -o pipefail`, so the status tested below is cmake's, not
tee's.
+ #
+ # DISABLE_AMQP and DISABLE_AZURE_CORE_OPENTELEMETRY are already the patched
+ # defaults; passing them here keeps the reason visible from the build
script.
+ local azure_attempt
+ local azure_attempts=5
+ local azure_backoff
+ local azure_log="${PWD}/doris-azure-configure.log"
+ for ((azure_attempt = 1; azure_attempt <= azure_attempts;
azure_attempt++)); do
+ if "${CMAKE_CMD}" -G "${GENERATOR}" -DCMAKE_POLICY_VERSION_MINIMUM=3.5
\
+ -DCMAKE_CXX_FLAGS="-Wno-maybe-uninitialized" \
+ "${azure_link_flags[@]}" \
+ -DVCPKG_TARGET_TRIPLET="${vcpkg_triplet}" \
+ -DVCPKG_OVERLAY_TRIPLETS="${vcpkg_triplet_dir}" \
+ -DDISABLE_RUST_IN_BUILD=ON -DDISABLE_AMQP=ON
-DDISABLE_AZURE_CORE_OPENTELEMETRY=ON \
+ -DBUILD_TESTING=OFF -DBUILD_SAMPLES=OFF
-DBUILD_PERFORMANCE_TESTS=OFF \
+ -DVCPKG_MANIFEST_MODE=ON
-DVCPKG_OVERLAY_PORTS="${azure_dir}/${AZURE_PORTS}"
-DVCPKG_MANIFEST_DIR="${azure_dir}/${AZURE_MANIFEST_DIR}"
-DWARNINGS_AS_ERRORS=FALSE -DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}"
-DCMAKE_BUILD_TYPE=Release .. 2>&1 | tee "${azure_log}"; then
+ break
+ fi
- rm -rf "${BUILD_DIR}"
- mkdir -p "${BUILD_DIR}"
- cd "${BUILD_DIR}"
+ if ! grep -qE 'Download failed, halting portfile|error: curl: \('
"${azure_log}"; then
+ echo "azure: cmake configure failed, and not on a download - see
above" >&2
+ exit 1
+ fi
- # We need use openssl 1.1.1n, which is already carried in
vcpkg-custom-ports
- AZURE_PORTS="vcpkg-custom-ports"
- AZURE_MANIFEST_DIR="."
+ if [[ "${azure_attempt}" -eq "${azure_attempts}" ]]; then
+ echo "azure: vcpkg could not download its sources in
${azure_attempts} attempts" >&2
+ exit 1
+ fi
- # Add -ldl for clang compatibility (libcrypto.a requires
dlopen/dlsym/dlclose/dlerror)
- "${CMAKE_CMD}" -G "${GENERATOR}" -DCMAKE_POLICY_VERSION_MINIMUM=3.5 \
- -DCMAKE_CXX_FLAGS="-Wno-maybe-uninitialized" \
- -DCMAKE_EXE_LINKER_FLAGS="-ldl" \
- -DCMAKE_SHARED_LINKER_FLAGS="-ldl" \
- -DDISABLE_RUST_IN_BUILD=ON -DVCPKG_MANIFEST_MODE=ON
-DVCPKG_OVERLAY_PORTS="${azure_dir}/${AZURE_PORTS}"
-DVCPKG_MANIFEST_DIR="${azure_dir}/${AZURE_MANIFEST_DIR}"
-DWARNINGS_AS_ERRORS=FALSE -DCMAKE_INSTALL_PREFIX="${TP_INSTALL_DIR}"
-DCMAKE_BUILD_TYPE=Release ..
- "${BUILD_SYSTEM}" -j "${PARALLEL}"
- "${BUILD_SYSTEM}" install
- fi
+ # A configure that died inside the vcpkg toolchain file leaves a cache
with no
+ # compiler in it, and cmake would report that instead of running vcpkg
again.
+ rm -rf CMakeCache.txt CMakeFiles
+
+ azure_backoff=$((azure_attempt * 120))
+ echo "azure: vcpkg could not download a source, retrying in
${azure_backoff}s" \
+ "(attempt $((azure_attempt + 1)) of ${azure_attempts})" >&2
+ sleep "${azure_backoff}"
+ done
+ rm -f "${azure_log}"
+
+ "${BUILD_SYSTEM}" -j "${PARALLEL}"
+ "${BUILD_SYSTEM}" install
}
# dragonbox
@@ -2372,14 +2504,10 @@ if [[ "${#packages[@]}" -eq 0 ]]; then
)
if [[ "$(uname -s)" == 'Darwin' ]]; then
read -r -a packages <<<"binutils gettext ${packages[*]}"
- # hadoop_libs, the 3.3.6 fork, stays Linux-only: it carries none of
the macOS fixes
- # apache/doris-thirdparty#407 made to the 3.4 fork, and nothing built
here reads its
- # hadoop_hdfs/ prefix - the cloud module, its only other consumer, is
Linux-only too.
- read -r -a packages <<<"${packages[*]} hadoop_libs_3_4"
- elif [[ "$(uname -s)" == 'Linux' ]]; then
- read -r -a packages <<<"${packages[*]} hadoop_libs"
- read -r -a packages <<<"${packages[*]} hadoop_libs_3_4"
fi
+ # hadoop_libs_3_4 runs last on every platform: its native build links
against
+ # what the packages above install into ${TP_INSTALL_DIR}.
+ read -r -a packages <<<"${packages[*]} hadoop_libs_3_4"
fi
# Map a package name to its source directory variable(s) and remove them to
free disk space.
@@ -2457,7 +2585,6 @@ cleanup_package_source() {
xxhash) src_var="XXHASH_SOURCE" ;;
concurrentqueue) src_var="CONCURRENTQUEUE_SOURCE" ;;
fast_float) src_var="FAST_FLOAT_SOURCE" ;;
- hadoop_libs) src_var="HADOOP_LIBS_SOURCE" ;;
hadoop_libs_3_4) src_var="HADOOP_LIBS_3_4_SOURCE" ;;
avx2neon) src_var="AVX2NEON_SOURCE" ;;
libdeflate) src_var="LIBDEFLATE_SOURCE" ;;
diff --git a/thirdparty/patches/azure-sdk-for-cpp-azure-core_1.16.0.patch
b/thirdparty/patches/azure-sdk-for-cpp-azure-core_1.16.0.patch
index fe0c9340c3b..1beb42ec68e 100644
--- a/thirdparty/patches/azure-sdk-for-cpp-azure-core_1.16.0.patch
+++ b/thirdparty/patches/azure-sdk-for-cpp-azure-core_1.16.0.patch
@@ -1,51 +1,166 @@
-From 0c18bd91955f6ad48582c01901ffadd4ad5a149c Mon Sep 17 00:00:00 2001
+From c155d4c6a000da7ae9a8e66dc7cfada8bf2bfc8a Mon Sep 17 00:00:00 2001
From: BiteTheDDDDt <[email protected]>
Date: Tue, 15 Jul 2025 12:57:25 +0800
-Subject: [PATCH 1/2] resolve missing uint8_t define
+Subject: [PATCH 1/3] resolve missing uint8_t define
---
.../azure-security-attestation/src/private/crypto/inc/crypto.hpp | 1 +
1 file changed, 1 insertion(+)
diff --git
a/sdk/attestation/azure-security-attestation/src/private/crypto/inc/crypto.hpp
b/sdk/attestation/azure-security-attestation/src/private/crypto/inc/crypto.hpp
-index 9a98f204..8f338436 100644
+index 9a98f20..8f33843 100644
---
a/sdk/attestation/azure-security-attestation/src/private/crypto/inc/crypto.hpp
+++
b/sdk/attestation/azure-security-attestation/src/private/crypto/inc/crypto.hpp
@@ -2,6 +2,7 @@
// Licensed under the MIT License.
-
+
#pragma once
+#include <cstdint>
#include <ctime>
#include <memory>
#include <string>
---
-2.43.5
+--
+2.50.1 (Apple Git-155)
+
-From 0c18bd91955f6ad48582c01901ffadd4ad5a149d Mon Sep 17 00:00:00 2001
-From: Claude <[email protected]>
-Date: Wed, 29 Jan 2026 16:00:00 +0800
-Subject: [PATCH 2/2] fix clang link error: always link libdl for openssl
+From 5f30d39de70fadcd1963a2c37fb64cc18068ea0a Mon Sep 17 00:00:00 2001
+From: Doris Thirdparty <[email protected]>
+Date: Thu, 29 Jan 2026 16:00:00 +0800
+Subject: [PATCH 2/3] link libdl for openssl on platforms that have it
-libcrypto.a requires dlopen/dlsym/dlclose/dlerror from libdl.
-With clang, find_library may not find libdl, but -ldl is still needed.
+libcrypto.a requires dlopen/dlsym/dlclose/dlerror from libdl. With clang,
+find_library may not find libdl, but -ldl is still needed. Apple has no libdl
+at all - those symbols live in libSystem - so -ldl must not be added there.
---
- vcpkg-custom-ports/openssl/vcpkg-cmake-wrapper.cmake.in | 3 ++-
- 1 file changed, 2 insertions(+), 1 deletion(-)
+ vcpkg-custom-ports/openssl/vcpkg-cmake-wrapper.cmake.in | 8 +++++++-
+ 1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/vcpkg-custom-ports/openssl/vcpkg-cmake-wrapper.cmake.in
b/vcpkg-custom-ports/openssl/vcpkg-cmake-wrapper.cmake.in
-index 1234567..abcdefg 100644
+index 4a5ee89..4211066 100644
--- a/vcpkg-custom-ports/openssl/vcpkg-cmake-wrapper.cmake.in
+++ b/vcpkg-custom-ports/openssl/vcpkg-cmake-wrapper.cmake.in
-@@ -53,7 +53,8 @@ if(OPENSSL_FOUND AND "@VCPKG_LIBRARY_LINKAGE@" STREQUAL
"static")
+@@ -53,7 +53,13 @@ if(OPENSSL_FOUND AND "@VCPKG_LIBRARY_LINKAGE@" STREQUAL
"static")
set_property(TARGET OpenSSL::SSL APPEND PROPERTY
INTERFACE_LINK_LIBRARIES "crypt32;ws2_32")
endif()
else()
- find_library(OPENSSL_DL_LIBRARY NAMES dl)
-+ # Always link dl for clang compatibility
-+ set(OPENSSL_DL_LIBRARY "dl")
++ # Always link dl for clang compatibility. Apple has no libdl at all -
dlopen
++ # and friends live in libSystem - so -ldl fails the link there.
++ if(APPLE)
++ set(OPENSSL_DL_LIBRARY "")
++ else()
++ set(OPENSSL_DL_LIBRARY "dl")
++ endif()
if(OPENSSL_DL_LIBRARY)
list(APPEND OPENSSL_LIBRARIES "dl")
if(TARGET OpenSSL::Crypto)
---
-2.43.5
+--
+2.50.1 (Apple Git-155)
+
+
+From 3318809e01a27b20a253dad916d48edc8c25b467 Mon Sep 17 00:00:00 2001
+From: Doris Thirdparty <[email protected]>
+Date: Mon, 17 Aug 2026 12:00:00 +0800
+Subject: [PATCH 3/3] build only the parts of the SDK that Doris links
+
+Doris links azure-core, azure-identity, azure-storage-common and
+azure-storage-blobs. Everything else the SDK builds by default is dead weight,
+and the opentelemetry-cpp dependency in vcpkg.json drags in protobuf, abseil
+and utf8-range, which dominate the build: 19 of the 26 minutes this package
+took in the apache/doris-thirdparty x86_64 build were spent on that closure.
+---
+ CMakeLists.txt | 19 ++++++++-----------
+ sdk/storage/CMakeLists.txt | 5 ++---
+ vcpkg.json | 18 ------------------
+ 3 files changed, 10 insertions(+), 32 deletions(-)
+
+diff --git a/CMakeLists.txt b/CMakeLists.txt
+index 433f22d..993fb4f 100644
+--- a/CMakeLists.txt
++++ b/CMakeLists.txt
+@@ -8,8 +8,12 @@ list(APPEND CMAKE_MODULE_PATH
"${CMAKE_CURRENT_SOURCE_DIR}/cmake-modules")
+
+ # Disable Rust based APIs in C++ SDK.
+ set(DISABLE_RUST_IN_BUILD OFF CACHE BOOL "Disable Rust based APIs in package")
+-set(DISABLE_AMQP OFF CACHE BOOL "Disable AMQP based functionality")
+-set(DISABLE_AZURE_CORE_OPENTELEMETRY OFF CACHE BOOL "Disable OpenTelemetry
based functionality")
++# Doris consumes azure-core, azure-identity and azure-storage-blobs only.
AMQP and the
++# OpenTelemetry tracing adapter are off by default here so that vcpkg.json
can drop
++# opentelemetry-cpp, uAMQP and their dependency closure (protobuf, abseil,
utf8-range),
++# which is by far the most expensive part of this build.
++set(DISABLE_AMQP ON CACHE BOOL "Disable AMQP based functionality")
++set(DISABLE_AZURE_CORE_OPENTELEMETRY ON CACHE BOOL "Disable OpenTelemetry
based functionality")
+
+ # The Rust OpenSSL library conflicts with the vcpkg based OpenSSL library,
which causes TLS operations to fail on Linux.
+ # For now, disabling rust functionality on Linux operating systems.
+@@ -177,18 +181,11 @@ if(BUILD_SAMPLES)
+ endif()
+
+ # sub-projects
++# Only the ones Doris links: appconfiguration, attestation, eventhubs,
keyvault,
++# tables and template are not built.
+ add_subdirectory(sdk/core)
+-add_subdirectory(sdk/appconfiguration)
+-add_subdirectory(sdk/attestation)
+-# AMQP doesn't work for UWP yet, and eventhubs depends on AMQP, so we cannot
include eventhubs on UWP.
+-if (NOT DISABLE_AMQP)
+- add_subdirectory(sdk/eventhubs)
+-endif()
+ add_subdirectory(sdk/identity)
+-add_subdirectory(sdk/keyvault)
+ add_subdirectory(sdk/storage)
+-add_subdirectory(sdk/template)
+-add_subdirectory(sdk/tables)
+
+ if(BUILD_SAMPLES)
+ add_subdirectory(samples/integration/vcpkg-all-smoke)
+diff --git a/sdk/storage/CMakeLists.txt b/sdk/storage/CMakeLists.txt
+index 55bc5d0..f199564 100644
+--- a/sdk/storage/CMakeLists.txt
++++ b/sdk/storage/CMakeLists.txt
+@@ -5,8 +5,7 @@ cmake_minimum_required (VERSION 3.13)
+
+ project (azure-storage LANGUAGES CXX)
+
++# Doris links azure-storage-common and azure-storage-blobs only; datalake,
file
++# shares and queues are not built.
+ add_subdirectory(azure-storage-common)
+ add_subdirectory(azure-storage-blobs)
+-add_subdirectory(azure-storage-files-datalake)
+-add_subdirectory(azure-storage-files-shares)
+-add_subdirectory(azure-storage-queues)
+diff --git a/vcpkg.json b/vcpkg.json
+index 23e6f7c..c697e03 100644
+--- a/vcpkg.json
++++ b/vcpkg.json
+@@ -14,27 +14,9 @@
+ {
+ "name": "openssl"
+ },
+- {
+- "name": "opentelemetry-cpp",
+- "features": ["otlp-http"],
+- "platform": "!(windows & !static)",
+- "version>=": "1.3.0"
+- },
+ {
+ "name": "wil",
+ "platform": "windows"
+- },
+- {
+- "name": "azure-c-shared-utility",
+- "platform": "!uwp"
+- },
+- {
+- "name": "azure-macro-utils-c",
+- "platform": "!uwp"
+- },
+- {
+- "name": "umock-c",
+- "platform": "!uwp"
+ }
+ ],
+ "overrides": [
+--
+2.50.1 (Apple Git-155)
+
diff --git a/thirdparty/test/azure-vcpkg-retry-test.sh
b/thirdparty/test/azure-vcpkg-retry-test.sh
new file mode 100755
index 00000000000..8719b7ae128
--- /dev/null
+++ b/thirdparty/test/azure-vcpkg-retry-test.sh
@@ -0,0 +1,167 @@
+#!/usr/bin/env bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# azure is the last package build-thirdparty.sh builds, and it is the only one
+# that fetches sources itself: vcpkg downloads curl, libxml2, openssl and zlib
+# while cmake configures, outside download-thirdparty.sh and its mirror. In
+# apache/doris run 32032837067 all three jobs spent between 1h26m and 2h19m
+# building the other 73 packages and then threw the tree away because
+# github.com/madler/zlib/archive/v1.3.1.tar.gz answered 429. This test pins the
+# two halves of the answer to that: a failed download is waited out, and any
+# other failure still stops on the first attempt rather than repeating a broken
+# build four more times.
+
+set -eo pipefail
+
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." &>/dev/null && pwd)"
+
+fail() {
+ echo "FAIL: $*" >&2
+ exit 1
+}
+
+assert_eq() {
+ local expected="$1"
+ local actual="$2"
+ local what="$3"
+ [[ "${actual}" == "${expected}" ]] || fail "${what}: expected
'${expected}', got '${actual}'"
+}
+
+tmpdir="$(mktemp -d)"
+trap 'rm -rf "${tmpdir}"' EXIT
+
+# build-thirdparty.sh downloads and compiles the whole tree before it defines a
+# single build_* function, so the function under test is lifted out of it
rather
+# than sourced. Extracting it keeps the test honest: it runs the same text that
+# ships, and it breaks loudly if build_azure is renamed or reindented.
+azure_fn="${tmpdir}/build_azure.sh"
+sed -n '/^build_azure() {$/,/^}$/p' "${ROOT}/build-thirdparty.sh"
>"${azure_fn}"
+[[ -s "${azure_fn}" ]] || fail "could not extract build_azure() from
build-thirdparty.sh"
+grep -q '^}$' "${azure_fn}" || fail "extracted build_azure() is not terminated"
+
+# A cmake that fails the first ${CMAKE_FAILURES} times it is called, the way
+# vcpkg reports whatever CMAKE_FAILURE_KIND asks for, and counts its calls.
+cmake_stub="${tmpdir}/cmake"
+cat >"${cmake_stub}" <<'EOF'
+#!/usr/bin/env bash
+set -eo pipefail
+calls="$(cat "${CMAKE_CALL_COUNT}" 2>/dev/null || echo 0)"
+calls=$((calls + 1))
+echo "${calls}" >"${CMAKE_CALL_COUNT}"
+if [[ "${calls}" -gt "${CMAKE_FAILURES}" ]]; then
+ echo '-- Configuring done'
+ exit 0
+fi
+if [[ "${CMAKE_FAILURE_KIND}" == 'download' ]]; then
+ echo 'error: curl: (22) The requested URL returned error: 429'
+ echo 'CMake Error at scripts/cmake/vcpkg_download_distfile.cmake:136
(message):'
+ echo ' Download failed, halting portfile.'
+else
+ echo 'CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage'
+fi
+exit 1
+EOF
+chmod +x "${cmake_stub}"
+
+ninja_stub="${tmpdir}/ninja"
+printf '#!/usr/bin/env bash\nexit 0\n' >"${ninja_stub}"
+chmod +x "${ninja_stub}"
+
+# Everything build_azure reads that the script normally sets up around it.
+export BUILD_AZURE='ON'
+export AZURE_SOURCE='azure-sdk-for-cpp-azure-core_1.16.0'
+export BUILD_DIR='doris_build'
+export TP_SOURCE_DIR="${tmpdir}/src"
+export TP_INSTALL_DIR="${tmpdir}/installed"
+export CMAKE_CMD="${cmake_stub}"
+export BUILD_SYSTEM="${ninja_stub}"
+export GENERATOR='Ninja'
+export PARALLEL=1
+KERNEL="$(uname -s)"
+export KERNEL
+export CMAKE_CALL_COUNT="${tmpdir}/cmake-calls"
+mkdir -p "${TP_SOURCE_DIR}/${AZURE_SOURCE}/vcpkg-custom-ports"
"${TP_INSTALL_DIR}"
+
+# shellcheck source=/dev/null
+. "${azure_fn}"
+
+# The script checks the unpacked source is there; the fixture above is enough.
+check_if_source_exist() { :; }
+
+SLEEP_LOG="${tmpdir}/sleeps"
+recorded_downloads="${tmpdir}/vcpkg-downloads"
+
+# build_azure calls exit on a failure it will not retry, so each case runs in a
+# subshell and reports back through files. The backoff is minutes by design;
the
+# stub records what it was asked to wait instead of waiting.
+run_case() {
+ export CMAKE_FAILURES="$1"
+ export CMAKE_FAILURE_KIND="$2"
+ rm -f "${CMAKE_CALL_COUNT}" "${SLEEP_LOG}"
+ : >"${SLEEP_LOG}"
+ status=0
+ (
+ # Called by build_azure, and VCPKG_DOWNLOADS is set by it.
+ # shellcheck disable=SC2329
+ sleep() { printf '%s\n' "$1" >>"${SLEEP_LOG}"; }
+ build_azure
+ # shellcheck disable=SC2153,SC2154
+ printf '%s\n' "${VCPKG_DOWNLOADS}" >"${recorded_downloads}"
+ ) >"${tmpdir}/out.log" 2>&1 || status=$?
+ calls="$(cat "${CMAKE_CALL_COUNT}" 2>/dev/null || echo 0)"
+ sleeps="$(awk 'END { print NR }' "${SLEEP_LOG}")"
+}
+
+# 1. Two failed downloads then a good one: build_azure rides it out.
+run_case 2 download
+assert_eq 0 "${status}" 'a download that recovers should not fail the build'
+assert_eq 3 "${calls}" 'cmake should be retried until the download works'
+assert_eq 2 "${sleeps}" 'each retry should back off first'
+first_backoff="$(sed -n '1p' "${SLEEP_LOG}")"
+second_backoff="$(sed -n '2p' "${SLEEP_LOG}")"
+[[ "${first_backoff}" -lt "${second_backoff}" ]] ||
+ fail "backoff should grow, got ${first_backoff} then ${second_backoff}"
+
+# 2. vcpkg keeps its downloads outside the build directory, which build_azure
+# wipes on entry, so a rerun does not refetch what already landed.
+vcpkg_downloads="$(cat "${recorded_downloads}")"
+[[ -n "${vcpkg_downloads}" ]] || fail 'VCPKG_DOWNLOADS should be set for vcpkg'
+[[ -d "${vcpkg_downloads}" ]] || fail "VCPKG_DOWNLOADS ${vcpkg_downloads}
should exist"
+case "${vcpkg_downloads}" in
+*"/${BUILD_DIR}/"* | *"/${BUILD_DIR}")
+ fail "VCPKG_DOWNLOADS ${vcpkg_downloads} is inside the wiped build
directory"
+ ;;
+*) ;;
+esac
+
+# 3. A failure that is not a download stops on the first attempt. Repeating a
+# configure that cannot work only buries the error under four more copies.
+run_case 99 hard
+[[ "${status}" -ne 0 ]] || fail 'a broken configure should fail the build'
+assert_eq 1 "${calls}" 'a non-download failure should not be retried'
+assert_eq 0 "${sleeps}" 'a non-download failure should not sleep'
+grep -q 'not on a download' "${tmpdir}/out.log" ||
+ fail 'the build should say why it did not retry'
+
+# 4. A download that never recovers gives up, but only after it has waited.
+run_case 99 download
+[[ "${status}" -ne 0 ]] || fail 'an unreachable source should fail the build'
+[[ "${calls}" -ge 3 ]] || fail "expected several attempts, got ${calls}"
+assert_eq "$((calls - 1))" "${sleeps}" 'every attempt but the first should
follow a sleep'
+
+echo 'PASS: build_azure waits out a failed vcpkg download and fails fast on
anything else'
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 6a42bd00cf7..951d1b9d786 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -498,12 +498,6 @@ FAST_FLOAT_SOURCE=fast_float-3.9.0
FAST_FLOAT_MD5SUM="5656b0d8b150a3b157cfb092d214f6ea"
# libhdfs
-HADOOP_LIBS_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.3.6.6-for-doris.tar.gz";
-HADOOP_LIBS_NAME="hadoop-3.3.6.6-for-doris.tar.gz"
-HADOOP_LIBS_SOURCE="doris-thirdparty-hadoop-3.3.6.6-for-doris"
-HADOOP_LIBS_MD5SUM="13b66d5f2abffd1740e692b65df5962e"
-
-# libhdfs 3.4
HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.4-for-doris.tar.gz";
HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.4-for-doris.tar.gz"
HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.4-for-doris"
@@ -664,7 +658,6 @@ export TP_ARCHIVES=(
'XXHASH'
'CONCURRENTQUEUE'
'FAST_FLOAT'
- 'HADOOP_LIBS'
'HADOOP_LIBS_3_4'
'AVX2NEON'
'LIBDEFLATE'
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
