This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 02fbf377c16 [chore](build) move hadoop_hdfs consumers onto 3.4.2.4 and
link azure on aarch64/macOS (#66908)
02fbf377c16 is described below
commit 02fbf377c16816f902cb969ed603248a86631491
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Wed Aug 19 21:01:03 2026 +0800
[chore](build) move hadoop_hdfs consumers onto 3.4.2.4 and link azure on
aarch64/macOS (#66908)
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66842
Problem Summary:
**Second half of a two-PR split. Do not merge before #66842 has landed
*and* the
prebuilt third-party archives have been rebuilt from it** - see "Merge
order" at the
bottom. #66842 carries the third-party build changes; this PR carries
the consumers.
## 1. Move the remaining `hadoop_hdfs/` consumers onto
`hadoop_hdfs_3_4/`
The tree carried two libhdfs builds - 3.3.6.6 under
`installed/{include,lib}/hadoop_hdfs/`
and 3.4.2.4 under `.../hadoop_hdfs_3_4/` - and the two prefixes were
mixed up on the
consumer side:
| consumer | header | library |
|---|---|---|
| `be/src/io/fs/hdfs.h` | `hadoop_hdfs_3_4/hdfs.h` (3.4.2.4) | — |
| `be/src/io/hdfs_builder.cpp` | `hadoop_hdfs/hdfs.h` (**3.3.6.6**) | —
|
| `be/CMakeLists.txt` | — | `hadoop_hdfs_3_4/native/libhdfs.a` (3.4.2.4)
|
| `cloud/src/recycler/hdfs_accessor.{h,cpp}` | `hadoop_hdfs/hdfs.h`
(**3.3.6.6**) | — |
| `cloud/CMakeLists.txt` | — | `hadoop_hdfs/native/libhdfs.a`
(**3.3.6.6**) |
Both headers share the `LIBHDFS_HDFS_H` include guard, so
`hdfs_builder.cpp` compiled
against the 3.3.6.6 header (it came first) while BE linked the 3.4.2.4
archive.
So `cloud/`, `build.sh`'s `LAST_THIRDPARTY_LIB` sentinel,
`run-be-ut.sh`,
`run-cloud-ut.sh` and the thirdparty lifecycle test move to
`hadoop_hdfs_3_4/`, and
`be/src/io/hdfs_builder.cpp` drops its 3.3.6.6 include - `io/fs/hdfs.h`,
two lines
below it, already re-exports the 3.4.2.4 one. The unit-test runners
still stage the
jars at `lib/hadoop_hdfs/`, so the classpath loops below them are
unchanged.
Both prefixes exist in every third-party archive built before #66842, so
this half is
safe against the images in use today. It is #66842 - which stops
building 3.3.6.6 at
all - that makes it mandatory.
## 2. Link azure on aarch64 and macOS
`env.sh` forced `DISABLE_BUILD_AZURE=ON` on aarch64 and macOS, so BE and
the cloud
meta-service lost `+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` there. #66842
made the
third-party recipe portable (`-ldl` does not exist on Apple; vcpkg ships
no prebuilt
tools for aarch64 Linux) and now builds azure on every platform. This
drops the
platform test so the consumers link it. `DISABLE_BUILD_AZURE=ON` still
opts out.
## Merge order
1. #66842 lands.
2. The apache/doris-thirdparty automation rebuilds the prebuilt archives
- it triggers
on `^thirdparty/` and takes ~3h - and the aarch64 build-env image is
rebuilt.
3. This PR lands.
Landing this one first fails at the BE link with
`ninja: error: '.../installed/lib/libazure-core.a' ... missing and no
known rule to
make it`, which is exactly what the BE UT (macOS) job of the un-split PR
hit.
### Release note
Azure Blob Storage support (`+AZURE_BLOB`, `+AZURE_STORAGE_VAULT`) is
now built on
aarch64 and macOS, not just x86_64 Linux.
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
`thirdparty/test/arrow-paimon-lifecycle-test.sh` covers the `build.sh`
sentinel
move; it is what the Arrow/Paimon Lifecycle Test job runs. The azure
half is
exercised by every BE/cloud build in this PR's own CI, against a
prebuilt that
by then carries the libraries.
- Behavior changed:
- [x] Yes.
- Azure is now built and linked on aarch64 and macOS. **The third-party
prebuilt
has to be rebuilt from #66842 before this lands**: BE/cloud on those
platforms
will pass `-DBUILD_AZURE=ON` and fail to link against an older archive
that has
no azure libraries. `DISABLE_BUILD_AZURE=ON` still opts out.
- Anything outside this repository reading the `hadoop_hdfs/` prefix
needs
updating; only `hadoop_hdfs_3_4/` remains after #66842.
- Does this need documentation?
- [x] No.
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Co-authored-by: daidai <[email protected]>
---
be/src/io/hdfs_builder.cpp | 3 -
build.sh | 2 +-
cloud/CMakeLists.txt | 2 +-
cloud/src/recycler/hdfs_accessor.cpp | 2 +-
cloud/src/recycler/hdfs_accessor.h | 2 +-
env.sh | 8 +-
regression-test/pipeline/common/github-utils.sh | 24 ++++-
run-be-ut.sh | 4 +-
run-cloud-ut.sh | 4 +-
thirdparty/CHANGELOG.md | 9 ++
thirdparty/arrow-paimon-vars.sh | 119 +++++++++++++++++++----
thirdparty/build-thirdparty.sh | 15 +++
thirdparty/test/arrow-paimon-lifecycle-test.sh | 123 +++++++++++++++++++++++-
13 files changed, 275 insertions(+), 42 deletions(-)
diff --git a/be/src/io/hdfs_builder.cpp b/be/src/io/hdfs_builder.cpp
index cb9c75fb29d..85248d178d0 100644
--- a/be/src/io/hdfs_builder.cpp
+++ b/be/src/io/hdfs_builder.cpp
@@ -28,9 +28,6 @@
#include "common/config.h"
#include "common/kerberos/kerberos_ticket_mgr.h"
#include "common/logging.h"
-#ifdef USE_HADOOP_HDFS
-#include "hadoop_hdfs/hdfs.h"
-#endif
#include "io/fs/hdfs.h"
#include "runtime/exec_env.h"
#include "util/string_util.h"
diff --git a/build.sh b/build.sh
index f2d476c04ce..7168a355d68 100755
--- a/build.sh
+++ b/build.sh
@@ -482,7 +482,7 @@ fi
if [[ "${TARGET_SYSTEM}" == 'Darwin' ]]; then
LAST_THIRDPARTY_LIB='libbrotlienc.a'
else
- LAST_THIRDPARTY_LIB='hadoop_hdfs/native/libhdfs.a'
+ LAST_THIRDPARTY_LIB='hadoop_hdfs_3_4/native/libhdfs.a'
fi
# The final-library sentinel only proves that some third-party build
completed. It cannot
diff --git a/cloud/CMakeLists.txt b/cloud/CMakeLists.txt
index d4e0b346659..e0b44d9cdff 100644
--- a/cloud/CMakeLists.txt
+++ b/cloud/CMakeLists.txt
@@ -302,7 +302,7 @@ set(WL_END_GROUP "-Wl,--end-group")
# Set Doris libraries
if ((ARCH_AMD64 OR ARCH_AARCH64) AND OS_LINUX)
add_library(hadoop_hdfs STATIC IMPORTED)
- set_target_properties(hadoop_hdfs PROPERTIES IMPORTED_LOCATION
${THIRDPARTY_DIR}/lib/hadoop_hdfs/native/libhdfs.a)
+ set_target_properties(hadoop_hdfs PROPERTIES IMPORTED_LOCATION
${THIRDPARTY_DIR}/lib/hadoop_hdfs_3_4/native/libhdfs.a)
set(COMMON_THIRDPARTY
${COMMON_THIRDPARTY}
diff --git a/cloud/src/recycler/hdfs_accessor.cpp
b/cloud/src/recycler/hdfs_accessor.cpp
index 8de638f81d8..4fddeab693f 100644
--- a/cloud/src/recycler/hdfs_accessor.cpp
+++ b/cloud/src/recycler/hdfs_accessor.cpp
@@ -24,7 +24,7 @@
#include "recycler/util.h"
#ifdef USE_HADOOP_HDFS
-#include <hadoop_hdfs/hdfs.h> // IWYU pragma: export
+#include <hadoop_hdfs_3_4/hdfs.h> // IWYU pragma: export
#else
#include <hdfs/hdfs.h> // IWYU pragma: export
#endif
diff --git a/cloud/src/recycler/hdfs_accessor.h
b/cloud/src/recycler/hdfs_accessor.h
index b8d6b16b53f..7eb182f9122 100644
--- a/cloud/src/recycler/hdfs_accessor.h
+++ b/cloud/src/recycler/hdfs_accessor.h
@@ -18,7 +18,7 @@
#pragma once
#ifdef USE_HADOOP_HDFS
-#include <hadoop_hdfs/hdfs.h> // IWYU pragma: export
+#include <hadoop_hdfs_3_4/hdfs.h> // IWYU pragma: export
#else
#include <hdfs/hdfs.h> // IWYU pragma: export
#endif
diff --git a/env.sh b/env.sh
index d5370195b05..510b52a224b 100755
--- a/env.sh
+++ b/env.sh
@@ -106,12 +106,10 @@ if [[ -z "${DORIS_THIRDPARTY}" ]]; then
fi
# set DISABLE_BUILD_AZURE
+# Azure is built on every platform now. It used to be skipped on aarch64 and
macOS
+# because the third-party recipe was x86_64-Linux-only; that is fixed in
build_azure.
if [[ -z "${DISABLE_BUILD_AZURE}" ]]; then
- if [[ "${TARGET_ARCH}" == *arm* || "${TARGET_ARCH}" == "aarch64" ||
"${TARGET_SYSTEM}" == 'Darwin' ]]; then
- export DISABLE_BUILD_AZURE='ON'
- else
- export DISABLE_BUILD_AZURE='OFF'
- fi
+ export DISABLE_BUILD_AZURE='OFF'
fi
# check python
diff --git a/regression-test/pipeline/common/github-utils.sh
b/regression-test/pipeline/common/github-utils.sh
index 048b282701e..b866b5d82a4 100644
--- a/regression-test/pipeline/common/github-utils.sh
+++ b/regression-test/pipeline/common/github-utils.sh
@@ -440,14 +440,36 @@ github_utils__maybe_enable_external_stage_timer() {
github_utils__maybe_enable_external_stage_timer
+# Paths under thirdparty/ that produce nothing in thirdparty/installed/.
+# Changing only these must not force the pipeline into a full thirdparty
+# source build, which costs more than an hour of build time for nothing.
+# This is a denylist on purpose: any new path under thirdparty/ that is not
+# listed here still triggers a rebuild, so the unknown case stays on the safe
+# side.
+_thirdparty_change_affects_artifacts() {
+ local af="$1"
+ if [[ "${af}" == 'thirdparty/test/'* ]] ||
+ [[ "${af}" == 'thirdparty/CHANGELOG.md' ]] ||
+ [[ "${af}" == 'thirdparty/LICENSE.txt' ]] ||
+ [[ "${af}" == 'thirdparty/arrow-paimon-vars.sh' ]] ||
+ [[ "${af}" == 'thirdparty/build-thirdparty.sh' ]]; then
+ return 1
+ fi
+ return 0
+}
+
file_changed_thirdparty() {
local all_files
all_files=$(cat all_files)
if [[ -z ${all_files} ]]; then echo "Failed to get pr changed files." &&
return 0; fi
for af in ${all_files}; do
if [[ "${af}" == 'thirdparty/'* ]]; then
+ if ! _thirdparty_change_affects_artifacts "${af}"; then
+ echo "thirdparty file ${af} does not affect built artifacts,
skip it"
+ continue
+ fi
echo "thirdparty changed" && return 0
fi
done
echo "thirdparty not changed" && return 1
-}
\ No newline at end of file
+}
diff --git a/run-be-ut.sh b/run-be-ut.sh
index 7eeece8329e..d04af7036ed 100755
--- a/run-be-ut.sh
+++ b/run-be-ut.sh
@@ -435,8 +435,8 @@ touch "${UT_TMP_DIR}/tmp_file"
LIB_DIR="${DORIS_TEST_BINARY_DIR}/lib/"
rm -rf "${LIB_DIR}"
mkdir "${LIB_DIR}"
-if [[ -d "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs/" ]]; then
- cp -r "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs/" "${LIB_DIR}"
+if [[ -d "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs_3_4/" ]]; then
+ cp -r "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs_3_4/"
"${LIB_DIR}/hadoop_hdfs"
fi
if [[ -f "${DORIS_HOME}/output/be/lib/java-udf-jar-with-dependencies.jar" ]];
then
cp "${DORIS_HOME}/output/be/lib/java-udf-jar-with-dependencies.jar"
"${LIB_DIR}/"
diff --git a/run-cloud-ut.sh b/run-cloud-ut.sh
index 0b467382be0..d74dcb122b2 100755
--- a/run-cloud-ut.sh
+++ b/run-cloud-ut.sh
@@ -289,8 +289,8 @@ mkdir -p "${CMAKE_BUILD_DIR}/test/log"
LIB_DIR="${CMAKE_BUILD_DIR}/test/lib"
rm -rf "${LIB_DIR}"
mkdir "${LIB_DIR}"
-if [[ -d "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs/" ]]; then
- cp -r "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs/" "${LIB_DIR}"
+if [[ -d "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs_3_4/" ]]; then
+ cp -r "${DORIS_THIRDPARTY}/installed/lib/hadoop_hdfs_3_4/"
"${LIB_DIR}/hadoop_hdfs"
fi
if [[ "${RUN}" -ne 1 ]]; then
diff --git a/thirdparty/CHANGELOG.md b/thirdparty/CHANGELOG.md
index 777789b1307..b483a7ee34e 100644
--- a/thirdparty/CHANGELOG.md
+++ b/thirdparty/CHANGELOG.md
@@ -2,6 +2,15 @@
This file contains version of the third-party dependency libraries in the
build-env image. The docker build-env image is apache/doris, and the tag is
`build-env-${version}`
+## 20260819
+
+- Added: `installed/include/hadoop_hdfs` and `installed/lib/hadoop_hdfs` are
symlinks to
+ their `hadoop_hdfs_3_4` counterparts, so a tree that still includes
+ `<hadoop_hdfs/hdfs.h>` or links `lib/hadoop_hdfs/native/libhdfs.a` -
branch-3.0,
+ branch-3.1, the cloud module, anything outside this repository - builds
against
+ 3.4.2.4 rather than failing to find the prefix. This is a compatibility shim
over a
+ single build, not a second one; drop it once nothing reads the old prefix.
+
## 20260817
- Removed: hadoop-libs 3.3.6.6, along with the
`installed/{include,lib}/hadoop_hdfs/`
diff --git a/thirdparty/arrow-paimon-vars.sh b/thirdparty/arrow-paimon-vars.sh
index 1091c4fc84a..e6f8cfb7c6f 100644
--- a/thirdparty/arrow-paimon-vars.sh
+++ b/thirdparty/arrow-paimon-vars.sh
@@ -45,6 +45,36 @@ PAIMON_CPP_NAME="paimon-cpp-0a4f4e2.tar.gz"
PAIMON_CPP_SOURCE="doris-thirdparty-paimon-cpp-0a4f4e2"
PAIMON_CPP_MD5SUM="b8599a0421dbf1ec05e2f1a481d64e87"
+# Bump the corresponding schema version whenever output-affecting build
options or
+# helper behavior in build_arrow() or build_paimon_cpp() changes. The
fingerprints
+# below intentionally describe only this component stack so master and release
+# branches can reuse the same shared prebuilt when their semantic inputs match.
+ARROW_BUILD_SCHEMA_VERSION="1"
+PAIMON_BUILD_SCHEMA_VERSION="1"
+
+# The shared automation prebuilt is still published from master, which
computes the
+# former whole-script fingerprint over env.sh, vars.sh, download-thirdparty.sh
and
+# build-thirdparty.sh. Every republication triggered by an unrelated
thirdparty edit
+# therefore stamps a different marker, so list each one still in circulation,
newest
+# first. These markers are only consulted while the selected inputs still hash
to the
+# pinned semantic fingerprint, and version plus complete artifact validation
stay
+# mandatory before reuse. Delete the lists once master has published a
prebuilt that
+# carries the semantic marker itself.
+ARROW_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT="ab79ab0bbfbf93f9860050fb751b20fee9e40d96"
+PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT="cb82e41ba46f534e611cdd52e66b53c227d49bf8"
+ARROW_LEGACY_BUILD_FINGERPRINTS=(
+ # published 2026-08-19, master after apache/doris#66842
+ 8356a3a464b93f3a59a6bdae6a51258947ab35d2
+ # published 2026-08-05, master after apache/doris#66221
+ 9d03645dd1cded5184a8126f5c7f4a6eb9b92b53
+)
+PAIMON_LEGACY_BUILD_FINGERPRINTS=(
+ # published 2026-08-19, master after apache/doris#66842
+ 2bbf52e719bdbc8aaa428caab200ac13848f92e5
+ # published 2026-08-05, master after apache/doris#66221
+ dbb6ca6e243cb3aa783b7a8011f97afda9e7ea28
+)
+
# Arrow consumes xsimd and Brotli as bundled source archives, but neither is a
# build target in the focused Arrow/Paimon recovery path.
ARROW_PAIMON_BUILD_PACKAGES=(arrow paimon_cpp)
@@ -82,23 +112,47 @@ prepare_arrow_paimon_download_packages() {
done
}
-# Identify the checked-in source, patch, and build inputs selected for Arrow.
-# Arrow and Paimon publish separate installed markers so a package-only build
-# cannot certify a component that it did not rebuild.
+# Print stable path-and-content records for fingerprint inputs. Including the
path
+# makes patch selection and ordering part of the contract, not only file
contents.
+arrow_paimon_fingerprint_files() {
+ local file
+ local blob
+
+ # Feed the sorted list through a pipe rather than a process substitution:
+ # build.sh is documented as "sh build.sh", and bash 3.2 invoked as sh
+ # (macOS /bin/sh) rejects "< <(...)" at parse time, which would abort
+ # every build before any component is compiled.
+ printf '%s\n' "$@" | LC_ALL=C sort | while IFS= read -r file; do
+ blob="$(git hash-object "${file}")" || exit 1
+ printf 'file=%s\n' "${file}"
+ printf 'blob=%s\n' "${blob}"
+ done
+}
+
+# Identify only the source, patch, and explicit build-schema inputs selected
for
+# Arrow. Arrow and Paimon publish separate installed markers so a package-only
+# build cannot certify a component that it did not rebuild.
arrow_build_fingerprint() {
local vars_dir
vars_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
(
+ set -o pipefail
cd "${vars_dir}" || return 1
- LC_ALL=C
- git hash-object \
- ../env.sh \
- arrow-paimon-vars.sh \
- vars.sh \
- download-thirdparty.sh \
- build-thirdparty.sh \
- patches/apache-arrow-"${ARROW_VERSION}"-*.patch |
- git hash-object --stdin
+ {
+ printf 'schema=%s\n' "${ARROW_BUILD_SCHEMA_VERSION}"
+ printf 'ARROW_VERSION=%s\n' "${ARROW_VERSION}"
+ printf 'ARROW_NAME=%s\n' "${ARROW_NAME}"
+ printf 'ARROW_SOURCE=%s\n' "${ARROW_SOURCE}"
+ printf 'ARROW_MD5SUM=%s\n' "${ARROW_MD5SUM}"
+ printf 'BROTLI_NAME=%s\n' "${BROTLI_NAME}"
+ printf 'BROTLI_SOURCE=%s\n' "${BROTLI_SOURCE}"
+ printf 'BROTLI_MD5SUM=%s\n' "${BROTLI_MD5SUM}"
+ printf 'XSIMD_NAME=%s\n' "${XSIMD_NAME}"
+ printf 'XSIMD_SOURCE=%s\n' "${XSIMD_SOURCE}"
+ printf 'XSIMD_MD5SUM=%s\n' "${XSIMD_MD5SUM}"
+ arrow_paimon_fingerprint_files \
+ patches/apache-arrow-"${ARROW_VERSION}"-*.patch
+ } | git hash-object --stdin
)
}
@@ -106,15 +160,15 @@ paimon_build_fingerprint() {
local vars_dir
vars_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
(
+ set -o pipefail
cd "${vars_dir}" || return 1
- LC_ALL=C
{
arrow_build_fingerprint
- git hash-object \
- arrow-paimon-vars.sh \
- vars.sh \
- download-thirdparty.sh \
- build-thirdparty.sh \
+ printf 'schema=%s\n' "${PAIMON_BUILD_SCHEMA_VERSION}"
+ printf 'PAIMON_CPP_NAME=%s\n' "${PAIMON_CPP_NAME}"
+ printf 'PAIMON_CPP_SOURCE=%s\n' "${PAIMON_CPP_SOURCE}"
+ printf 'PAIMON_CPP_MD5SUM=%s\n' "${PAIMON_CPP_MD5SUM}"
+ arrow_paimon_fingerprint_files \
paimon-cpp-cache.cmake \
patches/paimon-cpp-*.patch
} | git hash-object --stdin
@@ -130,6 +184,27 @@ arrow_paimon_build_fingerprint() {
} | git hash-object --stdin
}
+arrow_paimon_fingerprint_matches() {
+ local installed_fingerprint="$1"
+ local expected_fingerprint="$2"
+ local legacy_compatible_expected_fingerprint="$3"
+ shift 3
+ local compatible_fingerprint
+
+ if [[ "${installed_fingerprint}" == "${expected_fingerprint}" ]]; then
+ return 0
+ fi
+ if [[ "${expected_fingerprint}" !=
"${legacy_compatible_expected_fingerprint}" ]]; then
+ return 1
+ fi
+ for compatible_fingerprint in "$@"; do
+ if [[ "${installed_fingerprint}" == "${compatible_fingerprint}" ]];
then
+ return 0
+ fi
+ done
+ return 1
+}
+
ARROW_REQUIRED_LIBRARIES=(
libbrotlicommon.a
libbrotlidec.a
@@ -216,7 +291,9 @@ arrow_prebuilt_valid() {
fi
expected_fingerprint="$(arrow_build_fingerprint)"
installed_fingerprint="$(<"${arrow_fingerprint_mark}")"
- if [[ "${installed_fingerprint}" != "${expected_fingerprint}" ]]; then
+ if ! arrow_paimon_fingerprint_matches "${installed_fingerprint}" \
+ "${expected_fingerprint}"
"${ARROW_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}" \
+ "${ARROW_LEGACY_BUILD_FINGERPRINTS[@]}"; then
echo "Arrow build fingerprint does not match selected inputs" >&2
return 1
fi
@@ -235,7 +312,9 @@ paimon_prebuilt_valid() {
fi
expected_fingerprint="$(paimon_build_fingerprint)"
installed_fingerprint="$(<"${paimon_fingerprint_mark}")"
- if [[ "${installed_fingerprint}" != "${expected_fingerprint}" ]]; then
+ if ! arrow_paimon_fingerprint_matches "${installed_fingerprint}" \
+ "${expected_fingerprint}"
"${PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}" \
+ "${PAIMON_LEGACY_BUILD_FINGERPRINTS[@]}"; then
echo "Paimon build fingerprint does not match selected inputs" >&2
return 1
fi
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index 8d4b4b4ada8..df4bfe6d1fa 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -1950,6 +1950,21 @@ build_hadoop_libs_3_4() {
rm -rf "${TP_INSTALL_DIR}/lib/hadoop_hdfs_3_4/native/*.a"
find ./hadoop-dist/target/hadoop-3.4.2/lib/native/ -type f ! -name '*.a'
-exec cp {} "${TP_INSTALL_DIR}/lib/hadoop_hdfs_3_4/native/" \;
find ./hadoop-dist/target/hadoop-3.4.2/lib/native/ -type l -exec cp -P {}
"${TP_INSTALL_DIR}/lib/hadoop_hdfs_3_4/native/" \;
+
+ # 3.3.6.6 installed this same layout under hadoop_hdfs/, and that prefix
is what
+ # branch-3.0, branch-3.1, the cloud module and anything outside this tree
still
+ # include and link. Only 3.4.2.4 is built now, so point the old name at it
rather
+ # than ship a second 182MB copy per platform. Relative target, so the
prebuilt
+ # archive stays relocatable - the same shape as the lib -> lib64 link the
install
+ # prefix is set up with.
+ #
+ # No trailing slash: `rm -rf link/` deletes what the link points at on BSD
rm and
+ # does nothing on GNU rm, while `rm -rf link` removes just the link
everywhere.
+ # The removal has to run first - `ln -s` against an existing real
directory would
+ # land inside it instead of replacing it.
+ rm -rf "${TP_INSTALL_DIR}/include/hadoop_hdfs"
"${TP_INSTALL_DIR}/lib/hadoop_hdfs"
+ ln -sfn hadoop_hdfs_3_4 "${TP_INSTALL_DIR}/include/hadoop_hdfs"
+ ln -sfn hadoop_hdfs_3_4 "${TP_INSTALL_DIR}/lib/hadoop_hdfs"
}
# AvxToNeon
diff --git a/thirdparty/test/arrow-paimon-lifecycle-test.sh
b/thirdparty/test/arrow-paimon-lifecycle-test.sh
index ab48d3bcd22..f946be5c763 100755
--- a/thirdparty/test/arrow-paimon-lifecycle-test.sh
+++ b/thirdparty/test/arrow-paimon-lifecycle-test.sh
@@ -28,6 +28,101 @@ fail() {
tmpdir="$(mktemp -d)"
trap 'rm -rf "${tmpdir}"' EXIT
+create_fingerprint_fixture() {
+ local destination="$1"
+
+ mkdir -p "${destination}/patches"
+ cp "${ROOT}/arrow-paimon-vars.sh" "${destination}/arrow-paimon-vars.sh"
+ cp "${ROOT}/paimon-cpp-cache.cmake" "${destination}/paimon-cpp-cache.cmake"
+ cp "${ROOT}"/patches/apache-arrow-24.0.0-*.patch "${destination}/patches/"
+ cp "${ROOT}"/patches/paimon-cpp-*.patch "${destination}/patches/"
+}
+
+fingerprint_from_fixture() {
+ local fixture="$1"
+ local component="$2"
+ local result_variable="$3"
+ local fingerprint
+
+ fingerprint="$(
+ set -e
+ # shellcheck source=/dev/null
+ . "${fixture}/arrow-paimon-vars.sh"
+ "${component}_build_fingerprint"
+ )"
+ printf -v "${result_variable}" '%s' "${fingerprint}"
+}
+
+exercise_semantic_fingerprints() {
+ local first_fixture="${tmpdir}/fingerprint-first"
+ local second_fixture="${tmpdir}/fingerprint-second"
+ local first_arrow
+ local first_paimon
+ local second_arrow
+ local second_paimon
+ local changed_arrow
+ local changed_paimon
+
+ create_fingerprint_fixture "${first_fixture}"
+ create_fingerprint_fixture "${second_fixture}"
+
+ printf '%s\n' first-env >"${first_fixture}/env.sh"
+ printf '%s\n' second-env >"${second_fixture}/env.sh"
+ printf '%s\n' first-vars >"${first_fixture}/vars.sh"
+ printf '%s\n' second-vars >"${second_fixture}/vars.sh"
+ printf '%s\n' first-download >"${first_fixture}/download-thirdparty.sh"
+ printf '%s\n' second-download >"${second_fixture}/download-thirdparty.sh"
+ printf '%s\n' first-build >"${first_fixture}/build-thirdparty.sh"
+ printf '%s\n' second-build >"${second_fixture}/build-thirdparty.sh"
+ printf '%s\n' '# release-branch-only comment'
>>"${second_fixture}/arrow-paimon-vars.sh"
+
+ fingerprint_from_fixture "${first_fixture}" arrow first_arrow
+ fingerprint_from_fixture "${first_fixture}" paimon first_paimon
+ [[ "${first_arrow}" == "${ARROW_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}"
]] ||
+ fail "the Arrow legacy marker migration target is stale"
+ [[ "${first_paimon}" == "${PAIMON_LEGACY_COMPATIBLE_SEMANTIC_FINGERPRINT}"
]] ||
+ fail "the Paimon legacy marker migration target is stale"
+ fingerprint_from_fixture "${second_fixture}" arrow second_arrow
+ fingerprint_from_fixture "${second_fixture}" paimon second_paimon
+ [[ "${first_arrow}" == "${second_arrow}" ]] ||
+ fail "unrelated branch scripts changed the Arrow fingerprint"
+ [[ "${first_paimon}" == "${second_paimon}" ]] ||
+ fail "unrelated branch scripts changed the Paimon fingerprint"
+
+ printf '%s\n' semantic-change \
+ >>"${second_fixture}/patches/apache-arrow-24.0.0-lzo.patch"
+ fingerprint_from_fixture "${second_fixture}" arrow changed_arrow
+ fingerprint_from_fixture "${second_fixture}" paimon changed_paimon
+ [[ "${changed_arrow}" != "${first_arrow}" ]] ||
+ fail "an Arrow patch change did not change the Arrow fingerprint"
+ [[ "${changed_paimon}" != "${first_paimon}" ]] ||
+ fail "an Arrow patch change did not change the Paimon fingerprint"
+
+ cp "${first_fixture}/patches/apache-arrow-24.0.0-lzo.patch" \
+ "${second_fixture}/patches/apache-arrow-24.0.0-lzo.patch"
+ printf '%s\n' semantic-change >>"${second_fixture}/paimon-cpp-cache.cmake"
+ fingerprint_from_fixture "${second_fixture}" arrow second_arrow
+ fingerprint_from_fixture "${second_fixture}" paimon changed_paimon
+ [[ "${second_arrow}" == "${first_arrow}" ]] ||
+ fail "a Paimon-only cache change changed the Arrow fingerprint"
+ [[ "${changed_paimon}" != "${first_paimon}" ]] ||
+ fail "a Paimon cache change did not change the Paimon fingerprint"
+
+ changed_arrow="$(
+ set -e
+ # shellcheck source=/dev/null
+ . "${first_fixture}/arrow-paimon-vars.sh"
+ ARROW_BUILD_SCHEMA_VERSION="${ARROW_BUILD_SCHEMA_VERSION}-changed"
+ arrow_build_fingerprint
+ )"
+ [[ "${changed_arrow}" != "${first_arrow}" ]] ||
+ fail "an Arrow build-schema change did not change its fingerprint"
+}
+
+# shellcheck source=../arrow-paimon-vars.sh
+. "${ROOT}/arrow-paimon-vars.sh"
+exercise_semantic_fingerprints
+
harness="${tmpdir}/harness"
mkdir -p "${harness}/src" "${harness}/patches" "${harness}/installed"
cp "${ROOT}/download-thirdparty.sh" "${harness}/download-thirdparty.sh"
@@ -161,14 +256,14 @@ exercise_generic_recovery_dispatch() {
local package2
local extra
- mkdir -p "${thirdparty_dir}/installed/lib/hadoop_hdfs/native" \
- "${external_thirdparty_dir}/installed/lib/hadoop_hdfs/native" \
+ mkdir -p "${thirdparty_dir}/installed/lib/hadoop_hdfs_3_4/native" \
+ "${external_thirdparty_dir}/installed/lib/hadoop_hdfs_3_4/native" \
"${generic}/gensrc" "${generic}/fe" "${generic}/be/build_Release" \
"${generic}/be/output"
cp "${ROOT}/../build.sh" "${generic}/build.sh"
cp "${ROOT}/arrow-paimon-vars.sh" "${thirdparty_dir}/arrow-paimon-vars.sh"
- touch "${thirdparty_dir}/installed/lib/hadoop_hdfs/native/libhdfs.a"
- touch
"${external_thirdparty_dir}/installed/lib/hadoop_hdfs/native/libhdfs.a"
+ touch "${thirdparty_dir}/installed/lib/hadoop_hdfs_3_4/native/libhdfs.a"
+ touch
"${external_thirdparty_dir}/installed/lib/hadoop_hdfs_3_4/native/libhdfs.a"
printf '%s\n' 'clean: ; @:' >"${generic}/gensrc/Makefile"
# shellcheck disable=SC2016
printf '%s\n' '#!/usr/bin/env bash' '[[ "$1" == "clean" ]]' >"${fake_mvn}"
@@ -282,7 +377,6 @@ exercise_generic_recovery_dispatch
# A Paimon-only build may publish only its own fingerprint. It must not make a
# stale Arrow installation pass the shared prebuilt validation.
-. "${ROOT}/arrow-paimon-vars.sh"
prebuilt="${tmpdir}/prebuilt"
mkdir -p "${prebuilt}/include/arrow/util" "${prebuilt}/lib64"
printf '#define ARROW_VERSION_STRING "%s"\n' "${ARROW_VERSION}" \
@@ -304,6 +398,25 @@ if arrow_paimon_prebuilt_valid "${prebuilt}" >/dev/null
2>&1; then
fail "legacy combined marker certified an unversioned component closure"
fi
+printf '%s\n' "${ARROW_LEGACY_BUILD_FINGERPRINTS[0]}" \
+ >"${prebuilt}/arrow-build-fingerprint.txt"
+printf '%s\n' "${PAIMON_LEGACY_BUILD_FINGERPRINTS[0]}" \
+ >"${prebuilt}/paimon-build-fingerprint.txt"
+arrow_paimon_prebuilt_valid "${prebuilt}" ||
+ fail "the complete shared prebuilt was rejected during fingerprint
migration"
+if (
+ ARROW_BUILD_SCHEMA_VERSION="${ARROW_BUILD_SCHEMA_VERSION}-changed"
+ arrow_prebuilt_valid "${prebuilt}"
+) >/dev/null 2>&1; then
+ fail "the legacy Arrow marker survived a semantic fingerprint change"
+fi
+if (
+ PAIMON_BUILD_SCHEMA_VERSION="${PAIMON_BUILD_SCHEMA_VERSION}-changed"
+ paimon_prebuilt_valid "${prebuilt}"
+) >/dev/null 2>&1; then
+ fail "the legacy Paimon marker survived a semantic fingerprint change"
+fi
+
publish_arrow_prebuilt_marker "${prebuilt}"
publish_paimon_prebuilt_marker "${prebuilt}"
rm "${prebuilt}/lib64/libarrow_compute.a"
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
