This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new d239eba770b [chore](thirdparty) upgrade HADOOP_LIBS_3_4 to
hadoop-3.4.2.4-for-doris (#66808)
d239eba770b is described below
commit d239eba770b4816c8ad59b2f18b4be7d29e45681
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Mon Aug 17 09:45:23 2026 +0800
[chore](thirdparty) upgrade HADOOP_LIBS_3_4 to hadoop-3.4.2.4-for-doris
(#66808)
### What problem does this PR solve?
Related PR: #66783, #66800
Problem Summary:
Picks up
[apache/doris-thirdparty#408](https://github.com/apache/doris-thirdparty/pull/408),
tagged as `hadoop-3.4.2.4-for-doris`. The single change over
`hadoop-3.4.2.3-for-doris` is that macOS no longer builds `libhdfspp`.
That library does not link on macOS. It is compiled against Homebrew
LLVM's libc++ headers, which are configured with
`_LIBCPP_HAS_VENDOR_AVAILABILITY_ANNOTATIONS 0`, so
`std::exception_ptr`'s post-macOS-14 entry points are emitted whatever
`-mmacosx-version-min` says, while `-lc++` still resolves against the
SDK's system libc++. Against a macOS 14 SDK that leaves
`__cxa_init_primary_exception` and
`std::exception_ptr::__from_native_exception_pointer` undefined, and the
failing dylib takes the whole `make` down with it — `libhdfs.a`
included:
```
Undefined symbols for architecture arm64:
"std::exception_ptr::__from_native_exception_pointer(void*)", referenced
from:
std::__1::promise<hdfs::Status>::~promise() in filesystem_sync.cc.o
...
ld: symbol(s) not found for architecture arm64
make[2]: *** [native/target/usr/local/lib/libhdfspp.0.1.0.dylib] Error 1
```
Doris never links libhdfspp — `be/CMakeLists.txt:251` sets
`BUILD_LIBHDFSPP OFF` — so the fork now stops after the x-platform
object libraries `libhdfs` pulls out of that tree. `libhdfs.a`,
`libhdfs.dylib`, `libhadoop.dylib`, `hdfs.h` and the jars are all
unchanged.
Linux is unaffected by the fork change, so this bump is a no-op there
beyond the new tag. It is a prerequisite for #66800, which is where a
macOS third-party build starts producing `lib/hadoop_hdfs_3_4/` and
therefore where the failure above shows up — on the `macos-14` runner
specifically. Note that master's `build-thirdparty.sh` still leaves
`hadoop_libs_3_4` out of the macOS package list, so the macOS legs of
this PR's CI do not exercise the fork change yet; #66800 is what turns
it on.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
The fork was built end to end on macOS 26.5 / arm64 with the toolchain
the macOS CI uses (Homebrew `llvm@20`, CMake 3.22.1, JDK 17,
`MACOSX_DEPLOYMENT_TARGET=12.0`) before the tag was cut —
hadoop-common-project 9/9, hadoop-hdfs-project 7/7, hadoop-dist all
`BUILD SUCCESS`, producing
```
lib/native/ libhadoop.dylib 159K libhadoop.a 753K
libhdfs.dylib 133K libhdfs.a 1.4M arm64
include/ hdfs.h
```
with no `libhdfspp.dylib` (8.9M previously), and `libhdfs.a` still
exporting `hdfsSetLogger`, `hdfsGetLastExceptionRootCause`,
`hdfsBuilderSetKerb5Conf`, `hdfsBuilderSetKeyTabFile`,
`hdfsUnbufferFile`, `hdfsFileGetReadStatistics`, `hdfsHSync` and
`hdfsHFlush`.
The tarball this PR points at was downloaded twice; the MD5 is stable
and the archive unpacks to `doris-thirdparty-hadoop-3.4.2.4-for-doris/`
as `HADOOP_LIBS_3_4_SOURCE` expects.
- Behavior changed:
- [x] No. Linux builds byte for byte the same library set; macOS drops a
library nothing links.
- Does this need documentation?
- [x] No.
---------
Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
---
thirdparty/CHANGELOG.md | 4 ++++
thirdparty/build-thirdparty.sh | 4 ++++
thirdparty/vars.sh | 8 ++++----
3 files changed, 12 insertions(+), 4 deletions(-)
diff --git a/thirdparty/CHANGELOG.md b/thirdparty/CHANGELOG.md
index 1cb2e96a53b..8c049ea51bb 100644
--- a/thirdparty/CHANGELOG.md
+++ b/thirdparty/CHANGELOG.md
@@ -2,6 +2,10 @@
This file contains version of the third-party dependency libraries in the
build-env image. The docker build-env image is apache/doris, and the tag is
`build-env-${version}`
+## 20260816
+
+- Modified: hadoop-libs 3.4.2.3 -> 3.4.2.4
+
## 20260814
- Modified: hadoop-libs 3.4.2.2 -> 3.4.2.3
diff --git a/thirdparty/build-thirdparty.sh b/thirdparty/build-thirdparty.sh
index f44122ca1f3..6f0676e565a 100755
--- a/thirdparty/build-thirdparty.sh
+++ b/thirdparty/build-thirdparty.sh
@@ -2372,6 +2372,10 @@ if [[ "${#packages[@]}" -eq 0 ]]; then
)
if [[ "$(uname -s)" == 'Darwin' ]]; then
read -r -a packages <<<"binutils gettext ${packages[*]}"
+ # hadoop_libs, the 3.3.6 fork, stays Linux-only: it carries none of
the macOS fixes
+ # apache/doris-thirdparty#407 made to the 3.4 fork, and nothing built
here reads its
+ # hadoop_hdfs/ prefix - the cloud module, its only other consumer, is
Linux-only too.
+ read -r -a packages <<<"${packages[*]} hadoop_libs_3_4"
elif [[ "$(uname -s)" == 'Linux' ]]; then
read -r -a packages <<<"${packages[*]} hadoop_libs"
read -r -a packages <<<"${packages[*]} hadoop_libs_3_4"
diff --git a/thirdparty/vars.sh b/thirdparty/vars.sh
index 211954c27a8..6a42bd00cf7 100644
--- a/thirdparty/vars.sh
+++ b/thirdparty/vars.sh
@@ -504,10 +504,10 @@
HADOOP_LIBS_SOURCE="doris-thirdparty-hadoop-3.3.6.6-for-doris"
HADOOP_LIBS_MD5SUM="13b66d5f2abffd1740e692b65df5962e"
# libhdfs 3.4
-HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.3-for-doris.tar.gz";
-HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.3-for-doris.tar.gz"
-HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.3-for-doris"
-HADOOP_LIBS_3_4_MD5SUM="a751a42f3a927c4a272a99d4e3788648"
+HADOOP_LIBS_3_4_DOWNLOAD="https://github.com/apache/doris-thirdparty/archive/refs/tags/hadoop-3.4.2.4-for-doris.tar.gz";
+HADOOP_LIBS_3_4_NAME="hadoop-3.4.2.4-for-doris.tar.gz"
+HADOOP_LIBS_3_4_SOURCE="doris-thirdparty-hadoop-3.4.2.4-for-doris"
+HADOOP_LIBS_3_4_MD5SUM="729a7ef9126f65255d1da124a88ffd50"
# AvxToNeon
AVX2NEON_DOWNLOAD="https://github.com/kunpengcompute/AvxToNeon/archive/refs/tags/v1.0.0.tar.gz";
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
