morningman opened a new pull request, #408:
URL: https://github.com/apache/doris-thirdparty/pull/408
## What
On macOS the `libhdfspp` tree now stops after the x-platform object
libraries `libhdfs` pulls out of it. Linux and Windows are byte for byte
unchanged.
This supersedes the `NO_SASL` half of #407. That guard stays — SASL had
libhdfspp as its only consumer, so there is still no reason to look for it —
but its comment is rewritten, because the link failure it describes can no
longer happen.
## Why
apache/doris#66800 adds `hadoop_libs_3_4` to the macOS package list, so a
macOS third-party build now runs this tree for the first time. The `macos-14`
leg of that CI run fails while linking `libhdfspp.0.1.0.dylib`:
```
Undefined symbols for architecture arm64:
"std::exception_ptr::__from_native_exception_pointer(void*)", referenced
from:
std::__1::promise<hdfs::Status>::~promise() in filesystem_sync.cc.o
std::__1::promise<std::__1::tuple<hdfs::Status,
hdfs::FileHandle*>>::~promise() in filesystem_sync.cc.o
...
"___cxa_init_primary_exception", referenced from:
...
ld: symbol(s) not found for architecture arm64
clang++: error: linker command failed with exit code 1
make[2]: *** [native/target/usr/local/lib/libhdfspp.0.1.0.dylib] Error 1
```
### The symbols come from Homebrew's libc++ headers, the link comes from the
SDK's libc++
Doris' macOS toolchain is Homebrew `llvm@20` (`env.sh` puts it on `PATH` and
derives `CC`/`CXX` from it). Homebrew builds libc++ with
`_LIBCPP_HAS_VENDOR_AVAILABILITY_ANNOTATIONS 0` — it is right there in the
installed `__config_site`:
```
$ grep VENDOR_AVAILABILITY
/opt/homebrew/opt/llvm@20/include/c++/v1/__config_site
#define _LIBCPP_HAS_VENDOR_AVAILABILITY_ANNOTATIONS 0
```
With the annotations off, `std::make_exception_ptr`'s post-macOS-14 fast
path is emitted no matter what `-mmacosx-version-min` says. Two lines reproduce
it:
```
$ cat t.cc
#include <future>
#include <string>
std::promise<std::string> p;
void f() { p.set_value("x"); }
$ /opt/homebrew/opt/llvm@20/bin/clang++ -std=c++17 -stdlib=libc++ -arch
arm64 \
-mmacosx-version-min=12.0 -c t.cc -o t.o
$ nm -u t.o | grep exception
__ZNSt13exception_ptr31__from_native_exception_pointerEPv
___cxa_init_primary_exception
```
The other half is the link line, which carries `-stdlib=libc++` and nothing
else — no `-L$(brew --prefix llvm@20)/lib/c++`, the flag Homebrew's own caveat
tells you to add. So `-lc++`/`-lc++abi` resolve against the SDK's system
libc++, not against the one whose headers were used:
```
$ clang++ -stdlib=libc++ -arch arm64 -mmacosx-version-min=12.0 \
-isysroot .../MacOSX15.4.sdk -dynamiclib t.o -o t.dylib
$ otool -L t.dylib
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version
1900.178.0)
```
That link succeeds because the macOS 15 SDK exports both symbols from its
`libc++.tbd`. The macOS 14 SDK does not, which is the entire difference between
the two macOS legs of apache/doris#66800 — both run arm64 on the same Homebrew
clang 20.1.8 with the same `MACOSX_DEPLOYMENT_TARGET=12.0`:
| job | `runs-on` | SDK | result |
|---|---|---|---|
| Build Third Party Libraries (macOS) | `macos-15` | MacOSX15.5.sdk |
libhdfspp links |
| Build Third Party Libraries (macOS-arm64) | `macos-14` | MacOSX14.5.sdk |
undefined symbols |
This repo's own release automation is affected the same way:
`.github/workflows/build.yml` builds the `macOS-arm64` matrix entry — the one
that produces `doris-thirdparty-prebuilt-darwin-arm64` — on `os: macos-14`.
### Why not just link Homebrew's libc++
Adding `-L$(brew --prefix llvm@20)/lib/c++ -Wl,-rpath,...` does fix the
link, but it puts a Homebrew LLVM runtime dependency into an archive that gets
redistributed and unpacked on machines that have no such prefix. Raising the
runners to `macos-15` only moves the problem: the dylib would then carry a
system-libc++ symbol that does not exist on macOS 14, so it would fail to load
there.
Neither cost is worth paying for a library nothing consumes. Doris never
links libhdfspp — `be/CMakeLists.txt` sets `BUILD_LIBHDFSPP OFF` — so macOS
simply does not build it.
### Why the subdirectory is still added
#407 concluded that skipping `add_subdirectory(main/native/libhdfspp)`
outright is not workable, and that is correct: `libhdfs/CMakeLists.txt` pulls
`$<TARGET_OBJECTS:x_platform_obj>` and `$<TARGET_OBJECTS:x_platform_obj_c_api>`
out of that tree, and dropping it yields `No SOURCES given to target: hdfs`.
The whole directory does not have to go, though. `lib/x-platform` is
self-contained — six translation units, no protobuf, no Boost, no asio — so the
guard sits inside `libhdfspp/CMakeLists.txt`, after the `include_directories()`
and compile-flag blocks and before `add_subdirectory(third_party/uriparser2)`:
```cmake
if(APPLE)
add_subdirectory(lib/x-platform)
return()
endif()
```
Placing it there rather than in the parent is what keeps this a no-op for
the objects that survive: `-fPIC`, `-DHAVE_EXPLICIT_BZERO`,
`-DUSE_X_PLATFORM_DIRENT`, `CMAKE_CXX_STANDARD 11` and the `include lib` search
path are all already in effect above that line, so x-platform compiles exactly
as it does today.
Of the 63 objects that went into `libhdfspp.dylib` on the failing run, 59
are no longer compiled on macOS — including all 15 generated `.pb.cc` files —
and the dylib link that failed no longer happens.
## Verification
macOS 26.5 / arm64, with the toolchain Doris' macOS CI uses: Homebrew
`llvm@20` (clang 20.1.8), CMake 3.22.1, JDK 17,
`MACOSX_DEPLOYMENT_TARGET=12.0`, dependencies resolved from a Doris thirdparty
prefix.
`build.sh` completes end to end — hadoop-common-project 9/9,
hadoop-hdfs-project 7/7, hadoop-dist — and produces:
```
hadoop-3.4.2/lib/native/ libhadoop.1.0.0.dylib 159K arm64
libhadoop.a 753K
libhadoop.dylib 159K
libhdfs.0.0.0.dylib 133K
libhdfs.a 1.4M
libhdfs.dylib 133K
hadoop-3.4.2/include/ hdfs.h
hadoop-libhdfs-3.4.2/native/ libhdfs.a 1.4M arm64
include/ hdfs.h
common/ 5 jars
hdfs/ 10 jars
```
`libhdfspp.dylib` — 8.9M in #407's run — is gone; everything Doris consumes
is still there. `libhdfs.a` still exports the full API, including this fork's
`hdfsSetLogger`, plus `hdfsGetLastExceptionRootCause`,
`hdfsBuilderSetKerb5Conf`, `hdfsBuilderSetKeyTabFile`, `hdfsUnbufferFile`,
`hdfsFileGetReadStatistics`, `hdfsHSync` and `hdfsHFlush`.
Only the five x-platform objects are built out of the libhdfspp tree, and
they carry the same flags they did when the whole library was built:
```
$ find target/main/native/libhdfspp -name '*.o'
lib/x-platform/CMakeFiles/x_platform_obj.dir/dirent.cc.o
lib/x-platform/CMakeFiles/x_platform_obj.dir/syscall_linux.cc.o
lib/x-platform/CMakeFiles/x_platform_obj.dir/utils.cc.o
lib/x-platform/CMakeFiles/x_platform_obj_c_api.dir/c-api/dirent.cc.o
lib/x-platform/CMakeFiles/x_platform_obj_c_api.dir/c-api/syscall.cc.o
$ grep CXX_FLAGS
target/main/native/libhdfspp/lib/x-platform/CMakeFiles/x_platform_obj.dir/flags.make
CXX_FLAGS = -include cstdint -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64
-Wno-cast-function-type -Wno-unused-lambda-capture -Wall -Wextra -pedantic
-g -fPIC
-fno-strict-aliasing -stdlib=libc++ -Wno-deprecated-declarations
-Wno-unused-local-typedef -arch arm64 -isysroot ...
-mmacosx-version-min=12.0 -std=gnu++17
```
and they do reach `libhdfs.a` — `nm -g` finds 16 `XPlatform` symbols in it.
Note this cannot reproduce the original undefined-symbol failure locally: a
macOS 26 SDK exports both symbols, exactly as the macOS 15 SDK does. What it
does verify is that the macOS build no longer reaches that link at all, and
that dropping libhdfspp costs `libhdfs` nothing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
