morningman opened a new pull request, #66800: URL: https://github.com/apache/doris/pull/66800
### What problem does this PR solve? Related PR: #66783 Problem Summary: `build-thirdparty.sh` adds the hadoop packages to its default package list under `uname -s == Linux` alone, so no macOS build has ever installed `lib/hadoop_hdfs_3_4/` — neither a developer's local prefix, nor the `doris-thirdparty-prebuilt-darwin-x86_64` / `-arm64` archives, which apache/doris-thirdparty's automation produces by running this very script with no package arguments. A macOS prefix carries libhdfs3 (`lib64/libhdfs3.a`, `include/hdfs/`) and nothing else. That was correct while hadoop's native build could not complete on a Mac at all. [apache/doris-thirdparty#407](https://github.com/apache/doris-thirdparty/pull/407) fixed it, and `hadoop-3.4.2.3-for-doris` — selected since #66783 — carries the fix, so the 3.4 fork builds here now. `hadoop_libs`, the 3.3.6 fork, stays Linux-only: it has none of those fixes, and nothing built on macOS reads its `hadoop_hdfs/` prefix — the BE includes `hadoop_hdfs_3_4/hdfs.h`, and the cloud module, the only consumer of the 3.3.6 one, is not built on macOS. Making the BE link hadoop's `libhdfs.a` on macOS instead of libhdfs3 (`be/CMakeLists.txt` still selects libhdfs3 there) is a separate change. It needs this one first, and it needs the published prebuilt archives rebuilt with it — which the doris-thirdparty automation does on the next `thirdparty/` change, i.e. on this one. ### Release note None ### Check List (For Author) - Test - [x] Manual test (add detailed scripts or steps below) Verified on macOS 26.5 / arm64. With this change the default package list ends with `hadoop_libs_3_4` and still contains no `hadoop_libs`; `HADOOP_LIBS_3_4` is already unconditional in `TP_ARCHIVES`, so the source is downloaded on macOS today and `download-thirdparty.sh` needs no change. The package build itself was verified in #66783: `build-thirdparty.sh hadoop_libs_3_4` exits 0 and installs ``` installed/lib/hadoop_hdfs_3_4/native/ libhdfs.a 1.4M arm64 libhadoop.dylib 160K arm64 libhdfs.dylib 135K arm64 libhdfspp.dylib 8.9M arm64 installed/include/hadoop_hdfs_3_4/ hdfs.h installed/lib/hadoop_hdfs_3_4/ common/ (5 jars) hdfs/ (10 jars) ``` - Behavior changed: - [x] Yes. A macOS third-party build now also builds hadoop 3.4.2.3 and installs `lib/hadoop_hdfs_3_4/`. Linux is byte for byte unchanged. - Does this need documentation? - [x] No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
