morningman opened a new pull request, #66908:
URL: https://github.com/apache/doris/pull/66908
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66842
Problem Summary:
**Second half of a two-PR split. Do not merge before #66842 has landed *and*
the
prebuilt third-party archives have been rebuilt from it** - see "Merge
order" at the
bottom. #66842 carries the third-party build changes; this PR carries the
consumers.
## 1. Move the remaining `hadoop_hdfs/` consumers onto `hadoop_hdfs_3_4/`
The tree carried two libhdfs builds - 3.3.6.6 under
`installed/{include,lib}/hadoop_hdfs/`
and 3.4.2.4 under `.../hadoop_hdfs_3_4/` - and the two prefixes were mixed
up on the
consumer side:
| consumer | header | library |
|---|---|---|
| `be/src/io/fs/hdfs.h` | `hadoop_hdfs_3_4/hdfs.h` (3.4.2.4) | — |
| `be/src/io/hdfs_builder.cpp` | `hadoop_hdfs/hdfs.h` (**3.3.6.6**) | — |
| `be/CMakeLists.txt` | — | `hadoop_hdfs_3_4/native/libhdfs.a` (3.4.2.4) |
| `cloud/src/recycler/hdfs_accessor.{h,cpp}` | `hadoop_hdfs/hdfs.h`
(**3.3.6.6**) | — |
| `cloud/CMakeLists.txt` | — | `hadoop_hdfs/native/libhdfs.a` (**3.3.6.6**) |
Both headers share the `LIBHDFS_HDFS_H` include guard, so `hdfs_builder.cpp`
compiled
against the 3.3.6.6 header (it came first) while BE linked the 3.4.2.4
archive.
So `cloud/`, `build.sh`'s `LAST_THIRDPARTY_LIB` sentinel, `run-be-ut.sh`,
`run-cloud-ut.sh` and the thirdparty lifecycle test move to
`hadoop_hdfs_3_4/`, and
`be/src/io/hdfs_builder.cpp` drops its 3.3.6.6 include - `io/fs/hdfs.h`, two
lines
below it, already re-exports the 3.4.2.4 one. The unit-test runners still
stage the
jars at `lib/hadoop_hdfs/`, so the classpath loops below them are unchanged.
Both prefixes exist in every third-party archive built before #66842, so
this half is
safe against the images in use today. It is #66842 - which stops building
3.3.6.6 at
all - that makes it mandatory.
## 2. Link azure on aarch64 and macOS
`env.sh` forced `DISABLE_BUILD_AZURE=ON` on aarch64 and macOS, so BE and the
cloud
meta-service lost `+AZURE_BLOB` and `+AZURE_STORAGE_VAULT` there. #66842
made the
third-party recipe portable (`-ldl` does not exist on Apple; vcpkg ships no
prebuilt
tools for aarch64 Linux) and now builds azure on every platform. This drops
the
platform test so the consumers link it. `DISABLE_BUILD_AZURE=ON` still opts
out.
## Merge order
1. #66842 lands.
2. The apache/doris-thirdparty automation rebuilds the prebuilt archives -
it triggers
on `^thirdparty/` and takes ~3h - and the aarch64 build-env image is
rebuilt.
3. This PR lands.
Landing this one first fails at the BE link with
`ninja: error: '.../installed/lib/libazure-core.a' ... missing and no known
rule to
make it`, which is exactly what the BE UT (macOS) job of the un-split PR hit.
### Release note
Azure Blob Storage support (`+AZURE_BLOB`, `+AZURE_STORAGE_VAULT`) is now
built on
aarch64 and macOS, not just x86_64 Linux.
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
`thirdparty/test/arrow-paimon-lifecycle-test.sh` covers the `build.sh`
sentinel
move; it is what the Arrow/Paimon Lifecycle Test job runs. The azure
half is
exercised by every BE/cloud build in this PR's own CI, against a
prebuilt that
by then carries the libraries.
- Behavior changed:
- [x] Yes.
- Azure is now built and linked on aarch64 and macOS. **The
third-party prebuilt
has to be rebuilt from #66842 before this lands**: BE/cloud on those
platforms
will pass `-DBUILD_AZURE=ON` and fail to link against an older
archive that has
no azure libraries. `DISABLE_BUILD_AZURE=ON` still opts out.
- Anything outside this repository reading the `hadoop_hdfs/` prefix
needs
updating; only `hadoop_hdfs_3_4/` remains after #66842.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
