morningman opened a new pull request, #66935:
URL: https://github.com/apache/doris/pull/66935
### What problem does this PR solve?
Issue Number: close #xxx
Related PR: #66842, #66908
Problem Summary:
#66842 stopped building hadoop-libs 3.3.6.6 and removed the
`installed/{include,lib}/hadoop_hdfs/` prefix it produced. Only 3.4.2.4 under
`hadoop_hdfs_3_4/` is built now, so everything still reading the old prefix
fails
against a third-party archive built from it:
| consumer | reads |
|---|---|
| `cloud/src/recycler/hdfs_accessor.{h,cpp}` | `<hadoop_hdfs/hdfs.h>` |
| `cloud/CMakeLists.txt` | `lib/hadoop_hdfs/native/libhdfs.a` |
| `be/src/io/hdfs_builder.cpp` | `hadoop_hdfs/hdfs.h` |
| `build.sh` | `LAST_THIRDPARTY_LIB='hadoop_hdfs/native/libhdfs.a'` |
| `run-be-ut.sh`, `run-cloud-ut.sh` | `installed/lib/hadoop_hdfs/` |
| branch-3.0, branch-3.1, anything outside this repository | both |
#66908 moves this repository's consumers onto `hadoop_hdfs_3_4/`, and that
stays the
right fix for master. It does nothing for the release branches or for anyone
outside
this repository, and it cannot land until the prebuilt archives have been
rebuilt.
So `build_hadoop_libs_3_4` leaves the old prefix behind as a symlink to the
one build
that remains.
## Why a symlink rather than a second copy
`lib/hadoop_hdfs_3_4/` is 182MB per platform, and the two prefixes have
always had
identical layouts - the removed `build_hadoop_libs()` and
`build_hadoop_libs_3_4()`
are the same `cp` sequence with a different version number, so the old name
fits the
new tree exactly.
The install prefix is already built this way: `build-thirdparty.sh` creates
`installed/lib -> lib64` as a relative directory symlink before any package
runs, and
every prebuilt archive carries it. That is what shows the packaging and
distribution
path preserves symlinks - a prebuilt from #66842 unpacks with 55 of them.
## Two details that are load-bearing
- **The target is relative** (`hadoop_hdfs_3_4`, not
`${TP_INSTALL_DIR}/...`), so the
link resolves wherever the archive is unpacked rather than pointing back
at the
machine that built it.
- **`rm -rf` runs without a trailing slash, and before `ln -s`.** `rm -rf
link/`
deletes what the link points at on BSD rm and does nothing on GNU rm, while
`rm -rf link` removes just the link on both. And without removing first,
`ln -s`
against a leftover real `hadoop_hdfs/` directory - an older prebuilt, or a
build-env
image predating its removal - creates `hadoop_hdfs/hadoop_hdfs_3_4` inside
it and
leaves the stale 3.3.6.6 archive in place.
## Effect on the merge order of #66842 / #66908
#66842 noted a window between itself and #66908: a Linux tree picking up a
freshly
built archive would not find `installed/lib/hadoop_hdfs/native/libhdfs.a`,
which is
still what `build.sh`'s `LAST_THIRDPARTY_LIB` looks for, and would start a
full ~3h
third-party rebuild. Once the archives are rebuilt from this PR, the
sentinel resolves
through the link and that window closes without waiting for #66908.
#66908 should still land, sentinel move included - pointing at the real
directory
rather than at the shim keeps it working when the shim is eventually
dropped. One line
in its release note no longer holds: anything outside this repository
reading the
`hadoop_hdfs/` prefix keeps working.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (add detailed scripts or steps below)
`build_hadoop_libs_3_4` was driven end to end against a fixture
install prefix
reproducing `lib -> lib64`, on macOS (BSD `rm`/`ln`) and on
ubuntu-22.04 (GNU
coreutils 8.32), checking that:
- both links are created, with the relative target, and the `lib` one
lands in
`lib64/` where `lib` points;
- `build.sh`'s `hadoop_hdfs/native/libhdfs.a` sentinel and
`<hadoop_hdfs/hdfs.h>` resolve through them;
- a rerun neither nests the links nor empties the 3.4.2.4 tree behind
them;
- an installation carrying a real 3.3.6.6 `hadoop_hdfs/` directory is
taken over,
with the old prefix resolving to the 3.4.2.4 archive afterwards;
- the links survive `tar -czf` / `tar -xzf` into a different path,
which is how
the prebuilt archive is shipped.
This PR touches `thirdparty/**`, so its own `Build Third Party
Libraries` run
builds the whole tree on ubuntu-22.04, macos-15 and macos-14, which is
where the
change actually executes.
- Behavior changed:
- [x] Yes.
- `installed/{include,lib}/hadoop_hdfs` resolves to hadoop-libs
**3.4.2.4**, not
the 3.3.6.6 it named before #66842. A tree built against a new
archive through
the old prefix gets 3.4.2.4. That is the intended outcome - BE
already linked
the 3.4.2.4 archive while `hdfs_builder.cpp` included the 3.3.6.6
header, and
#66908 moves `cloud/` to 3.4.2.4 as well - but it is a version
change, not a
restoration. Release branches keep using the archives and build-env
images
matching their own third-party tree, so their CI is unaffected.
- This is a compatibility shim over a single build. It should be
dropped once
nothing reads the old prefix; the CHANGELOG entry says so.
- Does this need documentation?
- [x] No.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
