GauthamBanasandra opened a new pull request, #8844:
URL: https://github.com/apache/hadoop/pull/8844
### Description of PR
`mvn clean install` with `-Pnative-win` cannot complete on Windows. The
Docker image described in
BUILDING.txt no longer builds at all, and even with a working image the
native build fails in
libhdfspp and again in the YARN capacity-scheduler UI. There is no single
cause — eight separate
defects have accumulated, each of which independently stops the build.
**The build image no longer builds**
1. `Dockerfile_windows_10` installs Azul JDK 8. Hadoop requires JDK 17, so
the enforcer rejects the
build immediately. This line has not been touched since 2022.
2. `vcpkg install` fails because the pinned `protobuf 3.25.5` cannot be
resolved from vcpkg's
version database. The pin was added by HADOOP-19605 on 2025-11-10.
3. The pinned vcpkg tag `2025.03.19` references MSYS2 packages
(`msys2-runtime-3.5.4-2`) that have
since been purged from the upstream mirror. MSYS2 keeps only current
versions.
4. The `libzstd` and `rsync` MSYS2 pins in the Dockerfile have been purged
for the same reason.
**The native build fails**
5. Boost is not pinned explicitly — its version comes from
`builtin-baseline`, so it floats as the
baseline moves. Boost 1.87 removed `boost/asio/io_service.hpp`, which
five libhdfspp files still
include (`ioservice.h`, `ioservice_impl.h`, `mock_connection.{h,cc}`,
`remote_block_reader_test.cc`).
6. The protobuf/protoc compatibility check in `libhdfspp/CMakeLists.txt`
leaves
`CMAKE_REQUIRED_LIBRARIES` pointing at the Abseil targets. Abseil carries
C++ feature
requirements that a later C-language `check_symbol_exists()` cannot
satisfy under the Visual
Studio generator, so that check fails spuriously.
7. The legacy `PROTOBUF_LIBRARY`, `PROTOBUF_PROTOC_LIBRARY` and
`PROTOBUF_LIBRARIES` variables
resolve to import DLLs under vcpkg, which MSVC rejects with `LNK1107`.
8. `hadoop-yarn-capacity-scheduler-ui` sets `packageManagerScript` to
`node_modules/.bin/react-router`. On Windows that path is an `sh` shim,
which `node` cannot
execute.
**Why Linux is unaffected**
The Linux build *pins* its native dependencies inside the repository —
`dev-support/docker/install-boost.sh` hard-codes Boost 1.86.0 and
`install-protobuf.sh` hard-codes
Protobuf 3.25.5, both built from source. The Windows build instead delegates
to vcpkg and lets
versions float with the baseline. That asymmetry is the root cause of
defects 2, 5 and arguably 3:
the Windows build has no equivalent of the version lock the Linux build has
had all along.
**Fix**
- **Dockerfile**: install the Microsoft Build of OpenJDK 17; move to vcpkg
tag `2026.07.29`; refresh
the `libzstd` and `rsync` pins; refresh CMake to 3.30.2. Switch the base
image to
`windows/servercore`, which carries everything the build needs at a
fraction of the size, and
relax the password policy that Server Core enables by default so the
existing passwordless
`HadoopBuilder` account can still be created.
- **vcpkg.json**: narrow the Boost dependency from the `boost` metapackage
to the four ports
libhdfspp actually includes (asio, date-time, program-options, system),
and pin all 55
transitively required Boost ports to 1.86.0 — the same version
`install-boost.sh` pins for Linux.
Pin protobuf to a version vcpkg publishes.
- **libhdfspp**: reset `CMAKE_REQUIRED_INCLUDES` and
`CMAKE_REQUIRED_LIBRARIES` after the protoc
compatibility check, and migrate from the legacy `PROTOBUF_*` variables to
the
`protobuf::libprotobuf` and `protobuf::libprotoc` imported targets.
- **hadoop-yarn-capacity-scheduler-ui**: point `packageManagerScript` at the
real entry point,
`node_modules/@react-router/dev/bin.js`.
- **BUILDING.txt**: correct the Windows Docker section. It omitted that
Docker Desktop must be
switched to Windows containers; it documented the Maven repository mount
under
`ContainerAdministrator` although the Dockerfile ends with `USER
HadoopBuilder`, so the documented
mount is silently ignored; it recommended `docker cp`, which fails against
a running Hyper-V
isolated container; it omitted the container memory the build needs; and
the manual vcpkg route
still pinned the purged `2025.03.19` tag.
### How was this patch tested?
Tested in the Windows container built from this PR's own
`Dockerfile_windows_10`
(`windows/servercore:ltsc2019`, Hyper-V isolation, 8 GB), against a fresh
clone of trunk with these
changes applied.
**Full build** — a single clean `mvn clean install` with
`-Pnative-win,dist`, no resume:
- **120 of 120 modules SUCCESS**, no failures and no skipped modules
- 48:59 min against a warm Maven repository
- produces `winutils.exe`, `hadoop.dll` and `hdfs.dll`, present both in
`hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin` and inside the distribution
tarball
**winutils smoke test** — `systeminfo`, `ls`, `chmod` and `hardlink` all run
correctly against the
produced binary.
**Unit tests** — `hadoop-common`, 5564 tests in 39:02 min: 5063 passed, 38
failures, 78 errors, 385
skipped. None of the failures are attributable to this PR: no file under
`hadoop-common` or
`winutils` is modified here, so that module builds and tests as pristine
trunk. The failures are the
pre-existing Windows set (`TestSSLHttpServer`, `TestHttpServer`,
`TestShellBasedUnixGroupsMapping`,
`TestGroupFallback` and similar), plus some forked-JVM `OutOfMemoryError`s
caused by the 8 GB
container limit rather than by any code defect.
Two honest caveats, both pre-existing and outside this PR's scope:
- `hadoop checknative -a` reports `winutils: true` and `hadoop: true`, but
`zlib`, `bzip2`,
`openssl`, `ISA-L` and `PMDK` all `false`. This PR restores the ability to
build; it does not add
codec support.
- `-Drequire.openssl` does not appear to be enforced on the Windows native
path: vcpkg installs
openssl 3.6.3 and `libssl.lib`/`libcrypto.lib` are present, yet the build
reports "build does not
support openssl" and still succeeds. Worth a separate issue.
Separately, `-Dcmake.prefix.path` as documented in BUILDING.txt is read by
nothing — the only reader
in the tree is `${windows.cmake.prefix.path}` in
`hadoop-hdfs-native-client/pom.xml`, and neither
property is defined in any pom, so the documented flag left
`CMAKE_PREFIX_PATH` holding an
unexpanded literal. BUILDING.txt is corrected here. The same misspelling
also appears in
`dev-support/bin/hadoop.sh` and may deserve its own issue. Note that the
verified build above ran
with the *misspelled* flag, i.e. exactly as BUILDING.txt documented it
before this change.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.* — not applicable, no object storage code is touched.
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)? — no new dependency
is introduced. Boost, Protocol Buffers, OpenSSL and zlib were already used by
the Windows build; this PR only pins the versions vcpkg resolves at image-build
time.
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files? — not applicable, nothing new is bundled into Hadoop's
artifacts.
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
