[
https://issues.apache.org/jira/browse/HADOOP-20031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128826#comment-18128826
]
ASF GitHub Bot commented on HADOOP-20031:
-----------------------------------------
GauthamBanasandra opened a new pull request, #8844:
URL: https://github.com/apache/hadoop/pull/8844
### Description of PR
`mvn clean install` with `-Pnative-win` cannot complete on Windows. The
Docker image described in
BUILDING.txt no longer builds at all, and even with a working image the
native build fails in
libhdfspp and again in the YARN capacity-scheduler UI. There is no single
cause — eight separate
defects have accumulated, each of which independently stops the build.
**The build image no longer builds**
1. `Dockerfile_windows_10` installs Azul JDK 8. Hadoop requires JDK 17, so
the enforcer rejects the
build immediately. This line has not been touched since 2022.
2. `vcpkg install` fails because the pinned `protobuf 3.25.5` cannot be
resolved from vcpkg's
version database. The pin was added by HADOOP-19605 on 2025-11-10.
3. The pinned vcpkg tag `2025.03.19` references MSYS2 packages
(`msys2-runtime-3.5.4-2`) that have
since been purged from the upstream mirror. MSYS2 keeps only current
versions.
4. The `libzstd` and `rsync` MSYS2 pins in the Dockerfile have been purged
for the same reason.
**The native build fails**
5. Boost is not pinned explicitly — its version comes from
`builtin-baseline`, so it floats as the
baseline moves. Boost 1.87 removed `boost/asio/io_service.hpp`, which
five libhdfspp files still
include (`ioservice.h`, `ioservice_impl.h`, `mock_connection.{h,cc}`,
`remote_block_reader_test.cc`).
6. The protobuf/protoc compatibility check in `libhdfspp/CMakeLists.txt`
leaves
`CMAKE_REQUIRED_LIBRARIES` pointing at the Abseil targets. Abseil carries
C++ feature
requirements that a later C-language `check_symbol_exists()` cannot
satisfy under the Visual
Studio generator, so that check fails spuriously.
7. The legacy `PROTOBUF_LIBRARY`, `PROTOBUF_PROTOC_LIBRARY` and
`PROTOBUF_LIBRARIES` variables
resolve to import DLLs under vcpkg, which MSVC rejects with `LNK1107`.
8. `hadoop-yarn-capacity-scheduler-ui` sets `packageManagerScript` to
`node_modules/.bin/react-router`. On Windows that path is an `sh` shim,
which `node` cannot
execute.
**Why Linux is unaffected**
The Linux build *pins* its native dependencies inside the repository —
`dev-support/docker/install-boost.sh` hard-codes Boost 1.86.0 and
`install-protobuf.sh` hard-codes
Protobuf 3.25.5, both built from source. The Windows build instead delegates
to vcpkg and lets
versions float with the baseline. That asymmetry is the root cause of
defects 2, 5 and arguably 3:
the Windows build has no equivalent of the version lock the Linux build has
had all along.
**Fix**
- **Dockerfile**: install the Microsoft Build of OpenJDK 17; move to vcpkg
tag `2026.07.29`; refresh
the `libzstd` and `rsync` pins; refresh CMake to 3.30.2. Switch the base
image to
`windows/servercore`, which carries everything the build needs at a
fraction of the size, and
relax the password policy that Server Core enables by default so the
existing passwordless
`HadoopBuilder` account can still be created.
- **vcpkg.json**: narrow the Boost dependency from the `boost` metapackage
to the four ports
libhdfspp actually includes (asio, date-time, program-options, system),
and pin all 55
transitively required Boost ports to 1.86.0 — the same version
`install-boost.sh` pins for Linux.
Pin protobuf to a version vcpkg publishes.
- **libhdfspp**: reset `CMAKE_REQUIRED_INCLUDES` and
`CMAKE_REQUIRED_LIBRARIES` after the protoc
compatibility check, and migrate from the legacy `PROTOBUF_*` variables to
the
`protobuf::libprotobuf` and `protobuf::libprotoc` imported targets.
- **hadoop-yarn-capacity-scheduler-ui**: point `packageManagerScript` at the
real entry point,
`node_modules/@react-router/dev/bin.js`.
- **BUILDING.txt**: correct the Windows Docker section. It omitted that
Docker Desktop must be
switched to Windows containers; it documented the Maven repository mount
under
`ContainerAdministrator` although the Dockerfile ends with `USER
HadoopBuilder`, so the documented
mount is silently ignored; it recommended `docker cp`, which fails against
a running Hyper-V
isolated container; it omitted the container memory the build needs; and
the manual vcpkg route
still pinned the purged `2025.03.19` tag.
### How was this patch tested?
Tested in the Windows container built from this PR's own
`Dockerfile_windows_10`
(`windows/servercore:ltsc2019`, Hyper-V isolation, 8 GB), against a fresh
clone of trunk with these
changes applied.
**Full build** — a single clean `mvn clean install` with
`-Pnative-win,dist`, no resume:
- **120 of 120 modules SUCCESS**, no failures and no skipped modules
- 48:59 min against a warm Maven repository
- produces `winutils.exe`, `hadoop.dll` and `hdfs.dll`, present both in
`hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin` and inside the distribution
tarball
**winutils smoke test** — `systeminfo`, `ls`, `chmod` and `hardlink` all run
correctly against the
produced binary.
**Unit tests** — `hadoop-common`, 5564 tests in 39:02 min: 5063 passed, 38
failures, 78 errors, 385
skipped. None of the failures are attributable to this PR: no file under
`hadoop-common` or
`winutils` is modified here, so that module builds and tests as pristine
trunk. The failures are the
pre-existing Windows set (`TestSSLHttpServer`, `TestHttpServer`,
`TestShellBasedUnixGroupsMapping`,
`TestGroupFallback` and similar), plus some forked-JVM `OutOfMemoryError`s
caused by the 8 GB
container limit rather than by any code defect.
Two honest caveats, both pre-existing and outside this PR's scope:
- `hadoop checknative -a` reports `winutils: true` and `hadoop: true`, but
`zlib`, `bzip2`,
`openssl`, `ISA-L` and `PMDK` all `false`. This PR restores the ability to
build; it does not add
codec support.
- `-Drequire.openssl` does not appear to be enforced on the Windows native
path: vcpkg installs
openssl 3.6.3 and `libssl.lib`/`libcrypto.lib` are present, yet the build
reports "build does not
support openssl" and still succeeds. Worth a separate issue.
Separately, `-Dcmake.prefix.path` as documented in BUILDING.txt is read by
nothing — the only reader
in the tree is `${windows.cmake.prefix.path}` in
`hadoop-hdfs-native-client/pom.xml`, and neither
property is defined in any pom, so the documented flag left
`CMAKE_PREFIX_PATH` holding an
unexpanded literal. BUILDING.txt is corrected here. The same misspelling
also appears in
`dev-support/bin/hadoop.sh` and may deserve its own issue. Note that the
verified build above ran
with the *misspelled* flag, i.e. exactly as BUILDING.txt documented it
before this change.
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.* — not applicable, no object storage code is touched.
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)? — no new dependency
is introduced. Boost, Protocol Buffers, OpenSSL and zlib were already used by
the Windows build; this PR only pins the versions vcpkg resolves at image-build
time.
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files? — not applicable, nothing new is bundled into Hadoop's
artifacts.
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
Contains content generated by Claude Code.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
> Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI
> -----------------------------------------------------------------------
>
> Key: HADOOP-20031
> URL: https://issues.apache.org/jira/browse/HADOOP-20031
> Project: Hadoop Common
> Issue Type: Bug
> Components: hadoop-common
> Affects Versions: 3.5.0
> Environment: Windows 10/11
> Reporter: Gautham Banasandra
> Assignee: Gautham Banasandra
> Priority: Major
> Fix For: 3.5.0
>
>
> h2. Problem
> {{mvn clean install}} with {{-Pnative-win}} cannot complete on Windows. The
> Docker image described in BUILDING.txt no longer builds at all, and even with
> a
> working image the native build fails in libhdfspp and again in the YARN
> capacity-scheduler UI. There is no single cause -- eight separate defects have
> accumulated, each of which independently stops the build.
> h2. Root causes
> h3. The build image no longer builds
> # {{Dockerfile_windows_10}} installs Azul JDK 8. Hadoop requires JDK 17, so
> the
> enforcer rejects the build immediately. This line has not been touched since
> 2022.
> # {{vcpkg install}} fails because the pinned {{protobuf 3.25.5}} cannot be
> resolved from vcpkg's version database. The pin was added by HADOOP-19605 on
> 2025-11-10, so the Windows image has been unbuildable for roughly eleven
> months.
> # The pinned vcpkg tag {{2025.03.19}} references MSYS2 packages
> ({{msys2-runtime-3.5.4-2}}) that have since been purged from the upstream
> mirror. MSYS2 keeps only current versions.
> # The {{libzstd}} and {{rsync}} MSYS2 pins in the Dockerfile have been purged
> from the same mirror for the same reason.
> h3. The native build fails
> # Boost is not pinned explicitly -- its version comes from
> {{builtin-baseline}}, so it floats as the baseline moves. Boost 1.87 removed
> {{boost/asio/io_service.hpp}}, which five libhdfspp files still include
> (ioservice.h, ioservice_impl.h, mock_connection.{h,cc},
> remote_block_reader_test.cc).
> # The protobuf/protoc compatibility check in
> {{libhdfspp/CMakeLists.txt}} leaves {{CMAKE_REQUIRED_LIBRARIES}} pointing at
> the Abseil targets. Abseil carries C++ feature requirements that a later
> C-language {{check_symbol_exists()}} cannot satisfy under the Visual Studio
> generator, so that check fails spuriously.
> # The legacy {{PROTOBUF_LIBRARY}}, {{PROTOBUF_PROTOC_LIBRARY}} and
> {{PROTOBUF_LIBRARIES}} variables resolve to import DLLs under vcpkg, which
> MSVC rejects with {{LNK1107}}.
> # {{hadoop-yarn-capacity-scheduler-ui}} sets {{packageManagerScript}} to
> {{node_modules/.bin/react-router}}. On Windows that path is an {{sh}} shim,
> which {{node}} cannot execute.
> h2. Why Linux is unaffected
> The Linux build *pins* its native dependencies inside the repository --
> {{dev-support/docker/install-boost.sh}} hard-codes Boost 1.86.0 and
> {{install-protobuf.sh}} hard-codes Protobuf 3.25.5, both built from source.
> The
> Windows build instead delegates to vcpkg and lets the versions float with the
> baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3: the
> Windows build has no equivalent of the version lock the Linux build has had
> all
> along.
> h2. Fix
> * Dockerfile: install the Microsoft Build of OpenJDK 17; move to vcpkg tag
> {{2026.07.29}}; refresh the {{libzstd}} and {{rsync}} pins; refresh CMake to
> 3.30.2. Switch the base image to {{windows/servercore}}, which carries
> everything the build needs at a fraction of the size, and relax the password
> policy that Server Core enables by default so the existing passwordless
> {{HadoopBuilder}} account can still be created.
> * {{vcpkg.json}}: narrow the Boost dependency from the {{boost}} metapackage
> to
> the four ports libhdfspp actually includes (asio, date-time,
> program-options,
> system), and pin all 55 transitively required Boost ports to 1.86.0 -- the
> same version {{install-boost.sh}} pins for Linux. Pin protobuf to a version
> vcpkg publishes.
> * {{libhdfspp}}: reset {{CMAKE_REQUIRED_INCLUDES}} and
> {{CMAKE_REQUIRED_LIBRARIES}} after the protoc compatibility check, and
> migrate from the legacy {{PROTOBUF_*}} variables to the
> {{protobuf::libprotobuf}} and {{protobuf::libprotoc}} imported targets.
> * {{hadoop-yarn-capacity-scheduler-ui}}: point {{packageManagerScript}} at the
> real entry point, {{node_modules/@react-router/dev/bin.js}}.
> * BUILDING.txt: correct the Windows Docker section. It omitted that Docker
> Desktop must be switched to Windows containers; it documented the Maven
> repository mount under {{ContainerAdministrator}} although the Dockerfile
> ends with {{USER HadoopBuilder}}, so the documented mount is silently
> ignored; it recommended {{docker cp}}, which fails against a running Hyper-V
> isolated container; it omitted the container memory the build needs; and the
> manual vcpkg route still pinned the purged {{2025.03.19}} tag.
> h2. Testing
> Verified with a single clean {{mvn clean install}} inside the Windows
> container,
> from a fresh trunk clone with these changes applied:
> * *120 of 120 modules SUCCESS*, no failures and no skipped modules
> * 48:59 min, against a warm Maven repository
> * Produces {{winutils.exe}}, {{hadoop.dll}} and {{hdfs.dll}}, present both in
> {{hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin}} and inside the distribution
> tarball
> * {{winutils.exe systeminfo}}, {{ls}}, {{chmod}} and {{hardlink}} all run
> correctly against the produced binary
> No new tests are added: these are build, packaging and documentation changes,
> and the test is that the build completes and emits working native binaries.
> h2. Notes
> While investigating, {{-Dcmake.prefix.path}} was found to be read by nothing.
> The only reader in the tree is {{${windows.cmake.prefix.path}}} in
> {{hadoop-hdfs-native-client/pom.xml}}, and neither property is defined in any
> pom, so the flag documented in BUILDING.txt left {{CMAKE_PREFIX_PATH}} holding
> an unexpanded literal. BUILDING.txt is corrected here; the same misspelling
> also
> appears in {{dev-support/bin/hadoop.sh}} and may deserve a separate issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
