Gautham Banasandra created HADOOP-20031:
-------------------------------------------
Summary: Fix the Windows native build: Docker image, libhdfspp
CMake and YARN UI
Key: HADOOP-20031
URL: https://issues.apache.org/jira/browse/HADOOP-20031
Project: Hadoop Common
Issue Type: Bug
Components: hadoop-common
Affects Versions: 3.5.0
Environment: Windows 10/11
Reporter: Gautham Banasandra
Assignee: Gautham Banasandra
Fix For: 3.5.0
h2. Problem
{{mvn clean install}} with {{-Pnative-win}} cannot complete on Windows. The
Docker image described in BUILDING.txt no longer builds at all, and even with a
working image the native build fails in libhdfspp and again in the YARN
capacity-scheduler UI. There is no single cause -- eight separate defects have
accumulated, each of which independently stops the build.
h2. Root causes
h3. The build image no longer builds
# {{Dockerfile_windows_10}} installs Azul JDK 8. Hadoop requires JDK 17, so the
enforcer rejects the build immediately. This line has not been touched since
2022.
# {{vcpkg install}} fails because the pinned {{protobuf 3.25.5}} cannot be
resolved from vcpkg's version database. The pin was added by HADOOP-19605 on
2025-11-10, so the Windows image has been unbuildable for roughly eleven
months.
# The pinned vcpkg tag {{2025.03.19}} references MSYS2 packages
({{msys2-runtime-3.5.4-2}}) that have since been purged from the upstream
mirror. MSYS2 keeps only current versions.
# The {{libzstd}} and {{rsync}} MSYS2 pins in the Dockerfile have been purged
from the same mirror for the same reason.
h3. The native build fails
# Boost is not pinned explicitly -- its version comes from
{{builtin-baseline}}, so it floats as the baseline moves. Boost 1.87 removed
{{boost/asio/io_service.hpp}}, which five libhdfspp files still include
(ioservice.h, ioservice_impl.h, mock_connection.{h,cc},
remote_block_reader_test.cc).
# The protobuf/protoc compatibility check in
{{libhdfspp/CMakeLists.txt}} leaves {{CMAKE_REQUIRED_LIBRARIES}} pointing at
the Abseil targets. Abseil carries C++ feature requirements that a later
C-language {{check_symbol_exists()}} cannot satisfy under the Visual Studio
generator, so that check fails spuriously.
# The legacy {{PROTOBUF_LIBRARY}}, {{PROTOBUF_PROTOC_LIBRARY}} and
{{PROTOBUF_LIBRARIES}} variables resolve to import DLLs under vcpkg, which
MSVC rejects with {{LNK1107}}.
# {{hadoop-yarn-capacity-scheduler-ui}} sets {{packageManagerScript}} to
{{node_modules/.bin/react-router}}. On Windows that path is an {{sh}} shim,
which {{node}} cannot execute.
h2. Why Linux is unaffected
The Linux build *pins* its native dependencies inside the repository --
{{dev-support/docker/install-boost.sh}} hard-codes Boost 1.86.0 and
{{install-protobuf.sh}} hard-codes Protobuf 3.25.5, both built from source. The
Windows build instead delegates to vcpkg and lets the versions float with the
baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3: the
Windows build has no equivalent of the version lock the Linux build has had all
along.
h2. Fix
* Dockerfile: install the Microsoft Build of OpenJDK 17; move to vcpkg tag
{{2026.07.29}}; refresh the {{libzstd}} and {{rsync}} pins; refresh CMake to
3.30.2. Switch the base image to {{windows/servercore}}, which carries
everything the build needs at a fraction of the size, and relax the password
policy that Server Core enables by default so the existing passwordless
{{HadoopBuilder}} account can still be created.
* {{vcpkg.json}}: narrow the Boost dependency from the {{boost}} metapackage to
the four ports libhdfspp actually includes (asio, date-time, program-options,
system), and pin all 55 transitively required Boost ports to 1.86.0 -- the
same version {{install-boost.sh}} pins for Linux. Pin protobuf to a version
vcpkg publishes.
* {{libhdfspp}}: reset {{CMAKE_REQUIRED_INCLUDES}} and
{{CMAKE_REQUIRED_LIBRARIES}} after the protoc compatibility check, and
migrate from the legacy {{PROTOBUF_*}} variables to the
{{protobuf::libprotobuf}} and {{protobuf::libprotoc}} imported targets.
* {{hadoop-yarn-capacity-scheduler-ui}}: point {{packageManagerScript}} at the
real entry point, {{node_modules/@react-router/dev/bin.js}}.
* BUILDING.txt: correct the Windows Docker section. It omitted that Docker
Desktop must be switched to Windows containers; it documented the Maven
repository mount under {{ContainerAdministrator}} although the Dockerfile
ends with {{USER HadoopBuilder}}, so the documented mount is silently
ignored; it recommended {{docker cp}}, which fails against a running Hyper-V
isolated container; it omitted the container memory the build needs; and the
manual vcpkg route still pinned the purged {{2025.03.19}} tag.
h2. Testing
Verified with a single clean {{mvn clean install}} inside the Windows container,
from a fresh trunk clone with these changes applied:
* *120 of 120 modules SUCCESS*, no failures and no skipped modules
* 48:59 min, against a warm Maven repository
* Produces {{winutils.exe}}, {{hadoop.dll}} and {{hdfs.dll}}, present both in
{{hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin}} and inside the distribution
tarball
* {{winutils.exe systeminfo}}, {{ls}}, {{chmod}} and {{hardlink}} all run
correctly against the produced binary
No new tests are added: these are build, packaging and documentation changes,
and the test is that the build completes and emits working native binaries.
h2. Notes
While investigating, {{-Dcmake.prefix.path}} was found to be read by nothing.
The only reader in the tree is {{${windows.cmake.prefix.path}}} in
{{hadoop-hdfs-native-client/pom.xml}}, and neither property is defined in any
pom, so the flag documented in BUILDING.txt left {{CMAKE_PREFIX_PATH}} holding
an unexpanded literal. BUILDING.txt is corrected here; the same misspelling also
appears in {{dev-support/bin/hadoop.sh}} and may deserve a separate issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
