[
https://issues.apache.org/jira/browse/HADOOP-20031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HADOOP-20031:
------------------------------------
Labels: pull-request-available (was: )
> Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI
> -----------------------------------------------------------------------
>
> Key: HADOOP-20031
> URL: https://issues.apache.org/jira/browse/HADOOP-20031
> Project: Hadoop Common
> Issue Type: Bug
> Components: hadoop-common
> Affects Versions: 3.5.0
> Environment: Windows 10/11
> Reporter: Gautham Banasandra
> Assignee: Gautham Banasandra
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> h2. Problem
> {{mvn clean install}} with {{-Pnative-win}} cannot complete on Windows. The
> Docker image described in BUILDING.txt no longer builds at all, and even with
> a
> working image the native build fails in libhdfspp and again in the YARN
> capacity-scheduler UI. There is no single cause -- eight separate defects have
> accumulated, each of which independently stops the build.
> h2. Root causes
> h3. The build image no longer builds
> # {{Dockerfile_windows_10}} installs Azul JDK 8. Hadoop requires JDK 17, so
> the
> enforcer rejects the build immediately. This line has not been touched since
> 2022.
> # {{vcpkg install}} fails because the pinned {{protobuf 3.25.5}} cannot be
> resolved from vcpkg's version database. The pin was added by HADOOP-19605 on
> 2025-11-10, so the Windows image has been unbuildable for roughly eleven
> months.
> # The pinned vcpkg tag {{2025.03.19}} references MSYS2 packages
> ({{msys2-runtime-3.5.4-2}}) that have since been purged from the upstream
> mirror. MSYS2 keeps only current versions.
> # The {{libzstd}} and {{rsync}} MSYS2 pins in the Dockerfile have been purged
> from the same mirror for the same reason.
> h3. The native build fails
> # Boost is not pinned explicitly -- its version comes from
> {{builtin-baseline}}, so it floats as the baseline moves. Boost 1.87 removed
> {{boost/asio/io_service.hpp}}, which five libhdfspp files still include
> (ioservice.h, ioservice_impl.h, mock_connection.{h,cc},
> remote_block_reader_test.cc).
> # The protobuf/protoc compatibility check in
> {{libhdfspp/CMakeLists.txt}} leaves {{CMAKE_REQUIRED_LIBRARIES}} pointing at
> the Abseil targets. Abseil carries C++ feature requirements that a later
> C-language {{check_symbol_exists()}} cannot satisfy under the Visual Studio
> generator, so that check fails spuriously.
> # The legacy {{PROTOBUF_LIBRARY}}, {{PROTOBUF_PROTOC_LIBRARY}} and
> {{PROTOBUF_LIBRARIES}} variables resolve to import DLLs under vcpkg, which
> MSVC rejects with {{LNK1107}}.
> # {{hadoop-yarn-capacity-scheduler-ui}} sets {{packageManagerScript}} to
> {{node_modules/.bin/react-router}}. On Windows that path is an {{sh}} shim,
> which {{node}} cannot execute.
> h2. Why Linux is unaffected
> The Linux build *pins* its native dependencies inside the repository --
> {{dev-support/docker/install-boost.sh}} hard-codes Boost 1.86.0 and
> {{install-protobuf.sh}} hard-codes Protobuf 3.25.5, both built from source.
> The
> Windows build instead delegates to vcpkg and lets the versions float with the
> baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3: the
> Windows build has no equivalent of the version lock the Linux build has had
> all
> along.
> h2. Fix
> * Dockerfile: install the Microsoft Build of OpenJDK 17; move to vcpkg tag
> {{2026.07.29}}; refresh the {{libzstd}} and {{rsync}} pins; refresh CMake to
> 3.30.2. Switch the base image to {{windows/servercore}}, which carries
> everything the build needs at a fraction of the size, and relax the password
> policy that Server Core enables by default so the existing passwordless
> {{HadoopBuilder}} account can still be created.
> * {{vcpkg.json}}: narrow the Boost dependency from the {{boost}} metapackage
> to
> the four ports libhdfspp actually includes (asio, date-time,
> program-options,
> system), and pin all 55 transitively required Boost ports to 1.86.0 -- the
> same version {{install-boost.sh}} pins for Linux. Pin protobuf to a version
> vcpkg publishes.
> * {{libhdfspp}}: reset {{CMAKE_REQUIRED_INCLUDES}} and
> {{CMAKE_REQUIRED_LIBRARIES}} after the protoc compatibility check, and
> migrate from the legacy {{PROTOBUF_*}} variables to the
> {{protobuf::libprotobuf}} and {{protobuf::libprotoc}} imported targets.
> * {{hadoop-yarn-capacity-scheduler-ui}}: point {{packageManagerScript}} at the
> real entry point, {{node_modules/@react-router/dev/bin.js}}.
> * BUILDING.txt: correct the Windows Docker section. It omitted that Docker
> Desktop must be switched to Windows containers; it documented the Maven
> repository mount under {{ContainerAdministrator}} although the Dockerfile
> ends with {{USER HadoopBuilder}}, so the documented mount is silently
> ignored; it recommended {{docker cp}}, which fails against a running Hyper-V
> isolated container; it omitted the container memory the build needs; and the
> manual vcpkg route still pinned the purged {{2025.03.19}} tag.
> h2. Testing
> Verified with a single clean {{mvn clean install}} inside the Windows
> container,
> from a fresh trunk clone with these changes applied:
> * *120 of 120 modules SUCCESS*, no failures and no skipped modules
> * 48:59 min, against a warm Maven repository
> * Produces {{winutils.exe}}, {{hadoop.dll}} and {{hdfs.dll}}, present both in
> {{hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin}} and inside the distribution
> tarball
> * {{winutils.exe systeminfo}}, {{ls}}, {{chmod}} and {{hardlink}} all run
> correctly against the produced binary
> No new tests are added: these are build, packaging and documentation changes,
> and the test is that the build completes and emits working native binaries.
> h2. Notes
> While investigating, {{-Dcmake.prefix.path}} was found to be read by nothing.
> The only reader in the tree is {{${windows.cmake.prefix.path}}} in
> {{hadoop-hdfs-native-client/pom.xml}}, and neither property is defined in any
> pom, so the flag documented in BUILDING.txt left {{CMAKE_PREFIX_PATH}} holding
> an unexpanded literal. BUILDING.txt is corrected here; the same misspelling
> also
> appears in {{dev-support/bin/hadoop.sh}} and may deserve a separate issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
