[
https://issues.apache.org/jira/browse/HADOOP-20031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18131293#comment-18131293
]
ASF GitHub Bot commented on HADOOP-20031:
-----------------------------------------
hadoop-yetus commented on PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#issuecomment-6379776157
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 14s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 2m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 53m 44s | | trunk passed |
| +1 :green_heart: | compile | 20m 1s | | trunk passed with JDK Red
Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | compile | 20m 53s | | trunk passed with JDK Red
Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | mvnsite | 21m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 10m 38s | | trunk passed with JDK Red
Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | javadoc | 12m 32s | | trunk passed with JDK Red
Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | shadedclient | 58m 59s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 52m 27s | | the patch passed |
| +1 :green_heart: | compile | 20m 5s | | the patch passed with JDK
Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | cc | 20m 5s | | the patch passed |
| +1 :green_heart: | javac | 20m 5s | | the patch passed |
| +1 :green_heart: | compile | 20m 49s | | the patch passed with JDK
Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | cc | 20m 49s | | the patch passed |
| +1 :green_heart: | javac | 20m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 21m 46s | | the patch passed |
| +1 :green_heart: | xmllint | 0m 1s | | No new issues. |
| +1 :green_heart: | javadoc | 10m 38s | | the patch passed with JDK
Red Hat, Inc.-21.0.12.1+1-LTS |
| +1 :green_heart: | javadoc | 10m 21s | | the patch passed with JDK
Red Hat, Inc.-17.0.20.1+1-LTS |
| +1 :green_heart: | shadedclient | 59m 36s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 844m 55s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 3m 48s | | The patch does not
generate ASF License warnings. |
| | | 1208m 45s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.fs.s3a.prefetch.TestS3ACachingBlockManager |
| | hadoop.yarn.service.TestYarnNativeServices |
| | hadoop.mapreduce.v2.app.webapp.TestAMWebApp |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/8844 |
| Optional Tests | dupname asflicense codespell detsecrets jsonlint compile
cc mvnsite javac unit javadoc mvninstall shadedclient xmllint |
| uname | Linux ab52cd7ef2fe 5.15.0-198-generic #208-Ubuntu SMP Fri Sep 4
10:22:38 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 46038bf1feb960049822c6379e6f1fabeb57f3ea |
| Default Java | Red Hat, Inc.-17.0.20.1+1-LTS |
| Multi-JDK versions |
/usr/lib/jvm/java-21-openjdk-21.0.12.1.1-1.1.el8_10.x86_64:Red Hat,
Inc.-21.0.12.1+1-LTS
/usr/lib/jvm/java-17-openjdk-17.0.20.1.1-1.1.el8_10.x86_64:Red Hat,
Inc.-17.0.20.1+1-LTS |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/testReport/ |
| Max. process+thread count | 4111 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-capacity-scheduler-ui . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8844/2/console |
| versions | git=2.43.7 maven=3.9.15 xmllint=20907 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
> Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI
> -----------------------------------------------------------------------
>
> Key: HADOOP-20031
> URL: https://issues.apache.org/jira/browse/HADOOP-20031
> Project: Hadoop Common
> Issue Type: Bug
> Components: hadoop-common
> Affects Versions: 3.5.0
> Environment: Windows 10/11
> Reporter: Gautham Banasandra
> Assignee: Gautham Banasandra
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> h2. Problem
> {{mvn clean install}} with {{-Pnative-win}} cannot complete on Windows. The
> Docker image described in BUILDING.txt no longer builds at all, and even with
> a
> working image the native build fails in libhdfspp and again in the YARN
> capacity-scheduler UI. There is no single cause -- eight separate defects have
> accumulated, each of which independently stops the build.
> h2. Root causes
> h3. The build image no longer builds
> # {{Dockerfile_windows_10}} installs Azul JDK 8. Hadoop requires JDK 17, so
> the
> enforcer rejects the build immediately. This line has not been touched since
> 2022.
> # {{vcpkg install}} fails because the pinned {{protobuf 3.25.5}} cannot be
> resolved from vcpkg's version database. The pin was added by HADOOP-19605 on
> 2025-11-10, so the Windows image has been unbuildable for roughly eleven
> months.
> # The pinned vcpkg tag {{2025.03.19}} references MSYS2 packages
> ({{msys2-runtime-3.5.4-2}}) that have since been purged from the upstream
> mirror. MSYS2 keeps only current versions.
> # The {{libzstd}} and {{rsync}} MSYS2 pins in the Dockerfile have been purged
> from the same mirror for the same reason.
> h3. The native build fails
> # Boost is not pinned explicitly -- its version comes from
> {{builtin-baseline}}, so it floats as the baseline moves. Boost 1.87 removed
> {{boost/asio/io_service.hpp}}, which five libhdfspp files still include
> (ioservice.h, ioservice_impl.h, mock_connection.{h,cc},
> remote_block_reader_test.cc).
> # The protobuf/protoc compatibility check in
> {{libhdfspp/CMakeLists.txt}} leaves {{CMAKE_REQUIRED_LIBRARIES}} pointing at
> the Abseil targets. Abseil carries C++ feature requirements that a later
> C-language {{check_symbol_exists()}} cannot satisfy under the Visual Studio
> generator, so that check fails spuriously.
> # The legacy {{PROTOBUF_LIBRARY}}, {{PROTOBUF_PROTOC_LIBRARY}} and
> {{PROTOBUF_LIBRARIES}} variables resolve to import DLLs under vcpkg, which
> MSVC rejects with {{LNK1107}}.
> # {{hadoop-yarn-capacity-scheduler-ui}} sets {{packageManagerScript}} to
> {{node_modules/.bin/react-router}}. On Windows that path is an {{sh}} shim,
> which {{node}} cannot execute.
> h2. Why Linux is unaffected
> The Linux build *pins* its native dependencies inside the repository --
> {{dev-support/docker/install-boost.sh}} hard-codes Boost 1.86.0 and
> {{install-protobuf.sh}} hard-codes Protobuf 3.25.5, both built from source.
> The
> Windows build instead delegates to vcpkg and lets the versions float with the
> baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3: the
> Windows build has no equivalent of the version lock the Linux build has had
> all
> along.
> h2. Fix
> * Dockerfile: install the Microsoft Build of OpenJDK 17; move to vcpkg tag
> {{2026.07.29}}; refresh the {{libzstd}} and {{rsync}} pins; refresh CMake to
> 3.30.2. Switch the base image to {{windows/servercore}}, which carries
> everything the build needs at a fraction of the size, and relax the password
> policy that Server Core enables by default so the existing passwordless
> {{HadoopBuilder}} account can still be created.
> * {{vcpkg.json}}: narrow the Boost dependency from the {{boost}} metapackage
> to
> the four ports libhdfspp actually includes (asio, date-time,
> program-options,
> system), and pin all 55 transitively required Boost ports to 1.86.0 -- the
> same version {{install-boost.sh}} pins for Linux. Pin protobuf to a version
> vcpkg publishes.
> * {{libhdfspp}}: reset {{CMAKE_REQUIRED_INCLUDES}} and
> {{CMAKE_REQUIRED_LIBRARIES}} after the protoc compatibility check, and
> migrate from the legacy {{PROTOBUF_*}} variables to the
> {{protobuf::libprotobuf}} and {{protobuf::libprotoc}} imported targets.
> * {{hadoop-yarn-capacity-scheduler-ui}}: point {{packageManagerScript}} at the
> real entry point, {{node_modules/@react-router/dev/bin.js}}.
> * BUILDING.txt: correct the Windows Docker section. It omitted that Docker
> Desktop must be switched to Windows containers; it documented the Maven
> repository mount under {{ContainerAdministrator}} although the Dockerfile
> ends with {{USER HadoopBuilder}}, so the documented mount is silently
> ignored; it recommended {{docker cp}}, which fails against a running Hyper-V
> isolated container; it omitted the container memory the build needs; and the
> manual vcpkg route still pinned the purged {{2025.03.19}} tag.
> h2. Testing
> Verified with a single clean {{mvn clean install}} inside the Windows
> container,
> from a fresh trunk clone with these changes applied:
> * *120 of 120 modules SUCCESS*, no failures and no skipped modules
> * 48:59 min, against a warm Maven repository
> * Produces {{winutils.exe}}, {{hadoop.dll}} and {{hdfs.dll}}, present both in
> {{hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin}} and inside the distribution
> tarball
> * {{winutils.exe systeminfo}}, {{ls}}, {{chmod}} and {{hardlink}} all run
> correctly against the produced binary
> No new tests are added: these are build, packaging and documentation changes,
> and the test is that the build completes and emits working native binaries.
> h2. Notes
> While investigating, {{-Dcmake.prefix.path}} was found to be read by nothing.
> The only reader in the tree is {{${windows.cmake.prefix.path}}} in
> {{hadoop-hdfs-native-client/pom.xml}}, and neither property is defined in any
> pom, so the flag documented in BUILDING.txt left {{CMAKE_PREFIX_PATH}} holding
> an unexpanded literal. BUILDING.txt is corrected here; the same misspelling
> also
> appears in {{dev-support/bin/hadoop.sh}} and may deserve a separate issue.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
