[
https://issues.apache.org/jira/browse/HADOOP-20031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128846#comment-18128846
]
ASF GitHub Bot commented on HADOOP-20031:
-----------------------------------------
Copilot commented on code in PR #8844:
URL: https://github.com/apache/hadoop/pull/8844#discussion_r4316607782
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt:
##########
@@ -151,8 +151,8 @@ endif (NOT THREAD_LOCAL_SUPPORTED)
unset (PROTOC_IS_COMPATIBLE CACHE)
set (CMAKE_REQUIRED_INCLUDES ${PROTOBUF_INCLUDE_DIRS})
set (CMAKE_REQUIRED_LIBRARIES
- ${PROTOBUF_LIBRARY}
- ${PROTOBUF_PROTOC_LIBRARY}
+ protobuf::libprotobuf
+ protobuf::libprotoc
${protobuf_ABSL_USED_TARGETS})
Review Comment:
This check now links against `protobuf::libprotobuf` / `protobuf::libprotoc`
imported targets, but still relies on `${PROTOBUF_INCLUDE_DIRS}` for headers.
With Config-mode Protobuf, `${PROTOBUF_INCLUDE_DIRS}` may be unset (the target
carries include dirs instead), which can break `check_cxx_source_compiles()` by
failing to find `<google/protobuf/...>` headers. Prefer deriving required
include dirs from the imported target (e.g., via `get_target_property(...
INTERFACE_INCLUDE_DIRECTORIES ...)`) or using the correct variable for how
Protobuf is found in this build.
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program
Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI
https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile
$Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path
$Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI
https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile
C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList
'/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
+RUN powershell if (-not (Test-Path 'C:\Java\jdk17\bin\javac.exe')) { throw
'JDK 17 installation failed' }
+RUN powershell Remove-Item C:\jdk17.msi -Force
# Install Apache Maven.
RUN powershell Invoke-WebRequest -URI
https://archive.apache.org/dist/maven/maven-3/3.9.15/binaries/apache-maven-3.9.15-bin.zip
-OutFile $Env:TEMP\apache-maven-3.9.15-bin.zip
RUN powershell Expand-Archive -Path $Env:TEMP\apache-maven-3.9.15-bin.zip
-DestinationPath "C:\Maven"
-# Install CMake 3.19.0.
-RUN powershell Invoke-WebRequest -URI
https://cmake.org/files/v3.19/cmake-3.19.0-win64-x64.zip -OutFile
$Env:TEMP\cmake-3.19.0-win64-x64.zip
-RUN powershell Expand-Archive -Path $Env:TEMP\cmake-3.19.0-win64-x64.zip
-DestinationPath "C:\CMake"
+# Install CMake 3.30.2.
+RUN powershell Invoke-WebRequest -URI
https://cmake.org/files/v3.30/cmake-3.30.2-windows-x86_64.zip -OutFile
$Env:TEMP\cmake-3.30.2-windows-x86_64.zip
Review Comment:
These downloads/install steps don’t verify integrity (e.g., SHA-256) before
executing installers/unzipping. This increases supply-chain risk for the build
image. Consider adding checksum verification (or signature validation where
applicable) for the JDK MSI and CMake ZIP before installation/extraction.
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema":
"https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json";,
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
"protobuf",
"openssl",
"zlib"
],
- "builtin-baseline": "289a69379604112a433874fe8b9812dad3103341",
+ "builtin-baseline": "9e593bb18ea69cc5095e012465dcd675a822ed0d",
"overrides": [
{
"name": "protobuf",
- "version": "3.25.5"
+ "version": "4.25.1"
Review Comment:
This pins Windows Protobuf to `4.25.1`, while the PR description notes Linux
pins Protobuf `3.25.5` in-repo. Having different Protobuf major versions across
platforms can lead to divergent build behavior and harder-to-debug
inconsistencies in native components. Consider aligning the Protobuf version
across platforms (or explicitly documenting why Windows must use a newer major
and validating that the resulting native artifacts remain compatible with
project expectations).
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema":
"https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json";,
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
Review Comment:
The manifest now pins Boost by enumerating dozens of `boost-*` overrides,
which is hard to maintain as vcpkg port relationships evolve. If feasible,
consider selecting a `builtin-baseline` commit where Boost is already 1.86.0,
so you can drop most/all Boost overrides and only keep overrides for ports that
truly must diverge from that baseline (reducing long-term churn and review
surface).
##########
dev-support/docker/vcpkg/vcpkg.json:
##########
@@ -1,16 +1,239 @@
{
"$schema":
"https://raw.githubusercontent.com/microsoft/vcpkg-tool/main/docs/vcpkg.schema.json";,
"dependencies": [
- "boost",
+ "boost-asio",
+ "boost-date-time",
+ "boost-system",
+ "boost-program-options",
"protobuf",
"openssl",
"zlib"
],
- "builtin-baseline": "289a69379604112a433874fe8b9812dad3103341",
+ "builtin-baseline": "9e593bb18ea69cc5095e012465dcd675a822ed0d",
"overrides": [
{
"name": "protobuf",
- "version": "3.25.5"
+ "version": "4.25.1"
+ },
+ {
+ "name": "boost-algorithm",
+ "version": "1.86.0"
+ },
Review Comment:
The manifest now pins Boost by enumerating dozens of `boost-*` overrides,
which is hard to maintain as vcpkg port relationships evolve. If feasible,
consider selecting a `builtin-baseline` commit where Boost is already 1.86.0,
so you can drop most/all Boost overrides and only keep overrides for ports that
truly must diverge from that baseline (reducing long-term churn and review
surface).
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -50,54 +50,56 @@ RUN powershell Copy-Item -Recurse -Path 'C:\Program
Files\Git' -Destination C:\G
RUN powershell git clone https://github.com/microsoft/vcpkg.git \
&& cd vcpkg \
&& git fetch --all \
- && git checkout 2025.03.19 \
+ && git checkout 2026.07.29 \
&& .\bootstrap-vcpkg.bat
ADD vcpkg/vcpkg.json .
RUN powershell .\vcpkg\vcpkg.exe install --x-install-root .\vcpkg\installed
-# Install Azul Java 8 JDK.
-RUN powershell Invoke-WebRequest -URI
https://cdn.azul.com/zulu/bin/zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -OutFile
$Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip
-RUN powershell Expand-Archive -Path
$Env:TEMP\zulu8.62.0.19-ca-jdk8.0.332-win_x64.zip -DestinationPath "C:\Java"
+# Install Microsoft Build of OpenJDK 17.
+RUN powershell Invoke-WebRequest -URI
https://aka.ms/download-jdk/microsoft-jdk-17.0.20.1-windows-x64.msi -OutFile
C:\jdk17.msi
+RUN powershell Start-Process msiexec.exe -ArgumentList
'/i','C:\jdk17.msi','INSTALLDIR=C:\Java\jdk17\','/quiet','/norestart' -Wait
Review Comment:
These downloads/install steps don’t verify integrity (e.g., SHA-256) before
executing installers/unzipping. This increases supply-chain risk for the build
image. Consider adding checksum verification (or signature validation where
applicable) for the JDK MSI and CMake ZIP before installation/extraction.
##########
dev-support/docker/Dockerfile_windows_10:
##########
@@ -125,6 +127,13 @@ RUN powershell -Command Start-Process -FilePath
.\vcredist_x64.exe -ArgumentList
RUN powershell -Command Remove-Item vcredist_x64.exe
# Create a user HadoopBuilder with basic privileges and use it for building
Hadoop on Windows.
+# Windows Server Core enables password complexity by default (unlike the
+# Windows client image), which rejects the passwordless build account below.
+RUN powershell secedit /export /cfg "C:\pwpol.cfg"
+RUN powershell "(Get-Content C:\pwpol.cfg).Replace('PasswordComplexity = 1',
'PasswordComplexity = 0') | Out-File C:\pwpol.cfg"
+RUN powershell secedit /configure /db "C:\windows\security\local.sdb" /cfg
"C:\pwpol.cfg"
+RUN powershell Remove-Item -Force "C:\pwpol.cfg" -Confirm:$false
+
RUN powershell New-LocalUser -Name 'HadoopBuilder' -Description 'User account
for building Apache Hadoop' -Password ([securestring]::new())
-AccountNeverExpires -PasswordNeverExpires
Review Comment:
Disabling system-wide password complexity in the image weakens the
container’s local security posture beyond just the build user. A safer
alternative is to keep the policy enabled and create `HadoopBuilder` with a
compliant strong random password (the `USER` directive does not require the
password at runtime), avoiding the need to relax the policy.
##########
BUILDING.txt:
##########
@@ -603,12 +603,29 @@ still be immensely useful as a raw guide for all the
steps involved in creating
needed to build Hadoop on Windows 10.
Building using the Docker:
+Docker Desktop must first be switched to Windows containers. In the default
Linux container mode
+every command below fails immediately with "no matching manifest for
linux/amd64".
+> "%ProgramFiles%\Docker\Docker\DockerCli.exe" -SwitchWindowsEngine
+
+Wait for the engine to restart, then confirm - this must print "windows"
before proceeding.
+> docker info --format "{{.OSType}}"
+
We first need to build the Docker image for building Hadoop on Windows 10. Run
this command from
the root of the Hadoop repository.
-> docker build -t hadoop-windows-10-builder -f
.\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
+> docker build -t hadoop-windows-10-builder -m 8g -f
.\dev-support\docker\Dockerfile_windows_10 .\dev-support\docker\
+
+The -m 8g is needed because the vcpkg step compiles Boost, Protocol Buffers,
OpenSSL and Zlib from
+source and the default container memory is too small. Expect the image build
to take a few hours
+and the resulting image to be around 23 GB. Note that -m and --isolation are
options of the classic
+builder, so leave DOCKER_BUILDKIT unset; BuildKit does not accept them.
Review Comment:
The instruction to 'leave DOCKER_BUILDKIT unset' is easy to misapply if
users already have it set globally. Consider adding an explicit command example
for both cmd.exe and PowerShell to disable it (e.g., setting it to 0 / clearing
it) before running `docker build`, to reduce copy/paste failures.
##########
hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/CMakeLists.txt:
##########
@@ -171,6 +171,13 @@ if (NOT PROTOC_IS_COMPATIBLE)
compiler are compatible with each other.")
endif (NOT PROTOC_IS_COMPATIBLE)
+# The checks above leave CMAKE_REQUIRED_INCLUDES and CMAKE_REQUIRED_LIBRARIES
+# pointing at the Protobuf and Abseil targets. Abseil carries C++ feature
+# requirements, which a C-language check_symbol_exists() below cannot satisfy
+# under the Visual Studio generator. Reset them so later checks run clean.
+unset (CMAKE_REQUIRED_INCLUDES)
+unset (CMAKE_REQUIRED_LIBRARIES)
Review Comment:
Unsetting `CMAKE_REQUIRED_*` after the compatibility check changes global
state for the rest of this CMakeLists. To avoid unintended interactions with
other checks that might have set these variables earlier, consider saving the
previous values before the Protobuf check and restoring them afterward (rather
than unsetting unconditionally).
> Fix the Windows native build: Docker image, libhdfspp CMake and YARN UI
> -----------------------------------------------------------------------
>
> Key: HADOOP-20031
> URL: https://issues.apache.org/jira/browse/HADOOP-20031
> Project: Hadoop Common
> Issue Type: Bug
> Components: hadoop-common
> Affects Versions: 3.5.0
> Environment: Windows 10/11
> Reporter: Gautham Banasandra
> Assignee: Gautham Banasandra
> Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> h2. Problem
> {{mvn clean install}} with {{-Pnative-win}} cannot complete on Windows. The
> Docker image described in BUILDING.txt no longer builds at all, and even with
> a
> working image the native build fails in libhdfspp and again in the YARN
> capacity-scheduler UI. There is no single cause -- eight separate defects have
> accumulated, each of which independently stops the build.
> h2. Root causes
> h3. The build image no longer builds
> # {{Dockerfile_windows_10}} installs Azul JDK 8. Hadoop requires JDK 17, so
> the
> enforcer rejects the build immediately. This line has not been touched since
> 2022.
> # {{vcpkg install}} fails because the pinned {{protobuf 3.25.5}} cannot be
> resolved from vcpkg's version database. The pin was added by HADOOP-19605 on
> 2025-11-10, so the Windows image has been unbuildable for roughly eleven
> months.
> # The pinned vcpkg tag {{2025.03.19}} references MSYS2 packages
> ({{msys2-runtime-3.5.4-2}}) that have since been purged from the upstream
> mirror. MSYS2 keeps only current versions.
> # The {{libzstd}} and {{rsync}} MSYS2 pins in the Dockerfile have been purged
> from the same mirror for the same reason.
> h3. The native build fails
> # Boost is not pinned explicitly -- its version comes from
> {{builtin-baseline}}, so it floats as the baseline moves. Boost 1.87 removed
> {{boost/asio/io_service.hpp}}, which five libhdfspp files still include
> (ioservice.h, ioservice_impl.h, mock_connection.{h,cc},
> remote_block_reader_test.cc).
> # The protobuf/protoc compatibility check in
> {{libhdfspp/CMakeLists.txt}} leaves {{CMAKE_REQUIRED_LIBRARIES}} pointing at
> the Abseil targets. Abseil carries C++ feature requirements that a later
> C-language {{check_symbol_exists()}} cannot satisfy under the Visual Studio
> generator, so that check fails spuriously.
> # The legacy {{PROTOBUF_LIBRARY}}, {{PROTOBUF_PROTOC_LIBRARY}} and
> {{PROTOBUF_LIBRARIES}} variables resolve to import DLLs under vcpkg, which
> MSVC rejects with {{LNK1107}}.
> # {{hadoop-yarn-capacity-scheduler-ui}} sets {{packageManagerScript}} to
> {{node_modules/.bin/react-router}}. On Windows that path is an {{sh}} shim,
> which {{node}} cannot execute.
> h2. Why Linux is unaffected
> The Linux build *pins* its native dependencies inside the repository --
> {{dev-support/docker/install-boost.sh}} hard-codes Boost 1.86.0 and
> {{install-protobuf.sh}} hard-codes Protobuf 3.25.5, both built from source.
> The
> Windows build instead delegates to vcpkg and lets the versions float with the
> baseline. That asymmetry is the root cause of defects 2, 5 and arguably 3: the
> Windows build has no equivalent of the version lock the Linux build has had
> all
> along.
> h2. Fix
> * Dockerfile: install the Microsoft Build of OpenJDK 17; move to vcpkg tag
> {{2026.07.29}}; refresh the {{libzstd}} and {{rsync}} pins; refresh CMake to
> 3.30.2. Switch the base image to {{windows/servercore}}, which carries
> everything the build needs at a fraction of the size, and relax the password
> policy that Server Core enables by default so the existing passwordless
> {{HadoopBuilder}} account can still be created.
> * {{vcpkg.json}}: narrow the Boost dependency from the {{boost}} metapackage
> to
> the four ports libhdfspp actually includes (asio, date-time,
> program-options,
> system), and pin all 55 transitively required Boost ports to 1.86.0 -- the
> same version {{install-boost.sh}} pins for Linux. Pin protobuf to a version
> vcpkg publishes.
> * {{libhdfspp}}: reset {{CMAKE_REQUIRED_INCLUDES}} and
> {{CMAKE_REQUIRED_LIBRARIES}} after the protoc compatibility check, and
> migrate from the legacy {{PROTOBUF_*}} variables to the
> {{protobuf::libprotobuf}} and {{protobuf::libprotoc}} imported targets.
> * {{hadoop-yarn-capacity-scheduler-ui}}: point {{packageManagerScript}} at the
> real entry point, {{node_modules/@react-router/dev/bin.js}}.
> * BUILDING.txt: correct the Windows Docker section. It omitted that Docker
> Desktop must be switched to Windows containers; it documented the Maven
> repository mount under {{ContainerAdministrator}} although the Dockerfile
> ends with {{USER HadoopBuilder}}, so the documented mount is silently
> ignored; it recommended {{docker cp}}, which fails against a running Hyper-V
> isolated container; it omitted the container memory the build needs; and the
> manual vcpkg route still pinned the purged {{2025.03.19}} tag.
> h2. Testing
> Verified with a single clean {{mvn clean install}} inside the Windows
> container,
> from a fresh trunk clone with these changes applied:
> * *120 of 120 modules SUCCESS*, no failures and no skipped modules
> * 48:59 min, against a warm Maven repository
> * Produces {{winutils.exe}}, {{hadoop.dll}} and {{hdfs.dll}}, present both in
> {{hadoop-dist/target/hadoop-3.6.0-SNAPSHOT/bin}} and inside the distribution
> tarball
> * {{winutils.exe systeminfo}}, {{ls}}, {{chmod}} and {{hardlink}} all run
> correctly against the produced binary
> No new tests are added: these are build, packaging and documentation changes,
> and the test is that the build completes and emits working native binaries.
> h2. Notes
> While investigating, {{-Dcmake.prefix.path}} was found to be read by nothing.
> The only reader in the tree is {{${windows.cmake.prefix.path}}} in
> {{hadoop-hdfs-native-client/pom.xml}}, and neither property is defined in any
> pom, so the flag documented in BUILDING.txt left {{CMAKE_PREFIX_PATH}} holding
> an unexpanded literal. BUILDING.txt is corrected here; the same misspelling
> also
> appears in {{dev-support/bin/hadoop.sh}} and may deserve a separate issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
