[ https://issues.apache.org/jira/browse/HDFS-17246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17783690#comment-17783690 ]
Gautham Banasandra commented on HDFS-17246: ------------------------------------------- [~ayushtkn] I tried re-running the test by setting - {code:java} conf.set(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR, GenericTestUtils.getRandomizedTestDir().getAbsolutePath()); {code} Here are the changes that I made on top of today's trunk branch - {code} PS D:\projects\github\apache\hadoop> git branch --show-current trunk PS D:\projects\github\apache\hadoop> git diff diff --git a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java index 71cff2e3915..84122509787 100644 --- a/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java +++ b/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSUtilClient.java @@ -663,7 +663,8 @@ public static boolean isValidName(String src) { String element = components[i]; if (element.equals(".") || // For Windows, we must allow the : in the drive letter. - (!Shell.WINDOWS && i == 1 && element.contains(":")) || +// (!Shell.WINDOWS && i == 1 && element.contains(":")) || + (element.contains(":")) || (element.contains("/"))) { return false; } diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java index 6472a21f961..37892504a4a 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java @@ -338,7 +338,7 @@ public void serviceInit(Configuration conf) throws Exception { if (conf.get(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR) == null) { File nodeAttrDir = new File(getTestWorkDir(), "nodeattributes"); conf.set(YarnConfiguration.FS_NODE_ATTRIBUTE_STORE_ROOT_DIR, - nodeAttrDir.getCanonicalPath()); + GenericTestUtils.getRandomizedTestDir().getAbsolutePath()); } super.serviceInit( conf instanceof YarnConfiguration ? conf : new YarnConfiguration(conf)); PS D:\projects\github\apache\hadoop> {code} Here's the test that I ran - {code} @REM shadedclient - Windows mvn verify -fae^ -pl hadoop-client-modules/hadoop-client-integration-tests^ -Pdist^ -Dtar^ -Dtest=NoUnitTests^ -Dmaven.javadoc.skip=true^ -Dcheckstyle.skip=true^ -Dspotbugs.skip=true^ -Pnative-win^ -Dhttps.protocols=TLSv1.2^ -Drequire.openssl^ -Drequire.test.libhadoop^ -Dshell-executable=C:\PROGRA~1\Git\bin\bash.exe^ -Dopenssl.prefix=D:\projects\github\vcpkg\installed\x64-windows^ -Dcmake.prefix.path=D:\projects\github\vcpkg\installed\x64-windows^ -Dwindows.cmake.toolchain.file=D:\projects\github\vcpkg\scripts\buildsystems\vcpkg.cmake^ -Dwindows.cmake.build.type=RelWithDebInfo^ -Dwindows.build.hdfspp.dll=off^ -Dwindows.no.sasl=on^ -Duse.platformToolsetVersion=v142 {code} It failed with the following error, indicating that the *:* wasn't allowed in the path. {code} Caused by: java.lang.IllegalArgumentException: Pathname /H:/hadoop-client-modules/hadoop-client-integration-tests/target/test/data/9eyKX4U7Ew from H:/hadoop-client-modules/hadoop-client-integration-tests/target/test/data/9eyKX4U7Ew is not a valid DFS filename. at org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:258) at org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1497) ... 37 more [INFO] [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITUseMiniCluster.clusterUp:84 » YarnRuntime java.lang.IllegalArgumentException... [ERROR] ITUseMiniCluster.clusterUp:84 » YarnRuntime java.lang.IllegalArgumentException... [INFO] [ERROR] Tests run: 5, Failures: 0, Errors: 2, Skipped: 0 {code} The following returns a relative path - {code:java} GenericTestUtils.getRandomizedTestDir() {code} The problem has got to do with the subsequent call to - {code:java} getAbsolutePath() {code} I don't think there's any way around this, unless we get MiniYARNCluster to work with *hdfs://* paths. > Fix shaded client for building Hadoop on Windows > ------------------------------------------------ > > Key: HDFS-17246 > URL: https://issues.apache.org/jira/browse/HDFS-17246 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client > Affects Versions: 3.4.0 > Environment: Windows 10 > Reporter: Gautham Banasandra > Assignee: Gautham Banasandra > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: image-2023-11-03-17-31-14-990.png > > > Currently, the *shaded client* Yetus personality in Hadoop fails to build on > Windows - > https://github.com/apache/hadoop/blob/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/dev-support/bin/hadoop.sh#L541-L615. > This happens due to the integration test failures in Hadoop client modules - > https://github.com/apache/hadoop/tree/4c04a6768c0cb3d5081cfa5d84ffb389d92f5805/hadoop-client-modules/hadoop-client-integration-tests. > There are several issues that need to be addressed in order to get the > integration tests working - > # Set the HADOOP_HOME, needed by the Mini DFS and YARN clusters spawned by > the integration tests. > # Add Hadoop binaries to PATH, so that winutils.exe can be located. > # Create a new user with Symlink privilege in the Docker image. This is > needed for the proper working of Mini YARN cluster, spawned by the > integration tests. > # Fix a bug in DFSUtilClient.java that prevents colon ( *:* ) in the path. > The colon is used a delimiter for the PATH variable while specifying multiple > paths. However, this isn't a delimiter in the case of Windows and must be > handled appropriately. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org