[
https://issues.apache.org/jira/browse/HDFS-17855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18040816#comment-18040816
]
ASF GitHub Bot commented on HDFS-17855:
---------------------------------------
hadoop-yetus commented on PR #8102:
URL: https://github.com/apache/hadoop/pull/8102#issuecomment-3581232382
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
|||| _ trunk Compile Tests _ |
| +0 :ok: | mvndep | 7m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 27s | | trunk passed |
| +1 :green_heart: | compile | 15m 42s | | trunk passed with JDK
Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 15m 56s | | trunk passed with JDK
Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 2m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 38s | | trunk passed with JDK
Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 35s | | trunk passed with JDK
Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| -1 :x: | spotbugs | 3m 0s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 448 extant spotbugs
warnings. |
| -1 :x: | spotbugs | 4m 1s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-warnings.html)
| hadoop-hdfs-project/hadoop-hdfs in trunk has 291 extant spotbugs warnings.
|
| +1 :green_heart: | shadedclient | 29m 49s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 29s | | the patch passed |
| +1 :green_heart: | compile | 15m 11s | | the patch passed with JDK
Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 15m 11s | | the patch passed |
| +1 :green_heart: | compile | 16m 18s | | the patch passed with JDK
Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 9s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 24 unchanged - 0 fixed = 25 total (was
24) |
| +1 :green_heart: | mvnsite | 3m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 45s | | the patch passed with JDK
Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK
Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | spotbugs | 7m 25s | | the patch passed |
| +1 :green_heart: | shadedclient | 29m 44s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 21m 58s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 213m 18s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | asflicense | 1m 15s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 438m 13s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/8102 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c19fc805b987 5.15.0-156-generic #166-Ubuntu SMP Sat Aug 9
00:02:46 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b5428d0f5e3597d49555072bd51dc0ea7fe20984 |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions |
/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/testReport/ |
| Max. process+thread count | 3475 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8102/1/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> ViewFS with linkMergeSlash generates invalid paths during
> listStatus/listLocatedStatus operations, causing InvalidPathException or
> incorrect path resolution
> ------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17855
> URL: https://issues.apache.org/jira/browse/HDFS-17855
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: fs, viewfs
> Affects Versions: 2.10.2, 3.4.1
> Environment: * Hadoop version: 2.10.2
> * Configuration: ViewFS with linkMergeSlash enabled
> * Affected applications: JobHistoryServer, Hive, any application using ViewFS
> with linkMergeSlash
> Reporter: SeongHoon Ku
> Priority: Major
> Labels: pull-request-available
> Attachments: HADOOP-ViewFS-linkMergeSlash-fix.patch
>
>
> h1. Summary
> ViewFS with linkMergeSlash generates invalid paths during
> listStatus/listLocatedStatus operations, causing InvalidPathException or
> incorrect path resolution
> h1. Description
> When ViewFS is configured with {{linkMergeSlash}}, directory listing
> operations using *RemoteIterator* generate invalid paths, causing
> {{InvalidPathException}} errors in applications using the FileContext API.
> * Applications using *FileContext API (ViewFs)* with {{listLocatedStatus()}}
> or {{listStatusIterator()}}
> * Examples: JobHistoryServer, Hive/Tez applications
> * Specifically fails in {{ViewFs$WrappingRemoteIterator.next()}} method
> h2. Configuration Example
> {code:xml}
> <property>
> <name>fs.defaultFS</name>
> <value>viewfs://hadoop-cluster</value>
> </property>
> <property>
> <name>fs.viewfs.mounttable.hadoop-cluster.linkMergeSlash</name>
> <value>hdfs://hadoop-cluster</value>
> </property>
> {code}
> h2. Error Stack Trace
> *JobHistoryServer:*
> {noformat}
> org.apache.hadoop.fs.InvalidPathException: Invalid path name relative paths
> not allowed:
> hadoop-cluster/user/history/done/2021
> at
> org.apache.hadoop.fs.AbstractFileSystem.checkPath(AbstractFileSystem.java:370)
> at
> org.apache.hadoop.fs.AbstractFileSystem.makeQualified(AbstractFileSystem.java:428)
> at
> org.apache.hadoop.fs.viewfs.ViewFs$WrappingRemoteIterator.next(ViewFs.java:848)
> at
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:238)
> {noformat}
> *Hive (Tez):*
> {noformat}
> org.apache.tez.dag.app.dag.impl.AMUserCodeException: java.io.IOException:
> cannot find dir =
> viewfs://hadoop-cluster/user/hive/hadoop-cluster/tmp/hive/...
> {noformat}
> *Observed pattern:*
> * Invalid path:
> {{viewfs://hadoop-cluster/user/hive/hadoop-cluster/tmp/hive/...}}
> * Correct path: {{viewfs://hadoop-cluster/tmp/hive/...}}
> * Working directory and cluster name are duplicated in the path
> ----
> h1. Root Cause
> h2. Technical Analysis
> When {{linkMergeSlash}} is configured, the ViewFS root node is created with
> its path name ({{fullPath}}) incorrectly set to {{mountTableName}} instead of
> {{"/"}}.
> *Bug location in {{InodeTree.java}}:*
> {code:java}
> // Current (buggy) code
> if (isMergeSlashConfigured) {
> root = new INodeLink<T>(mountTableName, ugi, // "hadoop-cluster" - BUG!
> initAndGetTargetFs(), mergeSlashTarget);
> mountPoints.add(new MountPoint<T>("/", (INodeLink<T>) root));
> rootFallbackLink = null;
> }
> {code}
> This causes {{root.fullPath}} to be set to the cluster name (e.g.,
> {{"hadoop-cluster"}}) instead of {{"/"}}.
> h2. Impact Chain
> # During path resolution ({{InodeTree.java}}), {{root.fullPath}} is used as
> {{ResolveResult.resolvedPath}}:
> {code:java}
> if (root.isLink()) {
> ResolveResult<T> res = new ResolveResult<T>(ResultKind.EXTERNAL_DIR,
> getRootLink().getTargetFileSystem(), root.fullPath, remainingPath);
> // ^^^^^^^^^^^^^ Uses mountTableName!
> return res;
> }
> {code}
> # During path conversion in {{ViewFileSystem.getChrootedPath()}} (line 563):
> {code:java}
> return this.makeQualified(
> suffix.length() == 0 ? f : new Path(res.resolvedPath, suffix));
> // Creates: new Path("hadoop-cluster", "user/history/done")
> // Result: "hadoop-cluster/user/history/done" (RELATIVE PATH!)
> {code}
> # {{makeQualified()}} then prepends the working directory to this relative
> path:
> {noformat}
> Expected: viewfs://hadoop-cluster/user/history/done
> Actual: viewfs://hadoop-cluster/user/mapred/hadoop-cluster/user/history/done
> {noformat}
> h2. Why linkMergeSlash Should Use "/"
> {{linkMergeSlash}} is designed to merge the entire ViewFS root with a single
> target directory. Therefore:
> * ViewFS root ({{/}}) = Target directory specified by linkMergeSlash
> * The root node's {{fullPath}} should naturally be {{/}}
> * This maintains consistency with the {{MountPoint}} API which already
> returns {{/}}
> ----
> h1. Testing
> h2. Test Cases
> Added comprehensive test cases in {{TestViewFileSystemLinkMergeSlash.java}}:
> # *{{testListStatusReturnsCorrectPaths()}}*
> ** Verifies {{listStatus()}} returns proper ViewFS paths
> ** Checks scheme, authority, and path correctness
> # *{{testListLocatedStatusReturnsCorrectPaths()}}*
> ** Verifies {{listLocatedStatus()}} with RemoteIterator
> ** Ensures lazy evaluation works correctly
> # *{{testResolvedPathIsAbsolute()}}*
> ** Reproduces exact bug scenario (JobHistoryServer use case)
> ** Validates path resolution for {{/user/history/done/2021}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
