umamaheswararao commented on a change in pull request #2088:
URL: https://github.com/apache/hadoop/pull/2088#discussion_r443704500
##########
File path:
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/viewfs/ViewFileSystem.java
##########
@@ -1258,40 +1260,52 @@ public FileStatus getFileStatus(Path f) throws
IOException {
FileStatus status =
((ChRootedFileSystem)link.getTargetFileSystem())
.getMyFs().getFileStatus(new Path(linkedPath));
- result[i++] = new FileStatus(status.getLen(), status.isDirectory(),
- status.getReplication(), status.getBlockSize(),
- status.getModificationTime(), status.getAccessTime(),
- status.getPermission(), status.getOwner(), status.getGroup(),
- null, path);
+ linkStatuses.add(
+ new FileStatus(status.getLen(), status.isDirectory(),
+ status.getReplication(), status.getBlockSize(),
+ status.getModificationTime(), status.getAccessTime(),
+ status.getPermission(), status.getOwner(),
+ status.getGroup(), null, path));
} catch (FileNotFoundException ex) {
LOG.warn("Cannot get one of the children's(" + path
+ ") target path(" + link.getTargetFileSystem().getUri()
+ ") file status.", ex);
throw ex;
}
} else {
- result[i++] =
+ internalDirStatuses.add(
new FileStatus(0, true, 0, 0, creationTime, creationTime,
PERMISSION_555, ugi.getShortUserName(),
- ugi.getPrimaryGroupName(), path);
+ ugi.getPrimaryGroupName(), path));
}
}
+ FileStatus[] internalDirStatusesMergedWithFallBack = internalDirStatuses
+ .toArray(new FileStatus[internalDirStatuses.size()]);
if (fallbackStatuses.length > 0) {
- return consolidateFileStatuses(fallbackStatuses, result);
+ internalDirStatusesMergedWithFallBack =
+ merge(fallbackStatuses, internalDirStatusesMergedWithFallBack);
+ }
+
+ // we don't use target file status as we show the mount link as symlink.
+ if (showMountLinksAsSymlinks) {
+ return merge(internalDirStatusesMergedWithFallBack,
+ linkStatuses.toArray(new FileStatus[linkStatuses.size()]));
Review comment:
Hi @ayushtkn , thank you for questions.
```
>>Just a doubt regarding the above line.
Why in case showMountLinkAsSymlinks is true, we need to give preference to
the fallback paths?
```
If you notice in tests, I have set showMountLinkAsSymlinks to false to
represent more ViewFSOerloadScheme scenarios where we have
showMountLinkAsSymlinks=false bydefault.
In the case showMountLinkAsSymlinks=true, mount link wants to represent as
symlink. Here in symlink we will not actually taking any values from targetDirs
resolved. This is ViewFileSystem default behavior. In the case of InternalDir
we never represent as symlinks. So, this config will not be applicable to
internal dir.
Just to be clear: if mount path is /user/hibe/warehouse/partition-0 -->
o3fs://....
Here (/user or /user/hive or /user/hive/warehouse ) will be represented as
"InternalDirs"
that last dir partition-0 will only be represented as link. So, that
configuration will comeinto picture only for partition-0.
In InternalDIr case we are trying get from fallback if they are matching.
The reason is, we may want to implement to have mkdirs or other functions to
work on internalDir matching path if there is fallback. Example: mkdirs
/user/hive/warehouse2
Currently it will not work because we will say /user/hive is readOnly. But
the plan is to support and create /user/hive/warehouse2 in fallback fs. So, it
make sense to get the permission/others from fallback dir and represent. Does
this make sense? Basically we will want to apply
>>internalDirStatuses can not collide since ViewFs isn't allowing to mount
/user1/hive -> (some path) as well as /user1/hive/warehouse -> (some path)
which is allowed in RBF.
Yes, you are right. We don't allow to configure any other mount path in the
same tree. Here the idea is if there is fallback, that can be applied
everywhere where there is no link configured. That way we can bring the
scenario that a base cluster can be a fallback cluster and mount some of the
paths from it to some remote fs. So, only links will go to target fs and rest
of folder should be pointed to fallback fs. Because fallbackFS is just a single
global config, but not specific to any path.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
