[jira] [Commented] (HDFS-15370) listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
[
https://issues.apache.org/jira/browse/HDFS-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124002#comment-17124002
]
Srinivasu Majeti commented on HDFS-15370:
-
Thank you [~umamaheswararao] !
> listStatus and getFileStatus behave inconsistent in the case of ViewFs
> implementation for isDirectory
> -
>
> Key: HDFS-15370
> URL: https://issues.apache.org/jira/browse/HDFS-15370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Srinivasu Majeti
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: viewfs
>
> listStatus implementation in ViewFs and getFileStatus does not return
> consistent values for an element on isDirectory value. listStatus returns
> isDirectory of all softlinks as false and getFileStatus returns isDirectory
> as true.
> {code:java}
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus "/"
> FileStatus of viewfs://c3121/testme21may isDirectory:false
> FileStatus of viewfs://c3121/tmp isDirectory:false
> FileStatus of viewfs://c3121/foo isDirectory:false
> FileStatus of viewfs://c3121/tmp21may isDirectory:false
> FileStatus of viewfs://c3121/testme isDirectory:false
> FileStatus of viewfs://c3121/testme2 isDirectory:false <--- returns false
> FileStatus of / isDirectory:true
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus /testme2
> FileStatus of viewfs://c3121/testme2/dist-copynativelibs.sh isDirectory:false
> FileStatus of viewfs://c3121/testme2/newfolder isDirectory:true
> FileStatus of /testme2 isDirectory:true <--- returns true
> [hdfs@c3121-node2 ~]$ {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15370) listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
[
https://issues.apache.org/jira/browse/HDFS-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17123998#comment-17123998
]
Uma Maheswara Rao G commented on HDFS-15370:
HI [~smajeti] thank you for checking. I am just waiting for
[https://github.com/apache/hadoop/pull/2019] to go in as HADOOP-17029 also
changing the same code lines. I reviewed that Jira and provided my comments
there.
> listStatus and getFileStatus behave inconsistent in the case of ViewFs
> implementation for isDirectory
> -
>
> Key: HDFS-15370
> URL: https://issues.apache.org/jira/browse/HDFS-15370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Srinivasu Majeti
>Assignee: Uma Maheswara Rao G
>Priority: Major
> Labels: viewfs
>
> listStatus implementation in ViewFs and getFileStatus does not return
> consistent values for an element on isDirectory value. listStatus returns
> isDirectory of all softlinks as false and getFileStatus returns isDirectory
> as true.
> {code:java}
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus "/"
> FileStatus of viewfs://c3121/testme21may isDirectory:false
> FileStatus of viewfs://c3121/tmp isDirectory:false
> FileStatus of viewfs://c3121/foo isDirectory:false
> FileStatus of viewfs://c3121/tmp21may isDirectory:false
> FileStatus of viewfs://c3121/testme isDirectory:false
> FileStatus of viewfs://c3121/testme2 isDirectory:false <--- returns false
> FileStatus of / isDirectory:true
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus /testme2
> FileStatus of viewfs://c3121/testme2/dist-copynativelibs.sh isDirectory:false
> FileStatus of viewfs://c3121/testme2/newfolder isDirectory:true
> FileStatus of /testme2 isDirectory:true <--- returns true
> [hdfs@c3121-node2 ~]$ {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15370) listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
[
https://issues.apache.org/jira/browse/HDFS-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120934#comment-17120934
]
Srinivasu Majeti commented on HDFS-15370:
-
Thank you for the quick clarification [~umamaheswararao] . Can we take this up
for further review and have it resolved as we see Ambari Files View is blocked
due to this for ViewFS .
CC [~arpaga]
> listStatus and getFileStatus behave inconsistent in the case of ViewFs
> implementation for isDirectory
> -
>
> Key: HDFS-15370
> URL: https://issues.apache.org/jira/browse/HDFS-15370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Srinivasu Majeti
>Priority: Major
> Labels: viewfs
>
> listStatus implementation in ViewFs and getFileStatus does not return
> consistent values for an element on isDirectory value. listStatus returns
> isDirectory of all softlinks as false and getFileStatus returns isDirectory
> as true.
> {code:java}
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus "/"
> FileStatus of viewfs://c3121/testme21may isDirectory:false
> FileStatus of viewfs://c3121/tmp isDirectory:false
> FileStatus of viewfs://c3121/foo isDirectory:false
> FileStatus of viewfs://c3121/tmp21may isDirectory:false
> FileStatus of viewfs://c3121/testme isDirectory:false
> FileStatus of viewfs://c3121/testme2 isDirectory:false <--- returns false
> FileStatus of / isDirectory:true
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus /testme2
> FileStatus of viewfs://c3121/testme2/dist-copynativelibs.sh isDirectory:false
> FileStatus of viewfs://c3121/testme2/newfolder isDirectory:true
> FileStatus of /testme2 isDirectory:true <--- returns true
> [hdfs@c3121-node2 ~]$ {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15370) listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
[
https://issues.apache.org/jira/browse/HDFS-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116927#comment-17116927
]
Uma Maheswara Rao G commented on HDFS-15370:
Let me put out some summary about the original issue here first:
Take an example of mount link: /testme --> /a/b/c
Current behaviors:
* listStatus "/" would return FileStatus(path="/testme", *isDir=false*,
.. )
* getFileStatus("/testme") would return FileStatus(path="/testme",
*isDir=true,* )
From the implementation perspective, when doing the above operation on the
link directory directly, they will execute this operation on targetFileSystem.
Since getFileStatus runs on link, i.e "/testme", it is executing on target fs,
which was initialized with the directory "/a/b/c". Above listStatus call is
running on "/" which is not link, but it's an internal directory. So, when
executing ls on the internal directory, we are having the following checks .
{code:java}
listStatus(Path path){
if (inode.isLink()) {
return FileStatus(path, isDir=false, ...)
} else{
return FileStatus(path, isDir=true, .)
}
}
{code}
I think the correct behavior should be when it is link, it should get
fileStatus on target fs and return that directory status, instead of hardcoding
always as isDir=false.
*Proposed pseudo code:*
{code:java}
Class InternalDirViewFS{ FileStatus[] listStatus(Path path){
if (inode.isLink()) {
//gets the status from target link
FileStatus fileStatus = link.targetFileSystem
.getFileStatus(new Path(link.targetFileSystem.getUri()));
return FileStatus(path, isDir= fileStatus.isDirectory(),.);
} else{
//this is InternalDir. It's alway isDir true
return FileStatus(path, isDir=true,...);
}
}
}
{code}
Let me know if this make sense to you.
*Coming to your questions:*
{quote} Question 1:
Do we support symlink creation only in this usecase of viewfs to actual
namespace ( which is actually targeting only directories really in the target
namespace ) through core-site.xml ? Or we can do it for files also ? Dont
remember if there is any other way to do it . {quote}
There are two things mixing up here. One is mount link and other is actual
filesystem symlink. In viewFS perspective, we are trying show the mount links
as symlinks in FIleStatus. When targetFS supports it fs level symlinks, it just
honor that symlink behavior. Once target resolved from ViewFS, the target fs
behavior will not change.
For ViewFS, yes we have support only via xml config file. For target fs, if
that supports FileSystem level symlink APIs, we will just continue to support.
For ViewFS, you cannot create regular symlinks via API.
{code:java}
/**
* See {@link FileContext#createSymlink(Path, Path, boolean)}.
*/
*public* *void* createSymlink(*final* Path target, *final* Path link,
*final* *boolean* createParent) *throws* AccessControlException,
FileAlreadyExistsException, FileNotFoundException,
ParentNotDirectoryException, UnsupportedFileSystemException,
IOException {
// Supporting filesystems should override this method
*throw* *new* UnsupportedOperationException(
"Filesystem does not support symlinks!");
}
{code}
We support mount links via configuration and we represent them as symlinks in
FileStatus.
To know whether a particular fs supports actual symlinks, you can use this API
and check.
{code:java}
public boolean supportsSymlinks(){code}
{quote}Question 2:
So we get isSymlink() as true only for symlink_name (as configured in
fs.viewfs.mounttable.fsname.link./symlink_name=hdfs://namespace/target_dir)
though core-site customisation and no other hdfs cli or feature does it right
?{quote}
yes. From ViewFS perspective. Coming to target fs (that is in hdfs case), it
will support symlinks and many other fs might not support. Ex: S3aFileSystem we
are not supporting.
DistributedFileSystem.java#supportedSymlinks()
{code:java}
@Override
public boolean supportsSymlinks() {
return true;
}
{code}
Hope this answers will help. Thank you.
> listStatus and getFileStatus behave inconsistent in the case of ViewFs
> implementation for isDirectory
> -
>
> Key: HDFS-15370
> URL: https://issues.apache.org/jira/browse/HDFS-15370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Srinivasu Majeti
>Priority: Major
> Labels: viewfs
>
> listStatus implementation in ViewFs and getFileStatus does not return
> consistent values for an element on isDirectory value. listStatus returns
> isDirectory of all softlinks as false and getFileStatus
[jira] [Commented] (HDFS-15370) listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
[
https://issues.apache.org/jira/browse/HDFS-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113764#comment-17113764
]
Srinivasu Majeti commented on HDFS-15370:
-
Thank you [~umamaheswararao] for your quick confirmation.
So right now , i have two clarifications in this viewfs context:
# Do we support symlink creation only in this usecase of viewfs to actual
namespace ( which is actually targeting only directories really in the target
namespace ) through core-site.xml ? Or we can do it for files also ? Dont
remember if there is any other way to do it .
# So we get isSymlink() as true only for symlink_name (as configured in
fs.viewfs.mounttable.fsname.link./symlink_name=hdfs://namespace/target_dir)
though core-site customisation and no other hdfs cli or feature does it right ?
> listStatus and getFileStatus behave inconsistent in the case of ViewFs
> implementation for isDirectory
> -
>
> Key: HDFS-15370
> URL: https://issues.apache.org/jira/browse/HDFS-15370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Srinivasu Majeti
>Priority: Major
> Labels: viewfs
>
> listStatus implementation in ViewFs and getFileStatus does not return
> consistent values for an element on isDirectory value. listStatus returns
> isDirectory of all softlinks as false and getFileStatus returns isDirectory
> as true.
> {code:java}
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus "/"
> FileStatus of viewfs://c3121/testme21may isDirectory:false
> FileStatus of viewfs://c3121/tmp isDirectory:false
> FileStatus of viewfs://c3121/foo isDirectory:false
> FileStatus of viewfs://c3121/tmp21may isDirectory:false
> FileStatus of viewfs://c3121/testme isDirectory:false
> FileStatus of viewfs://c3121/testme2 isDirectory:false <--- returns false
> FileStatus of / isDirectory:true
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus /testme2
> FileStatus of viewfs://c3121/testme2/dist-copynativelibs.sh isDirectory:false
> FileStatus of viewfs://c3121/testme2/newfolder isDirectory:true
> FileStatus of /testme2 isDirectory:true <--- returns true
> [hdfs@c3121-node2 ~]$ {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-15370) listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectory
[
https://issues.apache.org/jira/browse/HDFS-15370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113760#comment-17113760
]
Uma Maheswara Rao G commented on HDFS-15370:
Thank you [~smajeti] for reporting it. I think you are right, in listStatus we
have checks for isLink but we don't have it for getFileStatus.
I think the motivation of showing the isDir as false was probably due to the
fact that they are links. By not representing link as regular directories in
ls, would create less confusions on navigating in tree. If any others remembers
the motivations why it was done like that please feel free to comment.
Coming to getFileStatus, I think we can make it similar to listStatus.
ListStatus checks:
{code:java}
if (inode.isLink()) {
INodeLink link = (INodeLink) inode;
result[i++] = new FileStatus(0, false, 0, 0,
creationTime, creationTime, PERMISSION_555,
ugi.getShortUserName(), ugi.getPrimaryGroupName(),
link.getTargetLink(),
new Path(inode.fullPath).makeQualified(
myUri, null));
} else {
result[i++] = new FileStatus(0, true, 0, 0,
creationTime, creationTime, PERMISSION_555,
ugi.getShortUserName(), ugi.getGroupNames()[0],
new Path(inode.fullPath).makeQualified(
myUri, null));
}
{code}
GetFileStatus:
{code:java}
return new FileStatus(0, true, 0, 0, creationTime, creationTime,
PERMISSION_555, ugi.getShortUserName(), ugi.getPrimaryGroupName(),
new Path(theInternalDir.fullPath).makeQualified(
myUri, ROOT_PATH));
{code}
> listStatus and getFileStatus behave inconsistent in the case of ViewFs
> implementation for isDirectory
> -
>
> Key: HDFS-15370
> URL: https://issues.apache.org/jira/browse/HDFS-15370
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Srinivasu Majeti
>Priority: Major
> Labels: viewfs
>
> listStatus implementation in ViewFs and getFileStatus does not return
> consistent values for an element on isDirectory value. listStatus returns
> isDirectory of all softlinks as false and getFileStatus returns isDirectory
> as true.
> {code:java}
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus "/"
> FileStatus of viewfs://c3121/testme21may isDirectory:false
> FileStatus of viewfs://c3121/tmp isDirectory:false
> FileStatus of viewfs://c3121/foo isDirectory:false
> FileStatus of viewfs://c3121/tmp21may isDirectory:false
> FileStatus of viewfs://c3121/testme isDirectory:false
> FileStatus of viewfs://c3121/testme2 isDirectory:false <--- returns false
> FileStatus of / isDirectory:true
> [hdfs@c3121-node2 ~]$ /usr/jdk64/jdk1.8.0_112/bin/java -cp `hadoop
> classpath`:./hdfs-append-1.0-SNAPSHOT.jar LauncherGetFileStatus /testme2
> FileStatus of viewfs://c3121/testme2/dist-copynativelibs.sh isDirectory:false
> FileStatus of viewfs://c3121/testme2/newfolder isDirectory:true
> FileStatus of /testme2 isDirectory:true <--- returns true
> [hdfs@c3121-node2 ~]$ {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
