[jira] [Commented] (HDFS-5293) Symlink resolution requires unnecessary RPCs
[
https://issues.apache.org/jira/browse/HDFS-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15282169#comment-15282169
]
Sangjin Lee commented on HDFS-5293:
---
There has been no movement on this for a while. We'll need to move it out of
scope for 2.8.0 soon. Let me know if you disagree.
> Symlink resolution requires unnecessary RPCs
>
>
> Key: HDFS-5293
> URL: https://issues.apache.org/jira/browse/HDFS-5293
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.0.0-alpha, 3.0.0-alpha1
>Reporter: Daryn Sharp
>Priority: Critical
>
> When the NN encounters a symlink, it throws an {{UnresolvedLinkException}}.
> This exception contains only the path that is a symlink. The client issues
> another RPC to obtain the link target, followed by another RPC with the link
> target + remainder of the original path.
> {{UnresolvedLinkException}} should be returning both the link and the target
> to avoid a costly and unnecessary intermediate RPC to obtain the link target.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-5293) Symlink resolution requires unnecessary RPCs
[
https://issues.apache.org/jira/browse/HDFS-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13788659#comment-13788659
]
Colin Patrick McCabe commented on HDFS-5293:
bq. That doesn't work in stacked/proxy filesystems or anything that wants to do
custom filtering. At a minimum, you'd need to make the NN chroot aware by
defining / to be another path, treat .. out of the root as an unresolved
link. Then the NN has to deal with cyclic links, etc. I'm not advocating any of
that.
I'm just saying the NameNode should handle resolving the symlinks that it can
resolve, and throw {{UnresolvedLinkException}} for the ones which it can't. A
symlink which points to a different scheme/authority or to something above
root obviously still needs to be handled via an {{UnresolvedLinkException}}.
Essentially, I am proposing an optimization which can used on
non-cross-filesystem symlinks. This is also how {{LocalFileSystem}} currently
works.
Anyway, we can continue discussing that optimization on HADOOP-9780 if you
want, since it's a separate issue from the RPC reduction issue you're
discussing here.
Symlink resolution requires unnecessary RPCs
Key: HDFS-5293
URL: https://issues.apache.org/jira/browse/HDFS-5293
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Critical
When the NN encounters a symlink, it throws an {{UnresolvedLinkException}}.
This exception contains only the path that is a symlink. The client issues
another RPC to obtain the link target, followed by another RPC with the link
target + remainder of the original path.
{{UnresolvedLinkException}} should be returning both the link and the target
to avoid a costly and unnecessary intermediate RPC to obtain the link target.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5293) Symlink resolution requires unnecessary RPCs
[
https://issues.apache.org/jira/browse/HDFS-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13786533#comment-13786533
]
Daryn Sharp commented on HDFS-5293:
---
That doesn't work in stacked/proxy filesystems or anything that wants to do
custom filtering. At a minimum, you'd need to make the NN chroot aware by
defining / to be another path, treat .. out of the root as an unresolved
link. Then the NN has to deal with cyclic links, etc. I'm not advocating any
of that.
Resolving symlinks is purely a client side operation. Think of unix: a mounted
fs does not resolve symlinks. Libc functions on the client do all the
resolution.
Symlink resolution requires unnecessary RPCs
Key: HDFS-5293
URL: https://issues.apache.org/jira/browse/HDFS-5293
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Critical
When the NN encounters a symlink, it throws an {{UnresolvedLinkException}}.
This exception contains only the path that is a symlink. The client issues
another RPC to obtain the link target, followed by another RPC with the link
target + remainder of the original path.
{{UnresolvedLinkException}} should be returning both the link and the target
to avoid a costly and unnecessary intermediate RPC to obtain the link target.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Commented] (HDFS-5293) Symlink resolution requires unnecessary RPCs
[
https://issues.apache.org/jira/browse/HDFS-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13785706#comment-13785706
]
Colin Patrick McCabe commented on HDFS-5293:
If we're going to do this, why not just make all NN operations resolve symlinks
as far as they can? That would remove all the performance concerns about
returning unresolved paths, at least in the context of non-cross-FS symlinks.
We already have many filesystems that do symlink resolution internally, such as
LocalFileSystem, Ceph, etc. etc. If HDFS's namenode did symlink resolution for
all RPCs, we could return unresolved paths everywhere and be happy.
Symlink resolution requires unnecessary RPCs
Key: HDFS-5293
URL: https://issues.apache.org/jira/browse/HDFS-5293
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Priority: Critical
When the NN encounters a symlink, it throws an {{UnresolvedLinkException}}.
This exception contains only the path that is a symlink. The client issues
another RPC to obtain the link target, followed by another RPC with the link
target + remainder of the original path.
{{UnresolvedLinkException}} should be returning both the link and the target
to avoid a costly and unnecessary intermediate RPC to obtain the link target.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
