[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuweimin updated HDFS-245: --- Attachment: symlink-0.20.0.patch this a copy of symLink14.patch for hadoop released version 0.20.0 usage: put symlink-0.20.0.patch into $HADOOP_HOME cd $HADOOP_HOME patch -p0 < symlink-0.20.0.patch ant jar mv hadoop-0.20.0-core.jar hadoop-0.20.0-core.jar.bak cp build/hadoop-0.20.1-dev-core.jar restart the hadoop system at last > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink16-mr.patch symlink16-hdfs.patch symlink16-common.patch symlink15.txt patches rebased against trunk post project split. Have not been merged in the last couple days so have conflicts with recent FileContext changes. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[
https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-245:
-
Attachment: designdocv1.txt
Uploaded a first draft of a design doc (followed the template in HADOOP-5587).
Content follows. Lemme know where I missed the boat. Thanks, Eli
h1. Symlinks Design Doc
h2. Problem definition
HDFS path resolution has the following limitations:
* Files and directories are only accessible via a single path.
* An HDFS namespace may not span multiple file systems.
Symbolic links address these limitations by providing an additional level of
indirection when resolving paths in an HDFS file system. A symbolic link is a
special type of file that contain a path to another file or directory. Paths
may be relative or absolute. Relative paths (eg {{../user}}) provide an
alternate path to a single file or directory in the file system. An absolute
path may be relative to the current file system (eg {{/user}}) or specify a URL
(eg {{hdfs://localhost:8020/foo}}) which allows the link to point to any file
or directory irrespective of the source and destination file systems.
Allowing multiple paths to resolve to the same file or directory, and HDFS
namespaces to span multiple file systems makes it easier to access files and
manage their underlying stoarge.
h2. Use Cases
If an application requires data be available by a particular path a symlink may
be used in lieu of copying the data from its current location. For example, a
user may want to create a symlink {{/data/latest}} that points to an existing
directory so that the latest data is accessible via it's current name and an
alias, eg:
{{$ hadoop fs -ln /data/20090922 /data/latest}}
The user may eventually want to archive this data so that it's accessible but
stored more efficiently. They could create an archive of the files in the
20090922 directory and make the original path a symlink to the HAR, eg:
{{$ hadoop fs -ln har:///data/20090922.har /data/20090922}}
They could also move the directory to another file system that is perhaps
lightly loaded or less expensive and make the existing directory a symlink, eg:
{{$ hadoop fs -ln hdfs://archive-host/data/20090922 /data/20090922}}
The archival file system could also be accessible via an alternative protocol
(eg FTP). In both cases the original data has moved but remains accessible by
its original path.
This technique can be used generally to balance storage by transparently making
a namespace span multiple file systems. For example, if a particular subtree of
a namespace outgrows the capabilities of the file system it resides on (eg
Namenode performance, number of files, etc) it it can be moved to a new file
system and linked into its current path.
A symbolic link is also a useful primitive that could be used to implement
atomic rename within a file system by atomically rewriting a symbolic link, or
to rename a file across partitions (if in the future a file system metadata is
partitioned across multiple hosts). See HADOOP-6240 for more info.
h2. Interaction with the Current System
The user may interact with symbolic links via the shell or indirectly via
applications (eg libhdfs clients like fuse mounts).
Symbolic links are transparent to most operations though some may want to
handle links specially. In general, the behavior should match POSIX where
appropriate. Note that linking across file systems is somehwat equivalent to
creating a symlink across mount points in POSIX.
Path resolution:
* Some commands operate on the link directly (eg stat, rm) if the link is the
target.
* Some commands (eg mv) should operate on the link target if a trailing slash
is used (eg if {{/bar}} is a link that points to a directory, {{mv /foo /bar}}
renames bar to foo while {{mv /foo /bar}} moves {{/foo}} into the directory
pointed to by bar).
* Symbolic links in archive URIs should fully resolve.
* Some APIs should operate on the link target (eg setting access and
modification times).
*Permissions*: access control properties of links are ignored, checks are
always performed against the link target (if it resides in an HDFS file
system). The ch* operations should operate directly on the target.
*Some utilities need to be link-aware*:
* distcp should not follow links by default.
* fsck should only look at each file once, could optionally report dangling
links.
* Symbolic links in HARs should be followed (so that a symlink to a HAR
preserves the original path resolution behavior).
Symbolic links exist independently of their targets, and may point to
non-existing files or directories.
h2. Requirements
Clients send the entire path to the Namenode for resolution. The path may
contain multiple links:
* If all links in the path are relative to the current file system then the
Namenode transparently (to the client) resolves the path.
* If the Namenode finds
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-245: -- Attachment: symlink17-hdfs.txt symlink17-common.txt Here is a patch that conforms to the modified consensus arrived earlier in the week. I would like everybody to review the symlink17-common.txt patch that has changes to the FileSystem API. This code is not yet tested in great detail and is uploaded only for review purposes. (Also, this patch does not yet modify the FileContext API.) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[
https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-245:
-
Attachment: designdocv2.txt
While writing tests I noticed the current API doesn't match POSIX semantics
that closely, eg if a path refers to a symlink then the symlink is not
resolved, eg {{getFileStatus}} returns the FileStatus of the link rather than
what it points to (ie behaves like {{lstat}} rather than {{stat}}), ditto for
{{open}}, {{setReplication}} etc. While some APIs should act on the symlink
itself (eg {{rename}}, {{delete}}) others need symlinks fully resolved. The
design doc should specify the intended behavior of the FileContext API wrt
symlinks. I attached an updated version and pasted the relevant section below.
What do people think?
h2. FileContext APIs
This section specifies the behavior of the FileContext API when links are
present in paths. The intent is to match POSIX semantics. For most functions,
if symlinks are supported, all links leading up to the target of a path should
automatically be resolved. Some functions will not resolve any links in a given
path. Some functions will, if given a path that refers to a symlink, operate on
the target of the symlink, while others will operate on the symlink itself. For
example, {{setReplication}} and {{getFileBlockLocations}} act on the symlink
target while {{delete}} and {{getFileStatus}} act on the symlink itself.
Behavior is specified both for filesystems that do and do not support symlinks.
To support symlink-aware utilities the FileContext API requires some new
interfaces (eg equivalent to {{lstat}}) to indicate whether a path refers to a
symlink.
- {{create}}, {{mkdir}} -- the path should not refer to a symlink since the
path must not currently exist.
- {{delete}}, {{deleteOnExit}} -- if path refers to a symlink then the symlink
is removed (like {{unlink}}).
- {{open}} -- if the given path refers to a symlink then the path is fully
resolved.
- {{set|getWorkingDirectory}} -- if the given path refers to a symlink then the
symlink is fully resolved when setting the working directory, ie if the working
directory is changed to {{/link1/link2}} then subsequent queries of the working
directory should return whatever {{link2}} points to.
- {{setReplication}} -- if the given path refers to a symlink then the path is
fully resolved.
- {{setPermission}} -- if the given path refers to a symlink then the path is
fully resolved (like {{chmod}}). Symlink access is determined by permissions of
the target of the symlink.
- {{setOwner}} -- if the given path refers to a symlink then the path is fully
resolved (like {{chown}}). We could add an {{lchown}} equivalent in the future.
- {{setTimes}} -- if the given path refers to a symlink then the path is fully
resolved. ySmlinks do not have access times.
- {{get|setFileChecksum}} -- if the given path refers to a symlink then the
path is fully resolved, ie there are no checksums associated with symlinks.
- {{getFileStatus}} -- if the given path refers to a symlink then the path is
fully resolved, ie returns the FileStatus of the file or directory the symlink
points to.
- *new* {{getLinkFileStatus}} -- like {{lstat}}, if the given path refers to a
symlink then the FileStatus of the symlink is returned, otherwise the results
as if {{getFileStatus}} was called. If symlink support is not enabled or the
underlying filesystem does not support symlinks then the results are the same
as if {{getFileStatus}} was called.
- {{isDirectory}}, {{isFile}} -- if the given path refers to a symlink then the
path is fully resolved, ie if the symlink points to a directory then
{{isDirectory}} returns true.
- *new* {{isLink}} -- returns true if the given path refers to a symlink. If
symlink support is not enabled or the underlying filesystem does not support
symlinks then {{isLink}} returns false.
- {{listStatus}} -- if the given path refers to a symlink then the path is
fully resolved, ie the result is equivalent to calling {{listStatus}} with the
target of the symlink.
- {{getFileBlockLocations}} -- if the given path refers to a symlink then the
path is fully resolved, ie symlinks are not associated with blocks.
- {{getFsStatus}} -- if the given path refers to a symlink then the path is
fully resolved, ie the FsStatus of the target of the symlink is returned.
- {{getLinkTarget}} -- only the first symlink in the given path is resolved. If
symlink support is not enabled or the underlying filesystem does not support
symlinks then an IOException is thrown.
- {{resolve}} -- all symlinks in the given path are resolved. If symlink
support is not enabled or the underlying filesystem does not support symlinks
then no symlinks are resolved.
- {{createSymlink(oldpath, newpath)}}
-- newpath should not refer to a symlink since the path must not currently
exist.
-- _No symlinks
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[
https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-245:
-
Attachment: symlink19-hdfs-delta.patch
symlink19-common-delta.patch
Here are patches that add more tests and implement the above FileContext API.
They are deltas against symlink19 so it's easier to review what's changed. They
also contain some resolutions from merging with trunk. I'm writing more tests
but wanted to post what I have so far for feedback.
Note that they change the default behavior to fully resolve symlinks and add an
API equivalent to {{lstat}} for clients that want to operate on symlinks. The
FileContext API could have been implemented without adding a new ClientProtocol
function by creating a function {{resolveFully}} that calls {{getLinkTarget}}
on the return value of {{resolve}}, but that would require an additional round
trip for all FileContext functions that need to full resolve links.
> Create symbolic links in HDFS
> -
>
> Key: HDFS-245
> URL: https://issues.apache.org/jira/browse/HDFS-245
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: dhruba borthakur
>Assignee: dhruba borthakur
> Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt,
> HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch,
> symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch,
> symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch,
> symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt,
> symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch,
> symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch,
> symlink19-hdfs.txt, symLink4.patch, symLink5.patch, symLink6.patch,
> symLink8.patch, symLink9.patch
>
>
> HDFS should support symbolic links. A symbolic link is a special type of file
> that contains a reference to another file or directory in the form of an
> absolute or relative path and that affects pathname resolution. Programs
> which read or write to files named by a symbolic link will behave as if
> operating directly on the target file. However, archiving utilities can
> handle symbolic links specially and manipulate them directly.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink20-common.patch symlink20-hdfs.patch Here are patches that combine Dhruba's latest symlink patch with the delta I posted that implements the API changes (modulo incorporating feedback from Dhruba like removing isLink in FileContext since isFile/isDirectory will be deprecated), adds more tests, and resolves merge conflicts. It passes test-core on both repos. I'm adding more tests but thought I'd ping the JIRA in case there's any additional feedback. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink22-hdfs.patch symlink22-common.patch Uploaded patches per the last comment that incorporate Doug's review and minor fixes like getting the offline image viewer test to work with symlinks. Also verified that hdfs oiv using a fsimage with symlinks look OK. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, > symlink22-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink23-common.patch symlink23-hdfs.patch Hey Sanjay, Thanks for taking a look. bq. * Add final to parameters of the new methods you have added. Fixed. bq. * GetLinkTarget() - should this be public? Good catch, made this protected. bq. * Resolve() checks isUriPathAbsolute(). Actually you also need to check that the the path is fully qualified (ie has a scheme). So if a Slash-relative path name is returned (a mistake in file system impl) the patch will *incorrectly* resolve it relative to the FileContext's default filesystem (ie its Slash). The file system should have resolved it internally in the first place. Hence it would be correct (but inefficient) for the client side to send this back to the file system in which the symlink occurred; we should simple report this an error rather then send it back to the file system. Good point, createSymlink in FileContext now makes the link target fully qualified. I added tests to TestLink that use fully qualified paths for both the link and the target. I also added an isFullyQualified method to Path (and tests to TestPath) and updated the comment for isAbsolute to match the implementation (which looks correct). Note that a slash-relative path being return is currently not a mistake in the file system impl. as the client handles all UnresolvedPathExceptions. Making the NN transparently resolve links when the target is in its namespace is definitely a worthwhile optimization. Need to add more tests first. bq. * Why pass the "this" as a parameter to resolve(this, path) - the FileContext is the "this" instance any way. Not following, resolve is a method of FSLinkResolver not FileContext. Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, > symlink22-hdfs.patch, symlink23-common.patch, symlink23-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink24-hdfs.patch Latest patch attached. Broke out the one huge test function into a bunch of junit4 style individual tests, and added more tests from the test plan. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, > symlink22-hdfs.patch, symlink23-common.patch, symlink23-hdfs.patch, > symlink24-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink-25-hdfs.patch Attached latest patch. Reflects changes necessary for adding symlinks to LocalFs and more tests, eg symlinks across file systems. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink-26-hdfs.patch Latest patch. Jives with the one posted to HADOOP-6421. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink-26-hdfs.patch > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink27-hdfs.patch Latest patch. Updated the implementation to reflect semantics discussed in HADOOP-6427 and requisite tests. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: designdocv3.txt Updated design doc. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink28-hdfs.patch Latest patch. * Implements path resolution as discussed in the above comment and HADOOP-6427 * Additional tests in TestLink to cover the above * Resolved against trunk * See HADOOP-6421 for the common changes > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink29-hdfs.patch Here's the latest patch that implements the resolution behavior posted to HDFS-245 that Doug and Sanjay and I discussed. Rebased against trunk and passes test-core. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink29-hdfs.patch Minor update to the last patch to address Cos's comment in HDFS-254. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink30-hdfs.patch Attached a minor update to the last patch to address javadoc and checkstyle. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.pat ch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink31-hdfs.patch Latest patch attached. It addresses feedback from Sanjay's review of the common patch on HADOOP-6421, mostly just pulls the bulk of the symlink tests into a common class. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink33-hdfs.patch Patch attached. Contains the hdfs side of the changes for HADOOP-6421, small reworking of permissions for createSymlink. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[
https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-245:
-
Attachment: symlink35-hdfs.patch
Hey Sanjay,
Thanks for the review. Latest patch attached, addresses your feedback and
merges in trunk.
bq. File Jira to resolve dot relative and volume-root relative names locally in
HDFS (this Jira can be completed later)
Filed HDFS-932.
bq. explicitly declare the unresolvedLinkException in methods that throw them-
we are moving towards declaring all exceptions (see HDFS-717). ClientProtocol,
Namenode, FSNamesystem, etc
Done. I had to modify quite a few functions both in hdfs and common since the
UnresolvedLinkException flows all the way up to FileContext. I left the tests
as is, I also didn't modify FileSystem.
bq. UnresolvedPathException, suggestion: adding a method called
"getUnresolvedPath for completeness.
Done.
bq. createSymlinkXXX -don't pass in permission object as parameter - it
suggests that permissions matter for symlinks
Done. I left the permissions as default since INodes do not support null
PermissionStatus or PermissionStatus' created with a null FsPermission.
Currently links are rwx everyone so access is checked on the link target. And
like files accessed to the links themselves should be subject to the containing
directory's permissions. I started to modify TestDFSPermission to cover this,
which requires converting the test to use FileContext, and just converting it
to use FileContext caused some of the tests to fail, which I'm looking into, if
you don't mind will address in a follow-on jira. I filed HDFS-876 for porting
tests over to FileContext a while back and HDFS-934 to extend those for
additional coverage after they've been ported.
bq. FSDirectory#addToParent - if not a symlink, then pass null instead of a
null string.
The symlink value needs to be a string since we serialize it to image/edit log,
ie readString and writeString don't support nulls.
bq. getExistingPathINodes() typo: javadoc for param resolveLink is missing the
param name
Fixed.
bq. Add comment that exception is thrown for symlinks occurs in the middle of
the path.
Done.
bq. The multiple if statements with same or negation of condition - can be
confusing and error prone in future
I rewrote this to use just two if that read like the logic: "If the current
node is a symlink then throw an UnresolvedPathException, unconditionally if we
are not on the last path component, and conditionally if we are on the last
path component and we want to resolve it." Think this is more clear.
{code}
if (curNode.isLink() && (!lastComp || (lastComp && resolveLink))) {
throw new UnresolvedPathException(...)
}
if (lastComp || !curNode.isDirectory()) {
break;
}
{code}
bq. FSDirectory#getFileBlocks add: if (symlink) return null.
Fixed. Shouldn't be executed in practice since we always complete a resolved
path.
bq. FSDirectory#getListing() calls getNode(srcs, false) // shouldn't
resolveLink be true
Fixed. FileContext#listStatus was not resolving links, fixed that. I added
FileContextSymlinkBaseTest#testListStatusUsingLink to test this case.
bq. FSDirectory#getPreferredBlockSize calls getNode(filename, false ) //
shouldn't resolveLink be true
Fixed. Looks like DFSClient#getBlockSize is the only caller and isn't covered
by any unit tests (you can remove the method and compile hdfs). Filed HDFS-929
to either remove it or add a test.
I replaced the throw of UnresolvedPathException in FSNameSystem, it wasn't
necessary as the call to getFileINode resolves the links. There's now only a
single place we throw UnresolvedPathException which was my original intent.
Latest patch attached here and to HADOOP-6421.
Thanks,
Eli
> Create symbolic links in HDFS
> -
>
> Key: HDFS-245
> URL: https://issues.apache.org/jira/browse/HDFS-245
> Project: Hadoop HDFS
> Issue Type: New Feature
>Reporter: dhruba borthakur
>Assignee: Eli Collins
> Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt,
> designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch,
> symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch,
> symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch,
> symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt,
> symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch,
> symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt,
> symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt,
> symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch,
> symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch,
> symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch,
> symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink36-hdfs.patch Hey Hairong, thanks for taking a look! bq. 1. INodeSymlink extends INodeFile in your patch. Should it extend INode? The current code extends INodeFile because logically a symlink is a file (that just contains a path), and it makes the change smaller (by avoiding overriding spaceConsumedInTree, collectSubtreeBlocksAndClear, computeContentSummary, and allowing all the code that handles INodeFile to cover files and symlinks). I like your suggestion though; explicitly differentiating between file and symlink inodes seems less error prone, and there's a minor win in that the code no longer has to interpret and serialize the INodeFile fields that are unused for symlinks. I made INodeSymlink extend INode and modified FSImage#loadFSImage and saveINode2Image, and ImageLoaderCurrent accordingly. Leveraged the existing method of using a negative block size value to indicate the inode type. I also had to add a couple additional checks INode#isLink in places where we were leveraging the fact that symlinks were files. 2. Should FSDirectory#mkdirs, rootDir.getExistingPathINodes(components, inodes, true) be rootDir.getExistingPathINodes(components, inodes, false)? Mkdirs does not need to resolve the symlink of the last component. Yup, there's no need for mkdir to try resolve the final link. I also added throws UnresolvedLinkException which was missing from the javadoc. 3. Your document does not discuss the symlink resolution of concat, setQuoto, and getContentSummary etc. Could you please add them? I filed HDFS-944 for FileContext and AFS to provide the full ClientProtocol interface, currently they're not reachable via FileContext or AFS. Since symlink resolution is partly implemented in FileContext clients must be ported to FileContet to use them, ie if you create a symlink via FileContext then access it via DistributedFileSystem#setQuota you'll get an UnresolvedLinkException. Symlinks won't be exposed to users until we've moved off FileSystem (HADOOP-6446). We first need to port (or rather have FileContext versions of) all the tests that haven't been covered (HDFS-876 and HDFS-934). I will update the design doc on the jira tomorrow to reflect the documentation I've put in the jira comments, and the following. Patch attached here and to HADOOP-6421, addresses the above and some additional feedback from Sanjay. Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, > symlink35-hdfs.patch, symlink36-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink37-hdfs.patch Hey Sanjay, Thanks for posting, these are all addressed in patches 34-36, modulo using FileContextTestHelper which I'll defer to when HADOOP-5916 is resolved. Attached a patch that's a minor update to the last one to fix a bug in ImageLoaderCurrent. Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, > symlink35-hdfs.patch, symlink36-hdfs.patch, symlink37-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: design-doc-v4.txt Attached the latest design doc that captures the final path resolution and API changes, as well as incorporates feedback from Hairong. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink38-hdfs.patch Attached patch merges with trunk to resolve with the DFSClient refactoring. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink39-hdfs.patch Attached patch merges with trunk. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink40-hdfs.patch Patch attached. Removes testSetTimes since it was moved to common but otherwise the same. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Patch Available (was: Open) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Open (was: Patch Available) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Patch Available (was: Open) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink41-hdfs.patch Hey Sanjay, The attached patch addresses your feedback and merges with trunk. I had to resolve a bunch of diffs with and add new code for HDFS-946 which introduced HdfsFileStatus so would be good to check out those changes in particular. Details: * Added missing throws of UnresolvedLinkException to DFSClient * Removed UnresolvedLinkException from NameNode * Fixed Lease#findPath per your suggestion, did the same to getCompleteBlocksTotal since it should also not throw an UnresolvedLinkException * Filed HADOOP-6585 and HDFS-995 to do the new FsStatus APIs and convert existing code to the new interfaces. * FSNameSystem#verifyParentDir now passes true for resolveLink, tests are in the base class so will be in an upcoming common change. * Replaced the numBlocks "> -2" check with "== -1" * When through whitespace changes, they should just remove trailing whitespaces now. * Updated HDFS-995 with your suggestions for checking #isDirectory() * I noticed INode#constructPath was using StringBuffer, changed it to use StringBuilder * FSNamesystem#metaSave no longers throws UnresolvedSymlinkException. It just use java.io.File for the local file so symlinks should be handled transparently by the local file system. * Also, for those following along, we should not have to worry about UnresolvedLinkExceptions in loadFSImage and friends because intermediate symlinks are resolved when objects are created, so the paths that get persisted are the fully resolved ones, ie creating /link/file will persist /dir/file so when we load the image we will call addToParent(.."/dir/file"..). This should be more clear when volume relative symlinks are resolved internally (HDFS-932). The code asserts on these cases. * getLinkTarget never throws an UnresolvedLinkException, it always resolves the first symlink in the path and ignores the remainder, so I left the throws clause as is Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Open (was: Patch Available) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Patch Available (was: Open) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-245: -- Resolution: Fixed Fix Version/s: 0.22.0 Release Note: HDFS-245. Adds a symlink implementation to HDFS. This complements the new symlink feature added in HADOOP-6421 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Fix For: 0.22.0 > > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
