[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuweimin updated HDFS-245: --- Attachment: symlink-0.20.0.patch this a copy of symLink14.patch for hadoop released version 0.20.0 usage: put symlink-0.20.0.patch into $HADOOP_HOME cd $HADOOP_HOME patch -p0 < symlink-0.20.0.patch ant jar mv hadoop-0.20.0-core.jar hadoop-0.20.0-core.jar.bak cp build/hadoop-0.20.1-dev-core.jar restart the hadoop system at last > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink16-mr.patch symlink16-hdfs.patch symlink16-common.patch symlink15.txt patches rebased against trunk post project split. Have not been merged in the last couple days so have conflicts with recent FileContext changes. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, HADOOP-4044-strawman.patch, > symlink-0.20.0.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: designdocv1.txt Uploaded a first draft of a design doc (followed the template in HADOOP-5587). Content follows. Lemme know where I missed the boat. Thanks, Eli h1. Symlinks Design Doc h2. Problem definition HDFS path resolution has the following limitations: * Files and directories are only accessible via a single path. * An HDFS namespace may not span multiple file systems. Symbolic links address these limitations by providing an additional level of indirection when resolving paths in an HDFS file system. A symbolic link is a special type of file that contain a path to another file or directory. Paths may be relative or absolute. Relative paths (eg {{../user}}) provide an alternate path to a single file or directory in the file system. An absolute path may be relative to the current file system (eg {{/user}}) or specify a URL (eg {{hdfs://localhost:8020/foo}}) which allows the link to point to any file or directory irrespective of the source and destination file systems. Allowing multiple paths to resolve to the same file or directory, and HDFS namespaces to span multiple file systems makes it easier to access files and manage their underlying stoarge. h2. Use Cases If an application requires data be available by a particular path a symlink may be used in lieu of copying the data from its current location. For example, a user may want to create a symlink {{/data/latest}} that points to an existing directory so that the latest data is accessible via it's current name and an alias, eg: {{$ hadoop fs -ln /data/20090922 /data/latest}} The user may eventually want to archive this data so that it's accessible but stored more efficiently. They could create an archive of the files in the 20090922 directory and make the original path a symlink to the HAR, eg: {{$ hadoop fs -ln har:///data/20090922.har /data/20090922}} They could also move the directory to another file system that is perhaps lightly loaded or less expensive and make the existing directory a symlink, eg: {{$ hadoop fs -ln hdfs://archive-host/data/20090922 /data/20090922}} The archival file system could also be accessible via an alternative protocol (eg FTP). In both cases the original data has moved but remains accessible by its original path. This technique can be used generally to balance storage by transparently making a namespace span multiple file systems. For example, if a particular subtree of a namespace outgrows the capabilities of the file system it resides on (eg Namenode performance, number of files, etc) it it can be moved to a new file system and linked into its current path. A symbolic link is also a useful primitive that could be used to implement atomic rename within a file system by atomically rewriting a symbolic link, or to rename a file across partitions (if in the future a file system metadata is partitioned across multiple hosts). See HADOOP-6240 for more info. h2. Interaction with the Current System The user may interact with symbolic links via the shell or indirectly via applications (eg libhdfs clients like fuse mounts). Symbolic links are transparent to most operations though some may want to handle links specially. In general, the behavior should match POSIX where appropriate. Note that linking across file systems is somehwat equivalent to creating a symlink across mount points in POSIX. Path resolution: * Some commands operate on the link directly (eg stat, rm) if the link is the target. * Some commands (eg mv) should operate on the link target if a trailing slash is used (eg if {{/bar}} is a link that points to a directory, {{mv /foo /bar}} renames bar to foo while {{mv /foo /bar}} moves {{/foo}} into the directory pointed to by bar). * Symbolic links in archive URIs should fully resolve. * Some APIs should operate on the link target (eg setting access and modification times). *Permissions*: access control properties of links are ignored, checks are always performed against the link target (if it resides in an HDFS file system). The ch* operations should operate directly on the target. *Some utilities need to be link-aware*: * distcp should not follow links by default. * fsck should only look at each file once, could optionally report dangling links. * Symbolic links in HARs should be followed (so that a symlink to a HAR preserves the original path resolution behavior). Symbolic links exist independently of their targets, and may point to non-existing files or directories. h2. Requirements Clients send the entire path to the Namenode for resolution. The path may contain multiple links: * If all links in the path are relative to the current file system then the Namenode transparently (to the client) resolves the path. * If the Namenode finds
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] dhruba borthakur updated HDFS-245: -- Attachment: symlink17-hdfs.txt symlink17-common.txt Here is a patch that conforms to the modified consensus arrived earlier in the week. I would like everybody to review the symlink17-common.txt patch that has changes to the FileSystem API. This code is not yet tested in great detail and is uploaded only for review purposes. (Also, this patch does not yet modify the FileContext API.) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: designdocv2.txt While writing tests I noticed the current API doesn't match POSIX semantics that closely, eg if a path refers to a symlink then the symlink is not resolved, eg {{getFileStatus}} returns the FileStatus of the link rather than what it points to (ie behaves like {{lstat}} rather than {{stat}}), ditto for {{open}}, {{setReplication}} etc. While some APIs should act on the symlink itself (eg {{rename}}, {{delete}}) others need symlinks fully resolved. The design doc should specify the intended behavior of the FileContext API wrt symlinks. I attached an updated version and pasted the relevant section below. What do people think? h2. FileContext APIs This section specifies the behavior of the FileContext API when links are present in paths. The intent is to match POSIX semantics. For most functions, if symlinks are supported, all links leading up to the target of a path should automatically be resolved. Some functions will not resolve any links in a given path. Some functions will, if given a path that refers to a symlink, operate on the target of the symlink, while others will operate on the symlink itself. For example, {{setReplication}} and {{getFileBlockLocations}} act on the symlink target while {{delete}} and {{getFileStatus}} act on the symlink itself. Behavior is specified both for filesystems that do and do not support symlinks. To support symlink-aware utilities the FileContext API requires some new interfaces (eg equivalent to {{lstat}}) to indicate whether a path refers to a symlink. - {{create}}, {{mkdir}} -- the path should not refer to a symlink since the path must not currently exist. - {{delete}}, {{deleteOnExit}} -- if path refers to a symlink then the symlink is removed (like {{unlink}}). - {{open}} -- if the given path refers to a symlink then the path is fully resolved. - {{set|getWorkingDirectory}} -- if the given path refers to a symlink then the symlink is fully resolved when setting the working directory, ie if the working directory is changed to {{/link1/link2}} then subsequent queries of the working directory should return whatever {{link2}} points to. - {{setReplication}} -- if the given path refers to a symlink then the path is fully resolved. - {{setPermission}} -- if the given path refers to a symlink then the path is fully resolved (like {{chmod}}). Symlink access is determined by permissions of the target of the symlink. - {{setOwner}} -- if the given path refers to a symlink then the path is fully resolved (like {{chown}}). We could add an {{lchown}} equivalent in the future. - {{setTimes}} -- if the given path refers to a symlink then the path is fully resolved. ySmlinks do not have access times. - {{get|setFileChecksum}} -- if the given path refers to a symlink then the path is fully resolved, ie there are no checksums associated with symlinks. - {{getFileStatus}} -- if the given path refers to a symlink then the path is fully resolved, ie returns the FileStatus of the file or directory the symlink points to. - *new* {{getLinkFileStatus}} -- like {{lstat}}, if the given path refers to a symlink then the FileStatus of the symlink is returned, otherwise the results as if {{getFileStatus}} was called. If symlink support is not enabled or the underlying filesystem does not support symlinks then the results are the same as if {{getFileStatus}} was called. - {{isDirectory}}, {{isFile}} -- if the given path refers to a symlink then the path is fully resolved, ie if the symlink points to a directory then {{isDirectory}} returns true. - *new* {{isLink}} -- returns true if the given path refers to a symlink. If symlink support is not enabled or the underlying filesystem does not support symlinks then {{isLink}} returns false. - {{listStatus}} -- if the given path refers to a symlink then the path is fully resolved, ie the result is equivalent to calling {{listStatus}} with the target of the symlink. - {{getFileBlockLocations}} -- if the given path refers to a symlink then the path is fully resolved, ie symlinks are not associated with blocks. - {{getFsStatus}} -- if the given path refers to a symlink then the path is fully resolved, ie the FsStatus of the target of the symlink is returned. - {{getLinkTarget}} -- only the first symlink in the given path is resolved. If symlink support is not enabled or the underlying filesystem does not support symlinks then an IOException is thrown. - {{resolve}} -- all symlinks in the given path are resolved. If symlink support is not enabled or the underlying filesystem does not support symlinks then no symlinks are resolved. - {{createSymlink(oldpath, newpath)}} -- newpath should not refer to a symlink since the path must not currently exist. -- _No symlinks
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink19-hdfs-delta.patch symlink19-common-delta.patch Here are patches that add more tests and implement the above FileContext API. They are deltas against symlink19 so it's easier to review what's changed. They also contain some resolutions from merging with trunk. I'm writing more tests but wanted to post what I have so far for feedback. Note that they change the default behavior to fully resolve symlinks and add an API equivalent to {{lstat}} for clients that want to operate on symlinks. The FileContext API could have been implemented without adding a new ClientProtocol function by creating a function {{resolveFully}} that calls {{getLinkTarget}} on the return value of {{resolve}}, but that would require an additional round trip for all FileContext functions that need to full resolve links. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink20-common.patch symlink20-hdfs.patch Here are patches that combine Dhruba's latest symlink patch with the delta I posted that implements the API changes (modulo incorporating feedback from Dhruba like removing isLink in FileContext since isFile/isDirectory will be deprecated), adds more tests, and resolves merge conflicts. It passes test-core on both repos. I'm adding more tests but thought I'd ping the JIRA in case there's any additional feedback. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink22-hdfs.patch symlink22-common.patch Uploaded patches per the last comment that incorporate Doug's review and minor fixes like getting the offline image viewer test to work with symlinks. Also verified that hdfs oiv using a fsimage with symlinks look OK. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, > symlink22-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink23-common.patch symlink23-hdfs.patch Hey Sanjay, Thanks for taking a look. bq. * Add final to parameters of the new methods you have added. Fixed. bq. * GetLinkTarget() - should this be public? Good catch, made this protected. bq. * Resolve() checks isUriPathAbsolute(). Actually you also need to check that the the path is fully qualified (ie has a scheme). So if a Slash-relative path name is returned (a mistake in file system impl) the patch will *incorrectly* resolve it relative to the FileContext's default filesystem (ie its Slash). The file system should have resolved it internally in the first place. Hence it would be correct (but inefficient) for the client side to send this back to the file system in which the symlink occurred; we should simple report this an error rather then send it back to the file system. Good point, createSymlink in FileContext now makes the link target fully qualified. I added tests to TestLink that use fully qualified paths for both the link and the target. I also added an isFullyQualified method to Path (and tests to TestPath) and updated the comment for isAbsolute to match the implementation (which looks correct). Note that a slash-relative path being return is currently not a mistake in the file system impl. as the client handles all UnresolvedPathExceptions. Making the NN transparently resolve links when the target is in its namespace is definitely a worthwhile optimization. Need to add more tests first. bq. * Why pass the "this" as a parameter to resolve(this, path) - the FileContext is the "this" instance any way. Not following, resolve is a method of FSLinkResolver not FileContext. Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, > symlink22-hdfs.patch, symlink23-common.patch, symlink23-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink24-hdfs.patch Latest patch attached. Broke out the one huge test function into a bunch of junit4 style individual tests, and added more tests from the test plan. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: dhruba borthakur > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symLink1.patch, > symLink1.patch, symLink11.patch, symLink12.patch, symLink13.patch, > symLink14.patch, symLink15.txt, symLink15.txt, symlink16-common.patch, > symlink16-hdfs.patch, symlink16-mr.patch, symlink17-common.txt, > symlink17-hdfs.txt, symlink18-common.txt, symlink19-common-delta.patch, > symlink19-common.txt, symlink19-common.txt, symlink19-hdfs-delta.patch, > symlink19-hdfs.txt, symlink20-common.patch, symlink20-hdfs.patch, > symlink21-common.patch, symlink21-hdfs.patch, symlink22-common.patch, > symlink22-hdfs.patch, symlink23-common.patch, symlink23-hdfs.patch, > symlink24-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink-25-hdfs.patch Attached latest patch. Reflects changes necessary for adding symlinks to LocalFs and more tests, eg symlinks across file systems. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink-26-hdfs.patch Latest patch. Jives with the one posted to HADOOP-6421. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, symLink11.patch, > symLink12.patch, symLink13.patch, symLink14.patch, symLink15.txt, > symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink-26-hdfs.patch > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink27-hdfs.patch Latest patch. Updated the implementation to reflect semantics discussed in HADOOP-6427 and requisite tests. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: designdocv3.txt Updated design doc. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink28-hdfs.patch Latest patch. * Implements path resolution as discussed in the above comment and HADOOP-6427 * Additional tests in TestLink to cover the above * Resolved against trunk * See HADOOP-6421 for the common changes > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink29-hdfs.patch Here's the latest patch that implements the resolution behavior posted to HDFS-245 that Doug and Sanjay and I discussed. Rebased against trunk and passes test-core. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink29-hdfs.patch Minor update to the last patch to address Cos's comment in HDFS-254. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink30-hdfs.patch Attached a minor update to the last patch to address javadoc and checkstyle. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.pat ch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink31-hdfs.patch Latest patch attached. It addresses feedback from Sanjay's review of the common patch on HADOOP-6421, mostly just pulls the bulk of the symlink tests into a common class. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink33-hdfs.patch Patch attached. Contains the hdfs side of the changes for HADOOP-6421, small reworking of permissions for createSymlink. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink35-hdfs.patch Hey Sanjay, Thanks for the review. Latest patch attached, addresses your feedback and merges in trunk. bq. File Jira to resolve dot relative and volume-root relative names locally in HDFS (this Jira can be completed later) Filed HDFS-932. bq. explicitly declare the unresolvedLinkException in methods that throw them- we are moving towards declaring all exceptions (see HDFS-717). ClientProtocol, Namenode, FSNamesystem, etc Done. I had to modify quite a few functions both in hdfs and common since the UnresolvedLinkException flows all the way up to FileContext. I left the tests as is, I also didn't modify FileSystem. bq. UnresolvedPathException, suggestion: adding a method called "getUnresolvedPath for completeness. Done. bq. createSymlinkXXX -don't pass in permission object as parameter - it suggests that permissions matter for symlinks Done. I left the permissions as default since INodes do not support null PermissionStatus or PermissionStatus' created with a null FsPermission. Currently links are rwx everyone so access is checked on the link target. And like files accessed to the links themselves should be subject to the containing directory's permissions. I started to modify TestDFSPermission to cover this, which requires converting the test to use FileContext, and just converting it to use FileContext caused some of the tests to fail, which I'm looking into, if you don't mind will address in a follow-on jira. I filed HDFS-876 for porting tests over to FileContext a while back and HDFS-934 to extend those for additional coverage after they've been ported. bq. FSDirectory#addToParent - if not a symlink, then pass null instead of a null string. The symlink value needs to be a string since we serialize it to image/edit log, ie readString and writeString don't support nulls. bq. getExistingPathINodes() typo: javadoc for param resolveLink is missing the param name Fixed. bq. Add comment that exception is thrown for symlinks occurs in the middle of the path. Done. bq. The multiple if statements with same or negation of condition - can be confusing and error prone in future I rewrote this to use just two if that read like the logic: "If the current node is a symlink then throw an UnresolvedPathException, unconditionally if we are not on the last path component, and conditionally if we are on the last path component and we want to resolve it." Think this is more clear. {code} if (curNode.isLink() && (!lastComp || (lastComp && resolveLink))) { throw new UnresolvedPathException(...) } if (lastComp || !curNode.isDirectory()) { break; } {code} bq. FSDirectory#getFileBlocks add: if (symlink) return null. Fixed. Shouldn't be executed in practice since we always complete a resolved path. bq. FSDirectory#getListing() calls getNode(srcs, false) // shouldn't resolveLink be true Fixed. FileContext#listStatus was not resolving links, fixed that. I added FileContextSymlinkBaseTest#testListStatusUsingLink to test this case. bq. FSDirectory#getPreferredBlockSize calls getNode(filename, false ) // shouldn't resolveLink be true Fixed. Looks like DFSClient#getBlockSize is the only caller and isn't covered by any unit tests (you can remove the method and compile hdfs). Filed HDFS-929 to either remove it or add a test. I replaced the throw of UnresolvedPathException in FSNameSystem, it wasn't necessary as the call to getFileINode resolves the links. There's now only a single place we throw UnresolvedPathException which was my original intent. Latest patch attached here and to HADOOP-6421. Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink36-hdfs.patch Hey Hairong, thanks for taking a look! bq. 1. INodeSymlink extends INodeFile in your patch. Should it extend INode? The current code extends INodeFile because logically a symlink is a file (that just contains a path), and it makes the change smaller (by avoiding overriding spaceConsumedInTree, collectSubtreeBlocksAndClear, computeContentSummary, and allowing all the code that handles INodeFile to cover files and symlinks). I like your suggestion though; explicitly differentiating between file and symlink inodes seems less error prone, and there's a minor win in that the code no longer has to interpret and serialize the INodeFile fields that are unused for symlinks. I made INodeSymlink extend INode and modified FSImage#loadFSImage and saveINode2Image, and ImageLoaderCurrent accordingly. Leveraged the existing method of using a negative block size value to indicate the inode type. I also had to add a couple additional checks INode#isLink in places where we were leveraging the fact that symlinks were files. 2. Should FSDirectory#mkdirs, rootDir.getExistingPathINodes(components, inodes, true) be rootDir.getExistingPathINodes(components, inodes, false)? Mkdirs does not need to resolve the symlink of the last component. Yup, there's no need for mkdir to try resolve the final link. I also added throws UnresolvedLinkException which was missing from the javadoc. 3. Your document does not discuss the symlink resolution of concat, setQuoto, and getContentSummary etc. Could you please add them? I filed HDFS-944 for FileContext and AFS to provide the full ClientProtocol interface, currently they're not reachable via FileContext or AFS. Since symlink resolution is partly implemented in FileContext clients must be ported to FileContet to use them, ie if you create a symlink via FileContext then access it via DistributedFileSystem#setQuota you'll get an UnresolvedLinkException. Symlinks won't be exposed to users until we've moved off FileSystem (HADOOP-6446). We first need to port (or rather have FileContext versions of) all the tests that haven't been covered (HDFS-876 and HDFS-934). I will update the design doc on the jira tomorrow to reflect the documentation I've put in the jira comments, and the following. Patch attached here and to HADOOP-6421, addresses the above and some additional feedback from Sanjay. Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, > symlink35-hdfs.patch, symlink36-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink37-hdfs.patch Hey Sanjay, Thanks for posting, these are all addressed in patches 34-36, modulo using FileContextTestHelper which I'll defer to when HADOOP-5916 is resolved. Attached a patch that's a minor update to the last one to fix a bug in ImageLoaderCurrent. Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, designdocv1.txt, designdocv2.txt, > designdocv3.txt, HADOOP-4044-strawman.patch, symlink-0.20.0.patch, > symlink-25-hdfs.patch, symlink-26-hdfs.patch, symlink-26-hdfs.patch, > symLink1.patch, symLink1.patch, symLink11.patch, symLink12.patch, > symLink13.patch, symLink14.patch, symLink15.txt, symLink15.txt, > symlink16-common.patch, symlink16-hdfs.patch, symlink16-mr.patch, > symlink17-common.txt, symlink17-hdfs.txt, symlink18-common.txt, > symlink19-common-delta.patch, symlink19-common.txt, symlink19-common.txt, > symlink19-hdfs-delta.patch, symlink19-hdfs.txt, symlink20-common.patch, > symlink20-hdfs.patch, symlink21-common.patch, symlink21-hdfs.patch, > symlink22-common.patch, symlink22-hdfs.patch, symlink23-common.patch, > symlink23-hdfs.patch, symlink24-hdfs.patch, symlink27-hdfs.patch, > symlink28-hdfs.patch, symlink29-hdfs.patch, symlink29-hdfs.patch, > symlink30-hdfs.patch, symlink31-hdfs.patch, symlink33-hdfs.patch, > symlink35-hdfs.patch, symlink36-hdfs.patch, symlink37-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: design-doc-v4.txt Attached the latest design doc that captures the final path resolution and API changes, as well as incorporates feedback from Hairong. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symLink4.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink38-hdfs.patch Attached patch merges with trunk to resolve with the DFSClient refactoring. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symLink4.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink39-hdfs.patch Attached patch merges with trunk. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symLink5.patch, symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink40-hdfs.patch Patch attached. Removes testSetTimes since it was moved to common but otherwise the same. > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Patch Available (was: Open) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Open (was: Patch Available) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Patch Available (was: Open) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symLink5.patch, symLink6.patch, > symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Attachment: symlink41-hdfs.patch Hey Sanjay, The attached patch addresses your feedback and merges with trunk. I had to resolve a bunch of diffs with and add new code for HDFS-946 which introduced HdfsFileStatus so would be good to check out those changes in particular. Details: * Added missing throws of UnresolvedLinkException to DFSClient * Removed UnresolvedLinkException from NameNode * Fixed Lease#findPath per your suggestion, did the same to getCompleteBlocksTotal since it should also not throw an UnresolvedLinkException * Filed HADOOP-6585 and HDFS-995 to do the new FsStatus APIs and convert existing code to the new interfaces. * FSNameSystem#verifyParentDir now passes true for resolveLink, tests are in the base class so will be in an upcoming common change. * Replaced the numBlocks "> -2" check with "== -1" * When through whitespace changes, they should just remove trailing whitespaces now. * Updated HDFS-995 with your suggestions for checking #isDirectory() * I noticed INode#constructPath was using StringBuffer, changed it to use StringBuilder * FSNamesystem#metaSave no longers throws UnresolvedSymlinkException. It just use java.io.File for the local file so symlinks should be handled transparently by the local file system. * Also, for those following along, we should not have to worry about UnresolvedLinkExceptions in loadFSImage and friends because intermediate symlinks are resolved when objects are created, so the paths that get persisted are the fully resolved ones, ie creating /link/file will persist /dir/file so when we load the image we will call addToParent(.."/dir/file"..). This should be more clear when volume relative symlinks are resolved internally (HDFS-932). The code asserts on these cases. * getLinkTarget never throws an UnresolvedLinkException, it always resolves the first symlink in the path and ignores the remainder, so I left the throws clause as is Thanks, Eli > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Open (was: Patch Available) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-245: - Status: Patch Available (was: Open) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-245) Create symbolic links in HDFS
[ https://issues.apache.org/jira/browse/HDFS-245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia updated HDFS-245: -- Resolution: Fixed Fix Version/s: 0.22.0 Release Note: HDFS-245. Adds a symlink implementation to HDFS. This complements the new symlink feature added in HADOOP-6421 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) > Create symbolic links in HDFS > - > > Key: HDFS-245 > URL: https://issues.apache.org/jira/browse/HDFS-245 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: dhruba borthakur >Assignee: Eli Collins > Fix For: 0.22.0 > > Attachments: 4044_20081030spi.java, design-doc-v4.txt, > designdocv1.txt, designdocv2.txt, designdocv3.txt, > HADOOP-4044-strawman.patch, symlink-0.20.0.patch, symlink-25-hdfs.patch, > symlink-26-hdfs.patch, symlink-26-hdfs.patch, symLink1.patch, symLink1.patch, > symLink11.patch, symLink12.patch, symLink13.patch, symLink14.patch, > symLink15.txt, symLink15.txt, symlink16-common.patch, symlink16-hdfs.patch, > symlink16-mr.patch, symlink17-common.txt, symlink17-hdfs.txt, > symlink18-common.txt, symlink19-common-delta.patch, symlink19-common.txt, > symlink19-common.txt, symlink19-hdfs-delta.patch, symlink19-hdfs.txt, > symlink20-common.patch, symlink20-hdfs.patch, symlink21-common.patch, > symlink21-hdfs.patch, symlink22-common.patch, symlink22-hdfs.patch, > symlink23-common.patch, symlink23-hdfs.patch, symlink24-hdfs.patch, > symlink27-hdfs.patch, symlink28-hdfs.patch, symlink29-hdfs.patch, > symlink29-hdfs.patch, symlink30-hdfs.patch, symlink31-hdfs.patch, > symlink33-hdfs.patch, symlink35-hdfs.patch, symlink36-hdfs.patch, > symlink37-hdfs.patch, symlink38-hdfs.patch, symlink39-hdfs.patch, > symLink4.patch, symlink40-hdfs.patch, symlink41-hdfs.patch, symLink5.patch, > symLink6.patch, symLink8.patch, symLink9.patch > > > HDFS should support symbolic links. A symbolic link is a special type of file > that contains a reference to another file or directory in the form of an > absolute or relative path and that affects pathname resolution. Programs > which read or write to files named by a symbolic link will behave as if > operating directly on the target file. However, archiving utilities can > handle symbolic links specially and manipulate them directly. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.