Re: Symbolic link in Hadoop HDFS?
Those patches won't apply to an existing release so you'd need to use trunk to get symlinks. They will be included in the next release, however only via the FileContext API. Thanks, Eli On Tue, Mar 9, 2010 at 6:16 PM, jiang licht licht_ji...@yahoo.com wrote: Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as I understand, symlink is available by using some relevant patches, right? From what you mentioned, it seems to me that symlink support is quite stable as of now. Just curious, why not included in a recent hadoop release yet or will it be considered in the next release? Thanks, Michael --- On Tue, 3/9/10, Eli Collins e...@cloudera.com wrote: From: Eli Collins e...@cloudera.com Subject: Re: Symbolic link in Hadoop HDFS? To: common-user@hadoop.apache.org Date: Tuesday, March 9, 2010, 8:01 PM Hey Michael, Symbolic links has been implemented [1] but are not yet available in a Hadoop release. The implementation is only available to clients that use the new FileContext API so clients like Hive need to be migrated from using FileSystem to FileContext. This is currently being done in Hadoop itself [2]. Thanks, Eli [1] http://issues.apache.org/jira/browse/HDFS-245 [2] http://issues.apache.org/jira/browse/HADOOP-6446 On Tue, Mar 9, 2010 at 4:10 PM, jiang licht licht_ji...@yahoo.com wrote: Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks! Michael
Re: Symbolic link in Hadoop HDFS?
Thanks, Eli. I c. Any plan to support symlink in hadoop fs shell? That'll make life easier instead of everyone writing similar codes. With that support in shell commands, with one physical copy of a big data set, different ppl can easily create different subset of it to work on by using symlinks. Thanks, Michael --- On Wed, 3/10/10, Eli Collins e...@cloudera.com wrote: From: Eli Collins e...@cloudera.com Subject: Re: Symbolic link in Hadoop HDFS? To: common-user@hadoop.apache.org Date: Wednesday, March 10, 2010, 11:08 AM Those patches won't apply to an existing release so you'd need to use trunk to get symlinks. They will be included in the next release, however only via the FileContext API. Thanks, Eli On Tue, Mar 9, 2010 at 6:16 PM, jiang licht licht_ji...@yahoo.com wrote: Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as I understand, symlink is available by using some relevant patches, right? From what you mentioned, it seems to me that symlink support is quite stable as of now. Just curious, why not included in a recent hadoop release yet or will it be considered in the next release? Thanks, Michael --- On Tue, 3/9/10, Eli Collins e...@cloudera.com wrote: From: Eli Collins e...@cloudera.com Subject: Re: Symbolic link in Hadoop HDFS? To: common-user@hadoop.apache.org Date: Tuesday, March 9, 2010, 8:01 PM Hey Michael, Symbolic links has been implemented [1] but are not yet available in a Hadoop release. The implementation is only available to clients that use the new FileContext API so clients like Hive need to be migrated from using FileSystem to FileContext. This is currently being done in Hadoop itself [2]. Thanks, Eli [1] http://issues.apache.org/jira/browse/HDFS-245 [2] http://issues.apache.org/jira/browse/HADOOP-6446 On Tue, Mar 9, 2010 at 4:10 PM, jiang licht licht_ji...@yahoo.com wrote: Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks! Michael
Re: Symbolic link in Hadoop HDFS?
Yup, it needs to be ported to FileContext first (HADOOP-6424). Thanks, Eli On Wed, Mar 10, 2010 at 9:47 AM, jiang licht licht_ji...@yahoo.com wrote: Thanks, Eli. I c. Any plan to support symlink in hadoop fs shell? That'll make life easier instead of everyone writing similar codes. With that support in shell commands, with one physical copy of a big data set, different ppl can easily create different subset of it to work on by using symlinks. Thanks, Michael --- On Wed, 3/10/10, Eli Collins e...@cloudera.com wrote: From: Eli Collins e...@cloudera.com Subject: Re: Symbolic link in Hadoop HDFS? To: common-user@hadoop.apache.org Date: Wednesday, March 10, 2010, 11:08 AM Those patches won't apply to an existing release so you'd need to use trunk to get symlinks. They will be included in the next release, however only via the FileContext API. Thanks, Eli On Tue, Mar 9, 2010 at 6:16 PM, jiang licht licht_ji...@yahoo.com wrote: Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as I understand, symlink is available by using some relevant patches, right? From what you mentioned, it seems to me that symlink support is quite stable as of now. Just curious, why not included in a recent hadoop release yet or will it be considered in the next release? Thanks, Michael --- On Tue, 3/9/10, Eli Collins e...@cloudera.com wrote: From: Eli Collins e...@cloudera.com Subject: Re: Symbolic link in Hadoop HDFS? To: common-user@hadoop.apache.org Date: Tuesday, March 9, 2010, 8:01 PM Hey Michael, Symbolic links has been implemented [1] but are not yet available in a Hadoop release. The implementation is only available to clients that use the new FileContext API so clients like Hive need to be migrated from using FileSystem to FileContext. This is currently being done in Hadoop itself [2]. Thanks, Eli [1] http://issues.apache.org/jira/browse/HDFS-245 [2] http://issues.apache.org/jira/browse/HADOOP-6446 On Tue, Mar 9, 2010 at 4:10 PM, jiang licht licht_ji...@yahoo.com wrote: Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks! Michael
Symbolic link in Hadoop HDFS?
Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks! Michael
Re: Symbolic link in Hadoop HDFS?
Hey Michael, Symbolic links has been implemented [1] but are not yet available in a Hadoop release. The implementation is only available to clients that use the new FileContext API so clients like Hive need to be migrated from using FileSystem to FileContext. This is currently being done in Hadoop itself [2]. Thanks, Eli [1] http://issues.apache.org/jira/browse/HDFS-245 [2] http://issues.apache.org/jira/browse/HADOOP-6446 On Tue, Mar 9, 2010 at 4:10 PM, jiang licht licht_ji...@yahoo.com wrote: Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks! Michael
Re: Symbolic link in Hadoop HDFS?
Thanks, Eli. I actually just read hdfs-245 before I saw your reply. So, as I understand, symlink is available by using some relevant patches, right? From what you mentioned, it seems to me that symlink support is quite stable as of now. Just curious, why not included in a recent hadoop release yet or will it be considered in the next release? Thanks, Michael --- On Tue, 3/9/10, Eli Collins e...@cloudera.com wrote: From: Eli Collins e...@cloudera.com Subject: Re: Symbolic link in Hadoop HDFS? To: common-user@hadoop.apache.org Date: Tuesday, March 9, 2010, 8:01 PM Hey Michael, Symbolic links has been implemented [1] but are not yet available in a Hadoop release. The implementation is only available to clients that use the new FileContext API so clients like Hive need to be migrated from using FileSystem to FileContext. This is currently being done in Hadoop itself [2]. Thanks, Eli [1] http://issues.apache.org/jira/browse/HDFS-245 [2] http://issues.apache.org/jira/browse/HADOOP-6446 On Tue, Mar 9, 2010 at 4:10 PM, jiang licht licht_ji...@yahoo.com wrote: Is there a way to create symlink in hdfs? And does LOAD function in Pig follows such a link? Thanks! Michael
