[jira] [Commented] (HADOOP-9984) FileSystem#listStatus should resolve symlinks by default
[
https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773735#comment-13773735
]
Hadoop QA commented on HADOOP-9984:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604319/HADOOP-9984.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.fs.TestFsShellReturnCode
org.apache.hadoop.fs.TestGlobPaths
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3115//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3115//console
This message is automatically generated.
> FileSystem#listStatus should resolve symlinks by default
>
>
> Key: HADOOP-9984
> URL: https://issues.apache.org/jira/browse/HADOOP-9984
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.1.0-beta
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Blocker
> Attachments: HADOOP-9984.001.patch
>
>
> During the process of adding symlink support to FileSystem, we realized that
> many existing HDFS clients would be broken by listStatus and globStatus
> returning symlinks. One example is applications that assume that
> !FileStatus#isFile implies that the inode is a directory. As we discussed in
> HADOOP-9972 and HADOOP-9912, we should default these APIs to returning
> resolved paths.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9980) LightWeightGSet's modification field should be volatile so that detect changes by other thread in iteration.
[
https://issues.apache.org/jira/browse/HADOOP-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773698#comment-13773698
]
Hadoop QA commented on HADOOP-9980:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604369/HADOOP-9980.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3114//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3114//console
This message is automatically generated.
> LightWeightGSet's modification field should be volatile so that detect
> changes by other thread in iteration.
>
>
> Key: HADOOP-9980
> URL: https://issues.apache.org/jira/browse/HADOOP-9980
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
>Reporter: Junping Du
>Assignee: Junping Du
> Fix For: 2.3.0
>
> Attachments: HADOOP-9980.patch
>
>
> LightWeightGSet should have a volatile modification field (like:
> LightWeightHashSet or LightWeightLinkedSet) that is used to detect updates
> while iterating so they can throw a ConcurrentModificationException.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Status: Patch Available (was: Open) > FileSystem#listStatus should resolve symlinks by default > > > Key: HADOOP-9984 > URL: https://issues.apache.org/jira/browse/HADOOP-9984 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Blocker > Attachments: HADOOP-9984.001.patch > > > During the process of adding symlink support to FileSystem, we realized that > many existing HDFS clients would be broken by listStatus and globStatus > returning symlinks. One example is applications that assume that > !FileStatus#isFile implies that the inode is a directory. As we discussed in > HADOOP-9972 and HADOOP-9912, we should default these APIs to returning > resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9980) LightWeightGSet's modification field should be volatile so that detect changes by other thread in iteration.
[ https://issues.apache.org/jira/browse/HADOOP-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-9980: --- Description: LightWeightGSet should have a volatile modification field (like: LightWeightHashSet or LightWeightLinkedSet) that is used to detect updates while iterating so they can throw a ConcurrentModificationException. (was: LightWeightGSet should have a volatile modification field (like: LightWeightHashSet or LightWeight) that is used to detect updates while iterating so they can throw a ConcurrentModificationException.) > LightWeightGSet's modification field should be volatile so that detect > changes by other thread in iteration. > > > Key: HADOOP-9980 > URL: https://issues.apache.org/jira/browse/HADOOP-9980 > Project: Hadoop Common > Issue Type: Bug > Components: util >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.3.0 > > Attachments: HADOOP-9980.patch > > > LightWeightGSet should have a volatile modification field (like: > LightWeightHashSet or LightWeightLinkedSet) that is used to detect updates > while iterating so they can throw a ConcurrentModificationException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9980) LightWeightGSet's modification field should be volatile so that detect changes by other thread in iteration.
[ https://issues.apache.org/jira/browse/HADOOP-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-9980: --- Attachment: HADOOP-9980.patch Upload a quick fix. > LightWeightGSet's modification field should be volatile so that detect > changes by other thread in iteration. > > > Key: HADOOP-9980 > URL: https://issues.apache.org/jira/browse/HADOOP-9980 > Project: Hadoop Common > Issue Type: Bug > Components: util >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.3.0 > > Attachments: HADOOP-9980.patch > > > LightWeightGSet should have a volatile modification field (like: > LightWeightHashSet or LightWeight) that is used to detect updates while > iterating so they can throw a ConcurrentModificationException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9980) LightWeightGSet's modification field should be volatile so that detect changes by other thread in iteration.
[ https://issues.apache.org/jira/browse/HADOOP-9980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junping Du updated HADOOP-9980: --- Status: Patch Available (was: Open) > LightWeightGSet's modification field should be volatile so that detect > changes by other thread in iteration. > > > Key: HADOOP-9980 > URL: https://issues.apache.org/jira/browse/HADOOP-9980 > Project: Hadoop Common > Issue Type: Bug > Components: util >Reporter: Junping Du >Assignee: Junping Du > Fix For: 2.3.0 > > Attachments: HADOOP-9980.patch > > > LightWeightGSet should have a volatile modification field (like: > LightWeightHashSet or LightWeight) that is used to detect updates while > iterating so they can throw a ConcurrentModificationException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9977) Hadoop services won't start with different keypass and keystorepass when https is enabled
[
https://issues.apache.org/jira/browse/HADOOP-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773635#comment-13773635
]
Hadoop QA commented on HADOOP-9977:
---
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12604344/HADOOP-9977.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3113//testReport/
Console output:
https://builds.apache.org/job/PreCommit-HADOOP-Build/3113//console
This message is automatically generated.
> Hadoop services won't start with different keypass and keystorepass when
> https is enabled
> -
>
> Key: HADOOP-9977
> URL: https://issues.apache.org/jira/browse/HADOOP-9977
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Yesha Vora
>Assignee: Chris Nauroth
> Attachments: HADOOP-9977.1.patch
>
>
> Enable ssl in the configuration. While creating keystore, give different
> keypass and keystore password. (here, keypass = hadoop and
> storepass=hadoopKey)
> keytool -genkey -alias host1 -keyalg RSA -keysize 1024 -dname
> "CN=host1,OU=cm,O=cm,L=san jose,ST=ca,C=us" -keypass hadoop -keystore
> keystore.jks -storepass hadoopKey
> In , ssl-server.xml set below two properties.
> ssl.server.keystore.keypasswordhadoop
> ssl.server.keystore.passwordhadoopKey
> Namenode, ResourceManager, Datanode, Nodemanager, SecondaryNamenode fails to
> start with below error.
> 2013-09-17 21:39:00,794 FATAL namenode.NameNode (NameNode.java:main(1325)) -
> Exception in namenode join
> java.io.IOException: java.security.UnrecoverableKeyException: Cannot recover
> key
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:222)
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer$1.(NameNodeHttpServer.java:76)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:74)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:626)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:488)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:684)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:669)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1254)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
> Caused by: java.security.UnrecoverableKeyException: Cannot recover key
> at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)
> at
> sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:138)
> at
> sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:55)
> at java.security.KeyStore.getKey(KeyStore.java:792)
> at
> sun.security.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:131)
> at
> sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:68)
> at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:259)
> at
> org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:170)
> at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:121)
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:220)
> ... 9 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9977) Hadoop services won't start with different keypass and keystorepass when https is enabled
[ https://issues.apache.org/jira/browse/HADOOP-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HADOOP-9977: -- Target Version/s: 3.0.0, 2.1.1-beta Status: Patch Available (was: In Progress) > Hadoop services won't start with different keypass and keystorepass when > https is enabled > - > > Key: HADOOP-9977 > URL: https://issues.apache.org/jira/browse/HADOOP-9977 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.1.0-beta >Reporter: Yesha Vora >Assignee: Chris Nauroth > Attachments: HADOOP-9977.1.patch > > > Enable ssl in the configuration. While creating keystore, give different > keypass and keystore password. (here, keypass = hadoop and > storepass=hadoopKey) > keytool -genkey -alias host1 -keyalg RSA -keysize 1024 -dname > "CN=host1,OU=cm,O=cm,L=san jose,ST=ca,C=us" -keypass hadoop -keystore > keystore.jks -storepass hadoopKey > In , ssl-server.xml set below two properties. > ssl.server.keystore.keypasswordhadoop > ssl.server.keystore.passwordhadoopKey > Namenode, ResourceManager, Datanode, Nodemanager, SecondaryNamenode fails to > start with below error. > 2013-09-17 21:39:00,794 FATAL namenode.NameNode (NameNode.java:main(1325)) - > Exception in namenode join > java.io.IOException: java.security.UnrecoverableKeyException: Cannot recover > key > at org.apache.hadoop.http.HttpServer.(HttpServer.java:222) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:174) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer$1.(NameNodeHttpServer.java:76) > at > org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:74) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:626) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:488) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:684) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:669) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1254) > at > org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320) > Caused by: java.security.UnrecoverableKeyException: Cannot recover key > at sun.security.provider.KeyProtector.recover(KeyProtector.java:328) > at > sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:138) > at > sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:55) > at java.security.KeyStore.getKey(KeyStore.java:792) > at > sun.security.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:131) > at > sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:68) > at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:259) > at > org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:170) > at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:121) > at org.apache.hadoop.http.HttpServer.(HttpServer.java:220) > ... 9 more -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9977) Hadoop services won't start with different keypass and keystorepass when https is enabled
[
https://issues.apache.org/jira/browse/HADOOP-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HADOOP-9977:
--
Attachment: HADOOP-9977.1.patch
I'm attaching a patch. I've also done some refactoring in {{KeyStoreTestUtil}}
to enable writing tests for this.
> Hadoop services won't start with different keypass and keystorepass when
> https is enabled
> -
>
> Key: HADOOP-9977
> URL: https://issues.apache.org/jira/browse/HADOOP-9977
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 2.1.0-beta
>Reporter: Yesha Vora
>Assignee: Chris Nauroth
> Attachments: HADOOP-9977.1.patch
>
>
> Enable ssl in the configuration. While creating keystore, give different
> keypass and keystore password. (here, keypass = hadoop and
> storepass=hadoopKey)
> keytool -genkey -alias host1 -keyalg RSA -keysize 1024 -dname
> "CN=host1,OU=cm,O=cm,L=san jose,ST=ca,C=us" -keypass hadoop -keystore
> keystore.jks -storepass hadoopKey
> In , ssl-server.xml set below two properties.
> ssl.server.keystore.keypasswordhadoop
> ssl.server.keystore.passwordhadoopKey
> Namenode, ResourceManager, Datanode, Nodemanager, SecondaryNamenode fails to
> start with below error.
> 2013-09-17 21:39:00,794 FATAL namenode.NameNode (NameNode.java:main(1325)) -
> Exception in namenode join
> java.io.IOException: java.security.UnrecoverableKeyException: Cannot recover
> key
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:222)
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:174)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer$1.(NameNodeHttpServer.java:76)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:74)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:626)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:488)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:684)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.(NameNode.java:669)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1254)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1320)
> Caused by: java.security.UnrecoverableKeyException: Cannot recover key
> at sun.security.provider.KeyProtector.recover(KeyProtector.java:328)
> at
> sun.security.provider.JavaKeyStore.engineGetKey(JavaKeyStore.java:138)
> at
> sun.security.provider.JavaKeyStore$JKS.engineGetKey(JavaKeyStore.java:55)
> at java.security.KeyStore.getKey(KeyStore.java:792)
> at
> sun.security.ssl.SunX509KeyManagerImpl.(SunX509KeyManagerImpl.java:131)
> at
> sun.security.ssl.KeyManagerFactoryImpl$SunX509.engineInit(KeyManagerFactoryImpl.java:68)
> at javax.net.ssl.KeyManagerFactory.init(KeyManagerFactory.java:259)
> at
> org.apache.hadoop.security.ssl.FileBasedKeyStoresFactory.init(FileBasedKeyStoresFactory.java:170)
> at org.apache.hadoop.security.ssl.SSLFactory.init(SSLFactory.java:121)
> at org.apache.hadoop.http.HttpServer.(HttpServer.java:220)
> ... 9 more
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9972) new APIs for listStatus and globStatus to deal with symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773584#comment-13773584
]
Binglin Chang commented on HADOOP-9972:
---
bq. Hmm. We could have a convenience method called listLinkStatus which just
called into listStatus with the correct PathOptions. I sort of lean towards
fewer APIs rather than more, but maybe it makes sense.
I mean listStatus(Path, PathOption) should call into listLinkStatus(it is
HDFS::listStatus which is a primitive RPC call), not the other way around. I
wonder how can we implement listStatus(Path, PathOption) without the primitive
of listLinkStatus(Path)?
bq. Shell globbing doesn't ignore all errors
What I say of globbing is just shell wildcard substitution, it indeed ignore
all errors, glob just substitute a string with wildcard to some string.
http://www.linuxjournal.com/content/bash-extended-globbing
http://tldp.org/LDP/abs/html/globbingref.html
{code}
drwxr-xr-x 2 decster staff 68 Sep 19 17:09 aa
drwxr-xr-x 2 decster staff 68 Sep 19 17:12 bb
decster:~/projects/test> echo *
aa bb
decster:~/projects/test> echo */cc
*/cc
{code}
In your example:
{code}
cmccabe@keter:~/mydir> ls b/c
ls: cannot access b/c: Permission denied
# this error is thrown by ls, not globbing
cmccabe@keter:~/mydir> ls *
a:
c
ls: cannot open directory b: Permission denied
# "ls *" first become "ls a c"
# then ls throw the error when process c
{code}
> new APIs for listStatus and globStatus to deal with symlinks
>
>
> Key: HADOOP-9972
> URL: https://issues.apache.org/jira/browse/HADOOP-9972
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.1.1-beta
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> Based on the discussion in HADOOP-9912, we need new APIs for FileSystem to
> deal with symlinks. The issue is that code has been written which is
> incompatible with the existence of things which are not files or directories.
> For example,
> there is a lot of code out there that looks at FileStatus#isFile, and
> if it returns false, assumes that what it is looking at is a
> directory. In the case of a symlink, this assumption is incorrect.
> It seems reasonable to make the default behavior of {{FileSystem#listStatus}}
> and {{FileSystem#globStatus}} be fully resolving symlinks, and ignoring
> dangling ones. This will prevent incompatibility with existing MR jobs and
> other HDFS users. We should also add new versions of listStatus and
> globStatus that allow new, symlink-aware code to deal with symlinks as
> symlinks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9989) Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file.
[ https://issues.apache.org/jira/browse/HADOOP-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinghui Wang updated HADOOP-9989: - Remaining Estimate: 0h Original Estimate: 0h > Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary > file but set it to the configuration as JSON file. > > > Key: HADOOP-9989 > URL: https://issues.apache.org/jira/browse/HADOOP-9989 > Project: Hadoop Common > Issue Type: Bug > Components: security, util >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HADOOP-9989.patch > > Original Estimate: 0h > Remaining Estimate: 0h > > The code in JIRA HADOOP-9374's patch introduced a bug, where the value of the > tokenCacheFile parameter is being parsed as a binary file and set it to the > mapreduce.job.credentials.json parameter in GenericOptionsParser, which > cannot be parsed by JobSubmitter when it gets the value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9989) Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file.
[ https://issues.apache.org/jira/browse/HADOOP-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinghui Wang updated HADOOP-9989: - Attachment: HADOOP-9989.patch > Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary > file but set it to the configuration as JSON file. > > > Key: HADOOP-9989 > URL: https://issues.apache.org/jira/browse/HADOOP-9989 > Project: Hadoop Common > Issue Type: Bug > Components: security, util >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HADOOP-9989.patch > > > The code in JIRA HADOOP-9374's patch introduced a bug, where the value of the > tokenCacheFile parameter is being parsed as a binary file and set it to the > mapreduce.job.credentials.json parameter in GenericOptionsParser, which > cannot be parsed by JobSubmitter when it gets the value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9989) Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file.
[ https://issues.apache.org/jira/browse/HADOOP-9989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773565#comment-13773565 ] Jinghui Wang commented on HADOOP-9989: -- Based on how JobSubmitter process the value of the mapreduce.job.credentials.json key, the tokenCacheFile passed in with the command line should be a JSON file, which makes sense as the GenericOptionsParser reads the value of this file and set it to the mapreduce.job.credentials.json in the configuration originally. Attached patch is based on the assumption that the tokenCacheFile is a JSON file. > Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary > file but set it to the configuration as JSON file. > > > Key: HADOOP-9989 > URL: https://issues.apache.org/jira/browse/HADOOP-9989 > Project: Hadoop Common > Issue Type: Bug > Components: security, util >Affects Versions: 2.1.0-beta > Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 >Reporter: Jinghui Wang > Fix For: 3.0.0, 2.1.1-beta > > Attachments: HADOOP-9989.patch > > > The code in JIRA HADOOP-9374's patch introduced a bug, where the value of the > tokenCacheFile parameter is being parsed as a binary file and set it to the > mapreduce.job.credentials.json parameter in GenericOptionsParser, which > cannot be parsed by JobSubmitter when it gets the value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HADOOP-9984: Target Version/s: 2.1.1-beta (was: 2.1.0-beta) > FileSystem#listStatus should resolve symlinks by default > > > Key: HADOOP-9984 > URL: https://issues.apache.org/jira/browse/HADOOP-9984 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Blocker > Attachments: HADOOP-9984.001.patch > > > During the process of adding symlink support to FileSystem, we realized that > many existing HDFS clients would be broken by listStatus and globStatus > returning symlinks. One example is applications that assume that > !FileStatus#isFile implies that the inode is a directory. As we discussed in > HADOOP-9972 and HADOOP-9912, we should default these APIs to returning > resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9989) Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file.
Jinghui Wang created HADOOP-9989: Summary: Bug introduced in HADOOP-9374, which parses the -tokenCacheFile as binary file but set it to the configuration as JSON file. Key: HADOOP-9989 URL: https://issues.apache.org/jira/browse/HADOOP-9989 Project: Hadoop Common Issue Type: Bug Components: security, util Affects Versions: 2.1.0-beta Environment: Red Hat Enterprise 6 with Sun Java 1.7 and IBM Java 1.6 Reporter: Jinghui Wang Fix For: 3.0.0, 2.1.1-beta The code in JIRA HADOOP-9374's patch introduced a bug, where the value of the tokenCacheFile parameter is being parsed as a binary file and set it to the mapreduce.job.credentials.json parameter in GenericOptionsParser, which cannot be parsed by JobSubmitter when it gets the value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-7344) globStatus doesn't grok groupings with a slash
[
https://issues.apache.org/jira/browse/HADOOP-7344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe resolved HADOOP-7344.
--
Tags:
Resolution: Fixed
Fix Version/s: 2.3.0
Assignee: Colin Patrick McCabe
This was fixed by the globber rework in HADOOP-9817.
{code}
cmccabe@keter:~/hadoop4> /h/bin/hadoop fs -mkdir -p /a/b/c
cmccabe@keter:~/hadoop4> /h/bin/hadoop fs -ls '/{a,a/b}'
Found 1 items
drwxr-xr-x - cmccabe supergroup 0 2013-09-20 15:20 /a/b
Found 1 items
drwxr-xr-x - cmccabe supergroup 0 2013-09-20 15:20 /a/b/c
{code}
> globStatus doesn't grok groupings with a slash
> --
>
> Key: HADOOP-7344
> URL: https://issues.apache.org/jira/browse/HADOOP-7344
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 0.23.0
>Reporter: Daryn Sharp
>Assignee: Colin Patrick McCabe
> Fix For: 2.3.0
>
>
> If a glob contains a grouping with a single item that contains a slash, ex.
> "{a/b}", then globStatus throws {{"Illegal file pattern: Unclosed group near
> index 2"}} -- regardless of whether the path exists. However, if the glob
> set contains more than one item, ex. "{a/b,c}", then it throws a
> {{NullPointerException}} from {{FileSystem.java:1277}}.
> {code}
> 1276: FileStatus[] files = globStatusInternal(new Path(filePattern), filter);
> 1277: for (FileStatus file : files) {
> 1278: results.add(file);
> 1279: }
> {code}
> The method {{globStatusInternal}} can return null, so the iterator fails with
> the NPE.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9988) HDFS Compatible ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lohit Vijayarenu resolved HADOOP-9988. -- Resolution: Duplicate Release Note: creating JIRA has some issues, closing this dup of HADOOP-9985 > HDFS Compatible ViewFileSystem > -- > > Key: HADOOP-9988 > URL: https://issues.apache.org/jira/browse/HADOOP-9988 > Project: Hadoop Common > Issue Type: Bug >Reporter: Lohit Vijayarenu > Fix For: 2.0.6-alpha > > > There are multiple scripts and projects like pig, hive, elephantbird refer to > HDFS URI as hdfs://namenodehostport/ or hdfs:/// . In federated namespace > this causes problem because supported scheme for federation is viewfs:// . We > will have to force all users to change their scripts/programs to be able to > access federated cluster. > It would be great if thee was a way to map viewfs scheme to hdfs scheme > without exposing it to users. Opening this JIRA to get inputs from people who > have thought about this in their clusters. > In our clusters we ended up created another class > HDFSCompatibleViewFileSystem which hijacks both hdfs.fs.impl and > viewfs.fs.impl and passes down filesystem calls to ViewFileSystem. Is there > any suggested approach other than this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9987) HDFS Compatible ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773428#comment-13773428 ] Lohit Vijayarenu commented on HADOOP-9987: -- Can somebody close this, this was opened by mistake as part of retry of HADOOP-9985. I dont seem to have option to close this JIRA > HDFS Compatible ViewFileSystem > -- > > Key: HADOOP-9987 > URL: https://issues.apache.org/jira/browse/HADOOP-9987 > Project: Hadoop Common > Issue Type: Bug >Reporter: Lohit Vijayarenu > Fix For: 2.0.6-alpha > > > There are multiple scripts and projects like pig, hive, elephantbird refer to > HDFS URI as hdfs://namenodehostport/ or hdfs:/// . In federated namespace > this causes problem because supported scheme for federation is viewfs:// . We > will have to force all users to change their scripts/programs to be able to > access federated cluster. > It would be great if thee was a way to map viewfs scheme to hdfs scheme > without exposing it to users. Opening this JIRA to get inputs from people who > have thought about this in their clusters. > In our clusters we ended up created another class > HDFSCompatibleViewFileSystem which hijacks both hdfs.fs.impl and > viewfs.fs.impl and passes down filesystem calls to ViewFileSystem. Is there > any suggested approach other than this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9986) HDFS Compatible ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773426#comment-13773426 ] Lohit Vijayarenu commented on HADOOP-9986: -- Can somebody close this, this was opened by mistake as part of retry of HADOOP-9985 > HDFS Compatible ViewFileSystem > -- > > Key: HADOOP-9986 > URL: https://issues.apache.org/jira/browse/HADOOP-9986 > Project: Hadoop Common > Issue Type: Bug >Reporter: Lohit Vijayarenu > Fix For: 2.0.6-alpha > > > There are multiple scripts and projects like pig, hive, elephantbird refer to > HDFS URI as hdfs://namenodehostport/ or hdfs:/// . In federated namespace > this causes problem because supported scheme for federation is viewfs:// . We > will have to force all users to change their scripts/programs to be able to > access federated cluster. > It would be great if thee was a way to map viewfs scheme to hdfs scheme > without exposing it to users. Opening this JIRA to get inputs from people who > have thought about this in their clusters. > In our clusters we ended up created another class > HDFSCompatibleViewFileSystem which hijacks both hdfs.fs.impl and > viewfs.fs.impl and passes down filesystem calls to ViewFileSystem. Is there > any suggested approach other than this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773484#comment-13773484 ] Colin Patrick McCabe commented on HADOOP-9912: -- By the way, as far as I know, HADOOP-9984 is the only one of the JIRAs in this area that needs to get into branch-2.1-beta. HADOOP-9981 is not in 2.1-beta > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Summary: FileSystem#listStatus should resolve symlinks by default (was: FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default) > FileSystem#listStatus should resolve symlinks by default > > > Key: HADOOP-9984 > URL: https://issues.apache.org/jira/browse/HADOOP-9984 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Blocker > Attachments: HADOOP-9984.001.patch > > > During the process of adding symlink support to FileSystem, we realized that > many existing HDFS clients would be broken by listStatus and globStatus > returning symlinks. One example is applications that assume that > !FileStatus#isFile implies that the inode is a directory. As we discussed in > HADOOP-9972 and HADOOP-9912, we should default these APIs to returning > resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Attachment: HADOOP-9984.001.patch Posting a preliminary version. I expect the full jenkins run will identify a few tests that need to be updated here. > FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by > default > -- > > Key: HADOOP-9984 > URL: https://issues.apache.org/jira/browse/HADOOP-9984 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Blocker > Attachments: HADOOP-9984.001.patch > > > During the process of adding symlink support to FileSystem, we realized that > many existing HDFS clients would be broken by listStatus and globStatus > returning symlinks. One example is applications that assume that > !FileStatus#isFile implies that the inode is a directory. As we discussed in > HADOOP-9972 and HADOOP-9912, we should default these APIs to returning > resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9972) new APIs for listStatus and globStatus to deal with symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773471#comment-13773471
]
Colin Patrick McCabe commented on HADOOP-9972:
--
bq.Just to be clear, what happens if the error handler does not rethrow the
exception?
If the error handler doesn't rethrow the exception, the listStatus / globStatus
operation continues normally and returns the remaining results. (We can't
return the result that had the error.) Unresolved symlinks are one type of
error. Whether to handle {{UnresolvedLinkException}} differently than other
exceptions is up to the {{PathErrorHandler}} you provide.
bq. I'm not sure if the difference between "log exception and continue" vs.
"ignore it completely" is a different return code from the error handler method
or just whether the handler logs or not.
I was proposing that the logging happen inside the {{PathErrorHandler}}. From
the point of file of FileSystem / FileContext, all we care about is whether the
{{PathErrorHandler}} rethrows the exception or not. (We can provide a class
implementing PathErrorHandler that logs to FileSystem#LOG if that is a common
use case.)
bq. I suppose one could derive a new interface from PathFilter that becomes
PathOptions and listStatus(Path, PathFilter) could check internally if it's
actually got a PathOption instead of a PathFilter and behave differently.
However I think an explicit, separate API would be preferable though, simply
for clarity of what the API expects from callers.
Yeah, I was proposing adding a new type, {{PathOptions}}, which could contain
an instance of {{PathFilter}}. We could add new methods to {{PathFilter}}, but
since it's a public/stable interface rather than an abstract class, that would
be an incompatible change.
> new APIs for listStatus and globStatus to deal with symlinks
>
>
> Key: HADOOP-9972
> URL: https://issues.apache.org/jira/browse/HADOOP-9972
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.1.1-beta
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> Based on the discussion in HADOOP-9912, we need new APIs for FileSystem to
> deal with symlinks. The issue is that code has been written which is
> incompatible with the existence of things which are not files or directories.
> For example,
> there is a lot of code out there that looks at FileStatus#isFile, and
> if it returns false, assumes that what it is looking at is a
> directory. In the case of a symlink, this assumption is incorrect.
> It seems reasonable to make the default behavior of {{FileSystem#listStatus}}
> and {{FileSystem#globStatus}} be fully resolving symlinks, and ignoring
> dangling ones. This will prevent incompatibility with existing MR jobs and
> other HDFS users. We should also add new versions of listStatus and
> globStatus that allow new, symlink-aware code to deal with symlinks as
> symlinks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9972) new APIs for listStatus and globStatus to deal with symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773417#comment-13773417
]
Jason Lowe commented on HADOOP-9972:
+1 to the idea of having a new API where symlinks resolution and a per-entrhy
error handler can be specified. That should allow the client to cover all the
three scenarios based on how the handler reacts to errors. Just to be clear,
what happens if the error handler does not rethrow the exception? Is the entry
removed from the listStatus results, returned as a raw symlink, or ...? Is it
controllable by the error handler? I'm not sure if the difference between "log
exception and continue" vs. "ignore it completely" is a different return code
from the error handler method or just whether the handler logs or not.
bq. At first glance, I like extending the PathFilters.
That's a twist on the approach, not sure that's been proposed. I suppose one
could derive a new interface from PathFilter that becomes PathOptions and
listStatus(Path, PathFilter) could check internally if it's actually got a
PathOption instead of a PathFilter and behave differently. However I think an
explicit, separate API would be preferable though, simply for clarity of what
the API expects from callers.
> new APIs for listStatus and globStatus to deal with symlinks
>
>
> Key: HADOOP-9972
> URL: https://issues.apache.org/jira/browse/HADOOP-9972
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.1.1-beta
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> Based on the discussion in HADOOP-9912, we need new APIs for FileSystem to
> deal with symlinks. The issue is that code has been written which is
> incompatible with the existence of things which are not files or directories.
> For example,
> there is a lot of code out there that looks at FileStatus#isFile, and
> if it returns false, assumes that what it is looking at is a
> directory. In the case of a symlink, this assumption is incorrect.
> It seems reasonable to make the default behavior of {{FileSystem#listStatus}}
> and {{FileSystem#globStatus}} be fully resolving symlinks, and ignoring
> dangling ones. This will prevent incompatibility with existing MR jobs and
> other HDFS users. We should also add new versions of listStatus and
> globStatus that allow new, symlink-aware code to deal with symlinks as
> symlinks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9985) HDFS Compatible ViewFileSystem
[ https://issues.apache.org/jira/browse/HADOOP-9985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773413#comment-13773413 ] Omkar Vinit Joshi commented on HADOOP-9985: --- I am not sure if it is related but take a loot at YARN-1203 (http / https) where now we are forcing AM to tell up front which scheme is supported and Yarn will use it for communication or else it will default to cluster default. I think something similar can be done here as well. > HDFS Compatible ViewFileSystem > -- > > Key: HADOOP-9985 > URL: https://issues.apache.org/jira/browse/HADOOP-9985 > Project: Hadoop Common > Issue Type: Bug >Reporter: Lohit Vijayarenu > Fix For: 2.0.6-alpha > > > There are multiple scripts and projects like pig, hive, elephantbird refer to > HDFS URI as hdfs://namenodehostport/ or hdfs:/// . In federated namespace > this causes problem because supported scheme for federation is viewfs:// . We > will have to force all users to change their scripts/programs to be able to > access federated cluster. > It would be great if thee was a way to map viewfs scheme to hdfs scheme > without exposing it to users. Opening this JIRA to get inputs from people who > have thought about this in their clusters. > In our clusters we ended up created another class > HDFSCompatibleViewFileSystem which hijacks both hdfs.fs.impl and > viewfs.fs.impl and passes down filesystem calls to ViewFileSystem. Is there > any suggested approach other than this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9988) HDFS Compatible ViewFileSystem
Lohit Vijayarenu created HADOOP-9988: Summary: HDFS Compatible ViewFileSystem Key: HADOOP-9988 URL: https://issues.apache.org/jira/browse/HADOOP-9988 Project: Hadoop Common Issue Type: Bug Reporter: Lohit Vijayarenu Fix For: 2.0.6-alpha There are multiple scripts and projects like pig, hive, elephantbird refer to HDFS URI as hdfs://namenodehostport/ or hdfs:/// . In federated namespace this causes problem because supported scheme for federation is viewfs:// . We will have to force all users to change their scripts/programs to be able to access federated cluster. It would be great if thee was a way to map viewfs scheme to hdfs scheme without exposing it to users. Opening this JIRA to get inputs from people who have thought about this in their clusters. In our clusters we ended up created another class HDFSCompatibleViewFileSystem which hijacks both hdfs.fs.impl and viewfs.fs.impl and passes down filesystem calls to ViewFileSystem. Is there any suggested approach other than this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9981) Listing in RawLocalFileSystem is inefficient
[ https://issues.apache.org/jira/browse/HADOOP-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773374#comment-13773374 ] Colin Patrick McCabe commented on HADOOP-9981: -- yes, I think the globber can be fixed to avoid listStatus on non-wildcard path components. It will be a little tricky to do this while not breaking support for symlinks, but I think it can be done. > Listing in RawLocalFileSystem is inefficient > > > Key: HADOOP-9981 > URL: https://issues.apache.org/jira/browse/HADOOP-9981 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Kihwal Lee >Priority: Critical > > After HADOOP-9652, listStatus() or globStatus() calls against a local file > system directory is very slow. A user was loading data from local file > system to Hive and it took about 30 seconds. The same operation took less > than a second pre-HADOOP-9652. > The input path had many other files beside the input files and strace showed > that fork & exec of stat against each and every one of them. jstack confirmed > that this was being done from getNativeFileLinkStatus(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773302#comment-13773302 ] Arun C Murthy commented on HADOOP-9912: --- Thanks [~cmccabe]. How about breaking HADOOP-9972 into two independent pieces: # Fix globStatus/listStatus to resolve and throw an exception when it can't. # Add new apis being discussed in HADOOP-9972. This way #1 can be expedited and we can unblock hadoop-2.2 (GA). #2 can come in hadoop-2.3. In hadoop-2.2 we can put appropriate notices that symlinks aren't yet ready for primetime. Thoughts? > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9985) HDFS Compatible ViewFileSystem
Lohit Vijayarenu created HADOOP-9985: Summary: HDFS Compatible ViewFileSystem Key: HADOOP-9985 URL: https://issues.apache.org/jira/browse/HADOOP-9985 Project: Hadoop Common Issue Type: Bug Reporter: Lohit Vijayarenu Fix For: 2.0.6-alpha There are multiple scripts and projects like pig, hive, elephantbird refer to HDFS URI as hdfs://namenodehostport/ or hdfs:/// . In federated namespace this causes problem because supported scheme for federation is viewfs:// . We will have to force all users to change their scripts/programs to be able to access federated cluster. It would be great if thee was a way to map viewfs scheme to hdfs scheme without exposing it to users. Opening this JIRA to get inputs from people who have thought about this in their clusters. In our clusters we ended up created another class HDFSCompatibleViewFileSystem which hijacks both hdfs.fs.impl and viewfs.fs.impl and passes down filesystem calls to ViewFileSystem. Is there any suggested approach other than this? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9852) UGI login user keytab and principal should not be static
[ https://issues.apache.org/jira/browse/HADOOP-9852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773397#comment-13773397 ] Daryn Sharp commented on HADOOP-9852: - Could someone please review? I've been focusing a lot lately on increasing performance, and almost fully desynch-ing the UGI requires this first basic change. > UGI login user keytab and principal should not be static > > > Key: HADOOP-9852 > URL: https://issues.apache.org/jira/browse/HADOOP-9852 > Project: Hadoop Common > Issue Type: Sub-task > Components: security >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HADOOP-9852.patch > > > The static keytab and principal for the login user is problematic. The login > conf explicitly references these statics. As a result, > loginUserFromKeytabAndReturnUGI is unnecessarily synch'ed on the class to > swap out the login user's keytab and principal, login, then restore the > keytab/principal. This method's synch blocks further de-synching of other > methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
[ https://issues.apache.org/jira/browse/HADOOP-9984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HADOOP-9984: - Component/s: fs Priority: Blocker (was: Major) Target Version/s: 2.1.0-beta > FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by > default > -- > > Key: HADOOP-9984 > URL: https://issues.apache.org/jira/browse/HADOOP-9984 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Blocker > > During the process of adding symlink support to FileSystem, we realized that > many existing HDFS clients would be broken by listStatus and globStatus > returning symlinks. One example is applications that assume that > !FileStatus#isFile implies that the inode is a directory. As we discussed in > HADOOP-9972 and HADOOP-9912, we should default these APIs to returning > resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773395#comment-13773395 ] Colin Patrick McCabe commented on HADOOP-9912: -- Sure. We can change the default behavior in a separate JIRA. I filed HADOOP-9984 to change the default behavior of FileSystem#globStatus and FileSystem#listStatus to be resolving symlinks. bq. Plus multiple exceptions are now erroneously being reported as FNF, etc. Ex. I know of at least AccessControlException (I think Colin fixed?) and StandbyException but I think we encountered more. This was already fixed in HADOOP-9929. bq. The new implementation is causing increased RPC load by listing parent directories when no patterns are present in the component. We're discussing this (plus a fix) on HADOOP-9981. > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[
https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773377#comment-13773377
]
Daryn Sharp commented on HADOOP-9912:
-
We also need to revert most of HADOOP-9817, or at least change the new
{{Globber}} class to behave identically (although resolving symlinks) to how it
did before being split out of FileSystem.
The new implementation is causing increased RPC load by listing parent
directories when no patterns are present in the component. We've had apps go
OOM, or run extremely slowly, because parent dirs had lots of items.
Plus multiple exceptions are now erroneously being reported as FNF, etc. Ex. I
know of at least AccessControlException (I think Colin fixed?) and
StandbyException but I think we encountered more.
> globStatus of a symlink to a directory does not report symlink as a directory
> -
>
> Key: HADOOP-9912
> URL: https://issues.apache.org/jira/browse/HADOOP-9912
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Affects Versions: 2.1.0-beta
>Reporter: Jason Lowe
>Priority: Blocker
> Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt,
> old-hdfs.txt, old-local.txt
>
>
> globStatus for a path that is a symlink to a directory used to report the
> resulting FileStatus as a directory but recently this has changed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9984) FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default
Colin Patrick McCabe created HADOOP-9984: Summary: FileSystem#globStatus and FileSystem#listStatus should resolve symlinks by default Key: HADOOP-9984 URL: https://issues.apache.org/jira/browse/HADOOP-9984 Project: Hadoop Common Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe During the process of adding symlink support to FileSystem, we realized that many existing HDFS clients would be broken by listStatus and globStatus returning symlinks. One example is applications that assume that !FileStatus#isFile implies that the inode is a directory. As we discussed in HADOOP-9972 and HADOOP-9912, we should default these APIs to returning resolved paths. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9981) Listing in RawLocalFileSystem is inefficient
[ https://issues.apache.org/jira/browse/HADOOP-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773248#comment-13773248 ] Colin Patrick McCabe commented on HADOOP-9981: -- I think a JNI implementation for JDK6 would not be too difficult to create. It's probably just a few lines of code. > Listing in RawLocalFileSystem is inefficient > > > Key: HADOOP-9981 > URL: https://issues.apache.org/jira/browse/HADOOP-9981 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Kihwal Lee >Priority: Critical > > After HADOOP-9652, listStatus() or globStatus() calls against a local file > system directory is very slow. A user was loading data from local file > system to Hive and it took about 30 seconds. The same operation took less > than a second pre-HADOOP-9652. > The input path had many other files beside the input files and strace showed > that fork & exec of stat against each and every one of them. jstack confirmed > that this was being done from getNativeFileLinkStatus(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773250#comment-13773250 ] Arun C Murthy commented on HADOOP-9912: --- IAC, any thoughts on how far we are from coming to a resolution? Everyone agrees with [~jlowe]'s proposal? How quickly can we resolve it? Or, should we revert HADOOP-9418 and move on? > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773246#comment-13773246 ] Arun C Murthy commented on HADOOP-9912: --- Excellent points [~jlowe], I here-by withdraw my crazy motion. :) > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9981) Listing in RawLocalFileSystem is inefficient
[ https://issues.apache.org/jira/browse/HADOOP-9981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773329#comment-13773329 ] Kihwal Lee commented on HADOOP-9981: I think the new Globber added in HADOOP-9817 causes listStaus() of parents when globStatus() is called as in the Hive use case. The large number of files were actually in the parent directory of the data directory. While fixing stat will make it go faster, the Globber needs to be fixed as well to avoid unnecessary stats. > Listing in RawLocalFileSystem is inefficient > > > Key: HADOOP-9981 > URL: https://issues.apache.org/jira/browse/HADOOP-9981 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.3.0 >Reporter: Kihwal Lee >Priority: Critical > > After HADOOP-9652, listStatus() or globStatus() calls against a local file > system directory is very slow. A user was loading data from local file > system to Hive and it took about 30 seconds. The same operation took less > than a second pre-HADOOP-9652. > The input path had many other files beside the input files and strace showed > that fork & exec of stat against each and every one of them. jstack confirmed > that this was being done from getNativeFileLinkStatus(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773310#comment-13773310 ] Suresh Srinivas commented on HADOOP-9912: - +1 for splitting this into two parts. As high priority, lets fix the globStatus/listStatus as proposed. Newer and cleaner APIs should be done in a separate jira. This is lower priority in my opinion and should be done separately and is not a blocker for 2.X GA. > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773271#comment-13773271 ] Colin Patrick McCabe commented on HADOOP-9912: -- +1 for jason's proposal. I'd like to do this as part of HADOOP-9972... would appreciate feedback there > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Gao updated HADOOP-9969: --- Attachment: JobTracker.log > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta >Reporter: Yu Gao > Attachments: HADOOP-9969.patch, JobTracker.log > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9969) TGT expiration doesn't trigger Kerberos relogin
[ https://issues.apache.org/jira/browse/HADOOP-9969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773289#comment-13773289 ] Yu Gao commented on HADOOP-9969: Sure. Attaching a relevant sub-section of JobTracker log throwing the expiration exception. I'm using Hadoop 2.1.0-beta + MapReduce 1.1.1, with IBM JDK 6. > TGT expiration doesn't trigger Kerberos relogin > --- > > Key: HADOOP-9969 > URL: https://issues.apache.org/jira/browse/HADOOP-9969 > Project: Hadoop Common > Issue Type: Bug > Components: ipc, security >Affects Versions: 2.1.0-beta >Reporter: Yu Gao > Attachments: HADOOP-9969.patch > > > In HADOOP-9698 & HADOOP-9850, RPC client and Sasl client have been changed to > respect the auth method advertised from server, instead of blindly attempting > the configured one at client side. However, when TGT has expired, an > exception will be thrown from SaslRpcClient#createSaslClient(SaslAuth > authType), and at this time the authMethod still holds the initial value > which is SIMPLE and never has a chance to be updated with the expected one > requested by server, so kerberos relogin will not happen. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9972) new APIs for listStatus and globStatus to deal with symlinks
[
https://issues.apache.org/jira/browse/HADOOP-9972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773255#comment-13773255
]
Colin Patrick McCabe commented on HADOOP-9972:
--
Hmm. We could have a convenience method called {{listLinkStatus}} which just
called into {{listStatus}} with the correct {{PathOptions}}. I sort of lean
towards fewer APIs rather than more, but maybe it makes sense.
Shell globbing doesn't ignore all errors, btw.
{code}
cmccabe@keter:~/mydir> mkdir a
cmccabe@keter:~/mydir> mkdir b
cmccabe@keter:~/mydir> touch a/c
cmccabe@keter:~/mydir> touch b/c
cmccabe@keter:~/mydir> sudo chmod 000 b
root's password:
cmccabe@keter:~/mydir> ls */c
a/c
cmccabe@keter:~/mydir> ls b/c
ls: cannot access b/c: Permission denied
cmccabe@keter:~/mydir> ls
a b
cmccabe@keter:~/mydir> ls *
a:
c
ls: cannot open directory b: Permission denied
{code}
It's interesting that it ignores the error in the case of {{ls */c}}, but not
in the case of {{ls *}}. Hadoop's shell would abort (and give no results) in
both of those cases, which I think is suboptimal.
> new APIs for listStatus and globStatus to deal with symlinks
>
>
> Key: HADOOP-9972
> URL: https://issues.apache.org/jira/browse/HADOOP-9972
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.1.1-beta
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>
> Based on the discussion in HADOOP-9912, we need new APIs for FileSystem to
> deal with symlinks. The issue is that code has been written which is
> incompatible with the existence of things which are not files or directories.
> For example,
> there is a lot of code out there that looks at FileStatus#isFile, and
> if it returns false, assumes that what it is looking at is a
> directory. In the case of a symlink, this assumption is incorrect.
> It seems reasonable to make the default behavior of {{FileSystem#listStatus}}
> and {{FileSystem#globStatus}} be fully resolving symlinks, and ignoring
> dangling ones. This will prevent incompatibility with existing MR jobs and
> other HDFS users. We should also add new versions of listStatus and
> globStatus that allow new, symlink-aware code to deal with symlinks as
> symlinks.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9966) Refactor XDR code into XDRReader and XDRWriter
[ https://issues.apache.org/jira/browse/HADOOP-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773238#comment-13773238 ] Brandon Li commented on HADOOP-9966: The patch looks good. Nit: can we rename the class name to XdrReader and XdrWriter to be consistent with the current naming convention? > Refactor XDR code into XDRReader and XDRWriter > -- > > Key: HADOOP-9966 > URL: https://issues.apache.org/jira/browse/HADOOP-9966 > Project: Hadoop Common > Issue Type: Improvement > Components: nfs >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HADOOP-9966.000.patch, HADOOP-9966.001.patch, > HADOOP-9966.002.patch > > > Several methods in the current XDR class have ambiguous semantics. For > example, Size() returns the actual size of internal byte array. The actual > size of current buffer, is also affected by read requests, which pull data > out of the buffer. > These ambiguous semantics makes removing redundant copies on the NFS paths > difficult. > This JIRA proposes to decompose the responsibilities of XDR into two separate > class: XDRReader and XDRWriter. The overall designs should closely follow > Java's *Reader / *Writer classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9912) globStatus of a symlink to a directory does not report symlink as a directory
[ https://issues.apache.org/jira/browse/HADOOP-9912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773027#comment-13773027 ] Jason Lowe commented on HADOOP-9912: bq. Another crazy thought I'd like to throw out - what if we just returned false for isDir if we cannot resolve the symlink rather than throw an exception? This sounds equivalent to the earlier proposal where "bad" symlinks are returned as the raw symlink. isDir() and isFile() both return false for symlinks, and old clients are not aware of isFile() since it was added with symlink support. An old client of listStatus will interpret the link as a file since isDir() is false, but we don't know if that's the proper thing to do since we don't know the client's intent. If a directory walker is concerned about directories and not files at some point in the traverse, it could end up silently skipping a "bad" symlink when it should have failed. i.e.: symlink to directory in remote filesystem but filesystem is temporarily unavailable, symlink to directory in permission-protected tree, symlink intended to point to a directory but typo'd the target when link was created, etc. I'm not sure how common that case really is in practice. Our recent proposal is trying to err on the side of caution so we don't accidentally drop data when we should have failed. It does mean some scenarios for old clients will fail when they should have succeeded despite "bad" symlinks, but it seems better to report a failure that can be corrected (i.e.: fix the "bad" symlink and re-run the app) than to potentially skip desired inputs. > globStatus of a symlink to a directory does not report symlink as a directory > - > > Key: HADOOP-9912 > URL: https://issues.apache.org/jira/browse/HADOOP-9912 > Project: Hadoop Common > Issue Type: Bug > Components: fs >Affects Versions: 2.1.0-beta >Reporter: Jason Lowe >Priority: Blocker > Attachments: HADOOP-9912-testcase.patch, new-hdfs.txt, new-local.txt, > old-hdfs.txt, old-local.txt > > > globStatus for a path that is a symlink to a directory used to report the > resulting FileStatus as a directory but recently this has changed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9963) container-executor.c: error: expected expression
[ https://issues.apache.org/jira/browse/HADOOP-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-9963. Resolution: Duplicate Fix Version/s: (was: 2.1.1-beta) Release Note: (was: patch https://issues.apache.org/jira/browse/HADOOP-9648 thx) > container-executor.c: error: expected expression > > > Key: HADOOP-9963 > URL: https://issues.apache.org/jira/browse/HADOOP-9963 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.1.0-beta > Environment: OS X 10.8.2 (12C3103) >Reporter: xiaopo >Priority: Critical > Original Estimate: 504h > Remaining Estimate: 504h > > while I install hadoop-yarn/hadoop-yarn-server on my os x,the error > ocurred,err msg : > [exec] > /Users/po/git/hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1201:7: > error: expected expression > [exec] stpncpy(buf, ep->d_name, strlen(ep->d_name)); > [exec] ^ > [exec] /usr/include/secure/_string.h:110:41: note: expanded from macro > 'stpncpy' > [exec] ((__darwin_obsz0 (dest) != (size_t -1)) > \ > [exec] ^ > [exec] fatal error: too many errors emitted, stopping now > [-ferror-limit=] > [exec] 2 warnings and 20 errors generated. > [exec] make[2]: *** > [CMakeFiles/container.dir/main/native/container-executor/impl/container-executor.c.o] > Error 1 > [exec] make[1]: *** [CMakeFiles/container.dir/all] Error 2 > [exec] make: *** [all] Error 2 > help~~~ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HADOOP-9963) container-executor.c: error: expected expression
[ https://issues.apache.org/jira/browse/HADOOP-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran reopened HADOOP-9963: > container-executor.c: error: expected expression > > > Key: HADOOP-9963 > URL: https://issues.apache.org/jira/browse/HADOOP-9963 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.1.0-beta > Environment: OS X 10.8.2 (12C3103) >Reporter: xiaopo >Priority: Critical > Fix For: 2.1.1-beta > > Original Estimate: 504h > Remaining Estimate: 504h > > while I install hadoop-yarn/hadoop-yarn-server on my os x,the error > ocurred,err msg : > [exec] > /Users/po/git/hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1201:7: > error: expected expression > [exec] stpncpy(buf, ep->d_name, strlen(ep->d_name)); > [exec] ^ > [exec] /usr/include/secure/_string.h:110:41: note: expanded from macro > 'stpncpy' > [exec] ((__darwin_obsz0 (dest) != (size_t -1)) > \ > [exec] ^ > [exec] fatal error: too many errors emitted, stopping now > [-ferror-limit=] > [exec] 2 warnings and 20 errors generated. > [exec] make[2]: *** > [CMakeFiles/container.dir/main/native/container-executor/impl/container-executor.c.o] > Error 1 > [exec] make[1]: *** [CMakeFiles/container.dir/all] Error 2 > [exec] make: *** [all] Error 2 > help~~~ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9983) SocketInputStream class (org.apache.hadoop.net.SocketInputStream) is not public class in 2.0.5-alpha
[ https://issues.apache.org/jira/browse/HADOOP-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran resolved HADOOP-9983. Resolution: Duplicate Fix Version/s: 2.1.0-beta > SocketInputStream class (org.apache.hadoop.net.SocketInputStream) is not > public class in 2.0.5-alpha > > > Key: HADOOP-9983 > URL: https://issues.apache.org/jira/browse/HADOOP-9983 > Project: Hadoop Common > Issue Type: Bug > Environment: OS - 3.8.0-30-generic > JDK 6 > HADOOP 2.0.5 > Accumulo -1.4.2 >Reporter: Puneet Srivastava >Priority: Blocker > Labels: hadoop-2.0.5-aplha > Fix For: 2.1.0-beta > > Original Estimate: 24h > Remaining Estimate: 24h > > When I am running Accumulo it is running without any roadblock but It is not > allowing me to see tablet server and I am getting this stack trace > [monitor.Monitor] ERROR: Thread "ZooKeeperStatus" died tried to access class > org.apache.hadoop.net.SocketInputStream from class > org.apache.accumulo.core.util.TTimeoutTransport > java.lang.IllegalAccessError: tried to access class > org.apache.hadoop.net.SocketInputStream from class > org.apache.accumulo.core.util.TTimeoutTransport > at > org.apache.accumulo.core.util.TTimeoutTransport.create(TTimeoutTransport.java:40) > at > org.apache.accumulo.server.monitor.ZooKeeperStatus.run(ZooKeeperStatus.java:82) > at > org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) > at java.lang.Thread.run(Thread.java:662) > Reason I know that in Hadoop 2.0.5-alpha SocketInputStream class in not > public so How to reslove this -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-9983) SocketInputStream class (org.apache.hadoop.net.SocketInputStream) is not public class in 2.0.5-alpha
[ https://issues.apache.org/jira/browse/HADOOP-9983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13772898#comment-13772898 ] Gopal V commented on HADOOP-9983: - It seems to be public in the 2.1.1-beta branch - https://github.com/apache/hadoop-common/blob/branch-2.1.1-beta/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/SocketInputStream.java#L42 > SocketInputStream class (org.apache.hadoop.net.SocketInputStream) is not > public class in 2.0.5-alpha > > > Key: HADOOP-9983 > URL: https://issues.apache.org/jira/browse/HADOOP-9983 > Project: Hadoop Common > Issue Type: Bug > Environment: OS - 3.8.0-30-generic > JDK 6 > HADOOP 2.0.5 > Accumulo -1.4.2 >Reporter: Puneet Srivastava >Priority: Blocker > Labels: hadoop-2.0.5-aplha > Original Estimate: 24h > Remaining Estimate: 24h > > When I am running Accumulo it is running without any roadblock but It is not > allowing me to see tablet server and I am getting this stack trace > [monitor.Monitor] ERROR: Thread "ZooKeeperStatus" died tried to access class > org.apache.hadoop.net.SocketInputStream from class > org.apache.accumulo.core.util.TTimeoutTransport > java.lang.IllegalAccessError: tried to access class > org.apache.hadoop.net.SocketInputStream from class > org.apache.accumulo.core.util.TTimeoutTransport > at > org.apache.accumulo.core.util.TTimeoutTransport.create(TTimeoutTransport.java:40) > at > org.apache.accumulo.server.monitor.ZooKeeperStatus.run(ZooKeeperStatus.java:82) > at > org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) > at java.lang.Thread.run(Thread.java:662) > Reason I know that in Hadoop 2.0.5-alpha SocketInputStream class in not > public so How to reslove this -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9983) SocketInputStream class (org.apache.hadoop.net.SocketInputStream) is not public class in 2.0.5-alpha
Puneet Srivastava created HADOOP-9983: - Summary: SocketInputStream class (org.apache.hadoop.net.SocketInputStream) is not public class in 2.0.5-alpha Key: HADOOP-9983 URL: https://issues.apache.org/jira/browse/HADOOP-9983 Project: Hadoop Common Issue Type: Bug Environment: OS - 3.8.0-30-generic JDK 6 HADOOP 2.0.5 Accumulo -1.4.2 Reporter: Puneet Srivastava Priority: Blocker When I am running Accumulo it is running without any roadblock but It is not allowing me to see tablet server and I am getting this stack trace [monitor.Monitor] ERROR: Thread "ZooKeeperStatus" died tried to access class org.apache.hadoop.net.SocketInputStream from class org.apache.accumulo.core.util.TTimeoutTransport java.lang.IllegalAccessError: tried to access class org.apache.hadoop.net.SocketInputStream from class org.apache.accumulo.core.util.TTimeoutTransport at org.apache.accumulo.core.util.TTimeoutTransport.create(TTimeoutTransport.java:40) at org.apache.accumulo.server.monitor.ZooKeeperStatus.run(ZooKeeperStatus.java:82) at org.apache.accumulo.core.util.LoggingRunnable.run(LoggingRunnable.java:34) at java.lang.Thread.run(Thread.java:662) Reason I know that in Hadoop 2.0.5-alpha SocketInputStream class in not public so How to reslove this -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-9963) container-executor.c: error: expected expression
[ https://issues.apache.org/jira/browse/HADOOP-9963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaopo resolved HADOOP-9963. Resolution: Fixed Release Note: patch https://issues.apache.org/jira/browse/HADOOP-9648 thx > container-executor.c: error: expected expression > > > Key: HADOOP-9963 > URL: https://issues.apache.org/jira/browse/HADOOP-9963 > Project: Hadoop Common > Issue Type: Bug > Components: native >Affects Versions: 2.1.0-beta > Environment: OS X 10.8.2 (12C3103) >Reporter: xiaopo >Priority: Critical > Fix For: 2.1.1-beta > > Original Estimate: 504h > Remaining Estimate: 504h > > while I install hadoop-yarn/hadoop-yarn-server on my os x,the error > ocurred,err msg : > [exec] > /Users/po/git/hadoop-common/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c:1201:7: > error: expected expression > [exec] stpncpy(buf, ep->d_name, strlen(ep->d_name)); > [exec] ^ > [exec] /usr/include/secure/_string.h:110:41: note: expanded from macro > 'stpncpy' > [exec] ((__darwin_obsz0 (dest) != (size_t -1)) > \ > [exec] ^ > [exec] fatal error: too many errors emitted, stopping now > [-ferror-limit=] > [exec] 2 warnings and 20 errors generated. > [exec] make[2]: *** > [CMakeFiles/container.dir/main/native/container-executor/impl/container-executor.c.o] > Error 1 > [exec] make[1]: *** [CMakeFiles/container.dir/all] Error 2 > [exec] make: *** [all] Error 2 > help~~~ -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-9982) Fix broken hyperlinks in hadoop document
[ https://issues.apache.org/jira/browse/HADOOP-9982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HADOOP-9982: -- Description: For example, the hyperlink 'Single Node Setup' doesn't work correctly in ['Cluster Setup' document|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html]. I also found other dead links. I'll try to fix them. was: In ['Cluster Setup' document|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html], the hyperlink 'Single Node Setup' doesn't work correctly. It should link to SingleCluster.html, instead of ClusterSetup.html#SingleCluster. Assignee: Akira AJISAKA Summary: Fix broken hyperlinks in hadoop document (was: A hyperlink doesn't work correctly in 'Cluster Setup' document) > Fix broken hyperlinks in hadoop document > > > Key: HADOOP-9982 > URL: https://issues.apache.org/jira/browse/HADOOP-9982 > Project: Hadoop Common > Issue Type: Bug > Components: documentation >Affects Versions: 3.0.0, 2.1.0-beta, 2.0.6-alpha >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA >Priority: Minor > > For example, the hyperlink 'Single Node Setup' doesn't work correctly in > ['Cluster Setup' > document|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html]. > I also found other dead links. I'll try to fix them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-9982) A hyperlink doesn't work correctly in 'Cluster Setup' document
Akira AJISAKA created HADOOP-9982: - Summary: A hyperlink doesn't work correctly in 'Cluster Setup' document Key: HADOOP-9982 URL: https://issues.apache.org/jira/browse/HADOOP-9982 Project: Hadoop Common Issue Type: Bug Components: documentation Affects Versions: 2.0.6-alpha, 2.1.0-beta, 3.0.0 Reporter: Akira AJISAKA Priority: Minor In ['Cluster Setup' document|http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html], the hyperlink 'Single Node Setup' doesn't work correctly. It should link to SingleCluster.html, instead of ClusterSetup.html#SingleCluster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
