Oleg Khaschansky created HADOOP-13905: -----------------------------------------
Summary: Cannot run wordcount example when there's a mounttable configured with a link to s3a. Key: HADOOP-13905 URL: https://issues.apache.org/jira/browse/HADOOP-13905 Project: Hadoop Common Issue Type: Bug Components: fs/s3 Affects Versions: 2.8.0 Reporter: Oleg Khaschansky Have 3 node setup: nn/slave/client. Client default fs is viewfs with the following mounttable: <configuration> <property> <name>fs.viewfs.mounttable.hadoopDemo.homedir</name> <value>/home</value> </property> <property> <name>fs.viewfs.mounttable.hadoopDemo.link./home</name> <value>hdfs://namenode:9000</value> </property> <property> <name>fs.viewfs.mounttable.hadoopDemo.link./tmp</name> <value>hdfs://namenode:9000/tmp</value> </property> <property> <name>fs.viewfs.mounttable.hadoopDemo.link./user</name> <value>hdfs://namenode:9000/user</value> </property> <property> <name>fs.viewfs.mounttable.hadoopDemo.link./s3a</name> <value>s3a://cloudply-hadoop-demo/</value> </property> </configuration> s3a credentials are configured in core-site.xml on the client node. Able to view/modify /s3a mount contents with hdfs commands. When I ran a wordcount example using this line (even without access to s3a share): hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/sources/hadoop-mapreduce-examples-2.8.0-SNAPSHOT-sources.jar org.apache.hadoop.examples.WordCount /home/input /home/output it fails with the following exception: 16/12/14 16:08:33 INFO client.RMProxy: Connecting to ResourceManager at namenode/172.18.0.2:8032 16/12/14 16:08:33 INFO mapreduce.Cluster: Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:136) at org.apache.hadoop.fs.AbstractFileSystem.createFileSystem(AbstractFileSystem.java:165) at org.apache.hadoop.fs.AbstractFileSystem.get(AbstractFileSystem.java:250) at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:342) at org.apache.hadoop.fs.FileContext$2.run(FileContext.java:339) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) at org.apache.hadoop.fs.FileContext.getAbstractFileSystem(FileContext.java:339) at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:456) at org.apache.hadoop.fs.FileContext.getFileContext(FileContext.java:482) at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:148) at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:132) at org.apache.hadoop.mapred.YARNRunner.<init>(YARNRunner.java:122) at org.apache.hadoop.mapred.YarnClientProtocolProvider.create(YarnClientProtocolProvider.java:34) at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:111) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:98) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:91) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1311) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) at org.apache.hadoop.mapreduce.Job.connect(Job.java:1307) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1335) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:234) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.fs.AbstractFileSystem.newInstance(AbstractFileSystem.java:134) ... 32 more Caused by: org.apache.hadoop.HadoopIllegalArgumentException: FileSystem implementation error - default port -1 is not valid at org.apache.hadoop.fs.AbstractFileSystem.getUri(AbstractFileSystem.java:306) at org.apache.hadoop.fs.AbstractFileSystem.<init>(AbstractFileSystem.java:266) at org.apache.hadoop.fs.viewfs.ChRootedFs.<init>(ChRootedFs.java:102) at org.apache.hadoop.fs.viewfs.ViewFs$1.getTargetFileSystem(ViewFs.java:220) at org.apache.hadoop.fs.viewfs.ViewFs$1.getTargetFileSystem(ViewFs.java:209) at org.apache.hadoop.fs.viewfs.InodeTree.createLink(InodeTree.java:261) at org.apache.hadoop.fs.viewfs.InodeTree.<init>(InodeTree.java:333) at org.apache.hadoop.fs.viewfs.ViewFs$1.<init>(ViewFs.java:209) at org.apache.hadoop.fs.viewfs.ViewFs.<init>(ViewFs.java:209) ... 37 more Exception in thread "main" java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses. at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:133) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:98) at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:91) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1311) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1711) at org.apache.hadoop.mapreduce.Job.connect(Job.java:1307) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1335) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1359) at org.apache.hadoop.examples.WordCount.main(WordCount.java:87) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.util.RunJar.run(RunJar.java:234) at org.apache.hadoop.util.RunJar.main(RunJar.java:148) When I remove s3a mount from mounttable - I am able to run wordcount example. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org