Add those dependencies fail with the following exception. Exception in thread "main" java.lang.AbstractMethodError: org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy()Lorg/apache/hadoop/io/retry/FailoverProxyProvider$ProxyInfo; at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) at org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:149) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:569) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:512) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) at org.apache.twill.yarn.YarnTwillRunnerService.createDefaultLocationFactory(YarnTwillRunnerService.java:615) at org.apache.twill.yarn.YarnTwillRunnerService.<init>(YarnTwillRunnerService.java:149) at deephacks.BundledJarExample.main(BundledJarExample.java:70) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
On Fri, Jan 22, 2016 at 10:58 PM, Terence Yim <[email protected]> wrote: > Hi, > > If you run it from IDE, you and simply add a dependency on hadoop with > version 2.7.1. E.g. if you are using Maven, you can add the following to > your pom.xml dependencies section. > > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-yarn-api</artifactId> > <version>2.7.1</version> > </dependency> > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-yarn-common</artifactId> > <version>2.7.1</version> > </dependency> > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-yarn-client</artifactId> > <version>2.7.1</version> > </dependency> > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-common</artifactId> > <version>2.7.1</version> > </dependency> > > Terence > > On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren <[email protected]> > wrote: > >> I run it from IDE right now, but would like to create a command line >> app eventually. >> >> I should clarify that the exception above is thrown on the YARN node, >> not in the IDE. >> >> On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim <[email protected]> wrote: >> > Hi Kristoffer, >> > >> > The example itself shouldn't need any modification. However, how do >> > you run that class? Do you run it from IDE or from command line using >> > "java" command? >> > >> > Terence >> > >> > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren <[email protected]> >> wrote: >> >> Hi Terence, >> >> >> >> I'm quite new to Twill and not sure how to do that exactly. Could you >> >> show me how to modify the following example to do the same? >> >> >> >> >> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >> >> >> >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim <[email protected]> wrote: >> >>> Hi Kristoffer, >> >>> >> >>> Seems like the exception comes from the YARN class "ConverterUtils". I >> >>> believe need to start the application with the version 2.7.1 Hadoop >> >>> Jars. How to do start the twill application? Usually on a cluster with >> >>> hadoop installed, you can get all the hadoop jars in the classpath by >> >>> running this: >> >>> >> >>> export CP=`hadoop classpath` >> >>> java -cp .:$CP YourApp ... >> >>> >> >>> Assuming your app classes and Twill jars are in the current directory. >> >>> >> >>> Terence >> >>> >> >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren <[email protected]> >> wrote: >> >>>> Here's the full stacktrace. >> >>>> >> >>>> Exception in thread "main" java.lang.reflect.InvocationTargetException >> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >> >>>> at >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >> >>>> at >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >> >>>> at java.lang.reflect.Method.invoke(Method.java:497) >> >>>> at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) >> >>>> Caused by: java.lang.RuntimeException: >> >>>> java.lang.reflect.InvocationTargetException >> >>>> at com.google.common.base.Throwables.propagate(Throwables.java:160) >> >>>> at >> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) >> >>>> at >> org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) >> >>>> ... 5 more >> >>>> Caused by: java.lang.reflect.InvocationTargetException >> >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> Method) >> >>>> at >> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) >> >>>> at >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >> >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) >> >>>> at >> org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) >> >>>> ... 6 more >> >>>> Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: >> >>>> container_e25_1453466340022_0004_01_000001 >> >>>> at >> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) >> >>>> at >> org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) >> >>>> at >> org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) >> >>>> at >> org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) >> >>>> ... 11 more >> >>>> Caused by: java.lang.NumberFormatException: For input string: "e25" >> >>>> at >> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) >> >>>> at java.lang.Long.parseLong(Long.java:589) >> >>>> at java.lang.Long.parseLong(Long.java:631) >> >>>> at >> org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) >> >>>> at >> org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) >> >>>> ... 14 more >> >>>> >> >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < >> [email protected]> wrote: >> >>>>> Hi >> >>>>> >> >>>>> I'm trying the basic example [1] on yarn 2.7.1 but get an exception >> as >> >>>>> soon as the application starts on the resource manager that tells me >> >>>>> the container id cannot be parsed. >> >>>>> >> >>>>> java.lang.IllegalArgumentException: Invalid containerId: >> >>>>> container_e04_1427159778706_0002_01_000001 >> >>>>> >> >>>>> I don't have the exact stacktrace but I recall it failing in >> >>>>> ConverterUtils.toContainerId because it assumes that that the first >> >>>>> token is an application attempt to be parsed as an integer. This >> class >> >>>>> resides in hadoop-yarn-common 2.3.0. >> >>>>> >> >>>>> Is there any way to either tweak the container id or make twill use >> >>>>> the 2.7.1 jar instead? >> >>>>> >> >>>>> Cheers, >> >>>>> -Kristoffer >> >>>>> >> >>>>> >> >>>>> [1] >> https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java >>
