Hi, It's due to a very old version of ASM library that bring it by hadoop/yarn. Please add exclusion of asm library to all hadoop dependencies.
<exclusion> <groupId>asm</groupId> <artifactId>asm</artifactId> </exclusion> Terence On Fri, Jan 22, 2016 at 2:34 PM, Kristoffer Sjögren <[email protected]> wrote: > Further adding the following dependencies cause another exception. > > <dependency> > <groupId>com.google.guava</groupId> > <artifactId>guava</artifactId> > <version>13.0</version> > </dependency> > <dependency> > <groupId>org.apache.hadoop</groupId> > <artifactId>hadoop-hdfs</artifactId> > <version>2.7.1</version> > </dependency> > > Exception in thread " STARTING" > java.lang.IncompatibleClassChangeError: class > org.apache.twill.internal.utils.Dependencies$DependencyClassVisitor > has interface org.objectweb.asm.ClassVisitor as super class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:760) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) > at java.net.URLClassLoader.access$100(URLClassLoader.java:73) > at java.net.URLClassLoader$1.run(URLClassLoader.java:368) > at java.net.URLClassLoader$1.run(URLClassLoader.java:362) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:361) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > at > org.apache.twill.internal.utils.Dependencies.findClassDependencies(Dependencies.java:86) > at > org.apache.twill.internal.ApplicationBundler.findDependencies(ApplicationBundler.java:198) > at > org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:155) > at > org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:126) > at > org.apache.twill.yarn.YarnTwillPreparer.createAppMasterJar(YarnTwillPreparer.java:402) > at > org.apache.twill.yarn.YarnTwillPreparer.access$200(YarnTwillPreparer.java:108) > at > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:299) > at > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:289) > at > org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:97) > at > org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:76) > at > org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:175) > at > com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) > at java.lang.Thread.run(Thread.java:745) > > On Fri, Jan 22, 2016 at 11:32 PM, Kristoffer Sjögren <[email protected]> > wrote: > > Add those dependencies fail with the following exception. > > > > Exception in thread "main" java.lang.AbstractMethodError: > > > org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider.getProxy()Lorg/apache/hadoop/io/retry/FailoverProxyProvider$ProxyInfo; > > at > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:73) > > at > org.apache.hadoop.io.retry.RetryInvocationHandler.<init>(RetryInvocationHandler.java:64) > > at org.apache.hadoop.io.retry.RetryProxy.create(RetryProxy.java:59) > > at > org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:149) > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:569) > > at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:512) > > at > org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:142) > > at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) > > at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) > > at > org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) > > at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) > > at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:170) > > at > org.apache.twill.yarn.YarnTwillRunnerService.createDefaultLocationFactory(YarnTwillRunnerService.java:615) > > at > org.apache.twill.yarn.YarnTwillRunnerService.<init>(YarnTwillRunnerService.java:149) > > at deephacks.BundledJarExample.main(BundledJarExample.java:70) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:497) > > at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144) > > > > On Fri, Jan 22, 2016 at 10:58 PM, Terence Yim <[email protected]> wrote: > >> Hi, > >> > >> If you run it from IDE, you and simply add a dependency on hadoop with > >> version 2.7.1. E.g. if you are using Maven, you can add the following to > >> your pom.xml dependencies section. > >> > >> <dependency> > >> <groupId>org.apache.hadoop</groupId> > >> <artifactId>hadoop-yarn-api</artifactId> > >> <version>2.7.1</version> > >> </dependency> > >> <dependency> > >> <groupId>org.apache.hadoop</groupId> > >> <artifactId>hadoop-yarn-common</artifactId> > >> <version>2.7.1</version> > >> </dependency> > >> <dependency> > >> <groupId>org.apache.hadoop</groupId> > >> <artifactId>hadoop-yarn-client</artifactId> > >> <version>2.7.1</version> > >> </dependency> > >> <dependency> > >> <groupId>org.apache.hadoop</groupId> > >> <artifactId>hadoop-common</artifactId> > >> <version>2.7.1</version> > >> </dependency> > >> > >> Terence > >> > >> On Fri, Jan 22, 2016 at 12:47 PM, Kristoffer Sjögren <[email protected]> > >> wrote: > >> > >>> I run it from IDE right now, but would like to create a command line > >>> app eventually. > >>> > >>> I should clarify that the exception above is thrown on the YARN node, > >>> not in the IDE. > >>> > >>> On Fri, Jan 22, 2016 at 9:32 PM, Terence Yim <[email protected]> wrote: > >>> > Hi Kristoffer, > >>> > > >>> > The example itself shouldn't need any modification. However, how do > >>> > you run that class? Do you run it from IDE or from command line using > >>> > "java" command? > >>> > > >>> > Terence > >>> > > >>> > On Fri, Jan 22, 2016 at 12:16 PM, Kristoffer Sjögren < > [email protected]> > >>> wrote: > >>> >> Hi Terence, > >>> >> > >>> >> I'm quite new to Twill and not sure how to do that exactly. Could > you > >>> >> show me how to modify the following example to do the same? > >>> >> > >>> >> > >>> > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java > >>> >> > >>> >> On Fri, Jan 22, 2016 at 7:24 PM, Terence Yim <[email protected]> > wrote: > >>> >>> Hi Kristoffer, > >>> >>> > >>> >>> Seems like the exception comes from the YARN class > "ConverterUtils". I > >>> >>> believe need to start the application with the version 2.7.1 Hadoop > >>> >>> Jars. How to do start the twill application? Usually on a cluster > with > >>> >>> hadoop installed, you can get all the hadoop jars in the classpath > by > >>> >>> running this: > >>> >>> > >>> >>> export CP=`hadoop classpath` > >>> >>> java -cp .:$CP YourApp ... > >>> >>> > >>> >>> Assuming your app classes and Twill jars are in the current > directory. > >>> >>> > >>> >>> Terence > >>> >>> > >>> >>> On Fri, Jan 22, 2016 at 4:54 AM, Kristoffer Sjögren < > [email protected]> > >>> wrote: > >>> >>>> Here's the full stacktrace. > >>> >>>> > >>> >>>> Exception in thread "main" > java.lang.reflect.InvocationTargetException > >>> >>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >>> >>>> at > >>> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> >>>> at > >>> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> >>>> at java.lang.reflect.Method.invoke(Method.java:497) > >>> >>>> at > org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:89) > >>> >>>> Caused by: java.lang.RuntimeException: > >>> >>>> java.lang.reflect.InvocationTargetException > >>> >>>> at > com.google.common.base.Throwables.propagate(Throwables.java:160) > >>> >>>> at > >>> > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61) > >>> >>>> at > >>> > org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:77) > >>> >>>> ... 5 more > >>> >>>> Caused by: java.lang.reflect.InvocationTargetException > >>> >>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >>> Method) > >>> >>>> at > >>> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > >>> >>>> at > >>> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >>> >>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:422) > >>> >>>> at > >>> > org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58) > >>> >>>> ... 6 more > >>> >>>> Caused by: java.lang.IllegalArgumentException: Invalid > ContainerId: > >>> >>>> container_e25_1453466340022_0004_01_000001 > >>> >>>> at > >>> > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182) > >>> >>>> at > >>> > org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83) > >>> >>>> at > >>> > org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65) > >>> >>>> at > >>> > org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34) > >>> >>>> ... 11 more > >>> >>>> Caused by: java.lang.NumberFormatException: For input string: > "e25" > >>> >>>> at > >>> > java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) > >>> >>>> at java.lang.Long.parseLong(Long.java:589) > >>> >>>> at java.lang.Long.parseLong(Long.java:631) > >>> >>>> at > >>> > org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137) > >>> >>>> at > >>> > org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177) > >>> >>>> ... 14 more > >>> >>>> > >>> >>>> On Thu, Jan 21, 2016 at 10:59 PM, Kristoffer Sjögren < > >>> [email protected]> wrote: > >>> >>>>> Hi > >>> >>>>> > >>> >>>>> I'm trying the basic example [1] on yarn 2.7.1 but get an > exception > >>> as > >>> >>>>> soon as the application starts on the resource manager that > tells me > >>> >>>>> the container id cannot be parsed. > >>> >>>>> > >>> >>>>> java.lang.IllegalArgumentException: Invalid containerId: > >>> >>>>> container_e04_1427159778706_0002_01_000001 > >>> >>>>> > >>> >>>>> I don't have the exact stacktrace but I recall it failing in > >>> >>>>> ConverterUtils.toContainerId because it assumes that that the > first > >>> >>>>> token is an application attempt to be parsed as an integer. This > >>> class > >>> >>>>> resides in hadoop-yarn-common 2.3.0. > >>> >>>>> > >>> >>>>> Is there any way to either tweak the container id or make twill > use > >>> >>>>> the 2.7.1 jar instead? > >>> >>>>> > >>> >>>>> Cheers, > >>> >>>>> -Kristoffer > >>> >>>>> > >>> >>>>> > >>> >>>>> [1] > >>> > https://github.com/apache/incubator-twill/blob/master/twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java > >>> >
