Twill uses Guava 13.0.1, which is likely incompatible with Guava 11.0.2. I also see that you have Guava 16.0.1 in your classpath, which is incompatible with Guava 13.0.1.
Could you try replacing the Guava 16.0.1 in your classpath with Guava 13.0.1? If that doesn't work, could you also try replacing Hadoop's Guava jar with 13.0.1? (make sure to backup your other Guava jars as I haven't actually tried this) Alvin On Mon, Aug 11, 2014 at 1:17 PM, Peterson, Michael < [email protected]> wrote: > I did a little more research on the run time error. > > When I run it with -verbose: > $ java -verbose -cp $CP org.apache.twill.example.yarn.HelloWorld > localhost:2181 > > I see that the problem is with the guava libraries - the Service$Listener > class in particular. > > [... lots of lines removed ...] > [Loaded com.google.common.util.concurrent.Service$Listener from > file:/home/yarn/twill2/guava-16.0.1.jar] > [Loaded java.lang.ClassFormatError from > /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > Exception in thread "main" [Loaded java.lang.Throwable$PrintStreamOrWriter > from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.lang.Throwable$WrappedPrintStream from > /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.util.IdentityHashMap from > /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.util.IdentityHashMap$KeySet from > /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > java.lang.IncompatibleClassChangeError: Implementing class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at > org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61) > [Loaded java.lang.Shutdown from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.lang.Shutdown$Lock from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > > > The version in my version of Hadoop is Guava 11.0.2: > > $ ls /usr/lib/hadoop/lib/guava* > /usr/lib/hadoop/lib/guava-11.0.2.jar > > > Has anyone hit this issue with Twill? And how should we resolve it? > > -Michael > > > -----Original Message----- > From: Peterson, Michael > Sent: Monday, August 11, 2014 11:16 AM > To: [email protected] > Subject: RE: Newbie: how to deploy/launch > > Hi Terence, > > Thanks for the detailed write up and explanation how to set up the > classpath to run a Twill app and around how Twill uses OSGi. > > Right now I'm trying to get the Hello example working without OSGi and > still having problems. > > Issue #1: > > > In the Twill example directory, we used the maven-shade-plugin to build > a fat jar that contains the HelloWorld class, as well as Twill classes and > its dependencies. > > When I build Twill on Windows (I have the latest code from > https://git-wip-us.apache.org/repos/asf/incubator-twill.git), it > generates a shaded jar in the yarn-example directory, but when I build it > on Linux, it does *not* create a shaded jar. > > But even the shaded jar on Windows doesn't have any of its dependencies in > it, so I don't think the pom is set up correctly? > > u070072@DUR-MPETER2-2W7 /d/eval/twill/incubator-twill (master) $ jar -tf > twill-examples/yarn/target/twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > META-INF/MANIFEST.MF > META-INF/ > META-INF/DEPENDENCIES > META-INF/LICENSE > META-INF/NOTICE > org/ > org/apache/ > org/apache/twill/ > org/apache/twill/example/ > org/apache/twill/example/yarn/ > org/apache/twill/example/yarn/BundledJarExample$1.class > org/apache/twill/example/yarn/BundledJarExample$ExampleBundledJarApp.class > org/apache/twill/example/yarn/BundledJarExample.class > org/apache/twill/example/yarn/HelloWorld$1.class > org/apache/twill/example/yarn/HelloWorld$HelloWorldRunnable.class > org/apache/twill/example/yarn/HelloWorld.class > META-INF/maven/ > META-INF/maven/org.apache.twill/ > META-INF/maven/org.apache.twill/twill-examples-yarn/ > META-INF/maven/org.apache.twill/twill-examples-yarn/pom.xml > META-INF/maven/org.apache.twill/twill-examples-yarn/pom.properties > > > So I had to copy all the binary jars to the Hadoop system and put them all > into a classpath env var, similar to how you suggested: > > $ echo $CP | tr ':' '\n' > guava-16.0.1.jar > twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar > original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar > twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar > slf4j-api-1.7.5.jar > twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar > twill-api-0.4.0-incubating-SNAPSHOT.jar > twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > twill-common-0.4.0-incubating-SNAPSHOT.jar > twill-ext-0.4.0-incubating-SNAPSHOT.jar > twill-core-0.4.0-incubating-SNAPSHOT.jar > twill-yarn-0.4.0-incubating-SNAPSHOT.jar > twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar > twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar > /etc/hadoop/conf > /usr/lib/hadoop/lib/* > /usr/lib/hadoop/.//* > /usr/lib/hadoop-hdfs/./ > /usr/lib/hadoop-hdfs/lib/* > /usr/lib/hadoop-hdfs/.//* > /usr/lib/hadoop-yarn/lib/* > /usr/lib/hadoop-yarn/.//* > /usr/lib/hadoop-mapreduce/lib/* > /usr/lib/hadoop-mapreduce/.//* > /usr/share/java/mysql-connector-java-5.1.17.jar > /usr/share/java/mysql-connector-java.jar > /usr/lib/hadoop-mapreduce/* > /usr/lib/tez/* > /usr/lib/tez/lib/* > /etc/tez/conf > > > > Issue #2: > > When I run the HelloWorld after setting up the classpath, I get an error: > > $ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181 > Exception in thread "main" java.lang.IncompatibleClassChangeError: > Implementing class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at > org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61) > > > Researching this it looks like a binary incompatibility so I recompiled > all the Twill jars on the Linux Hadoop machine that I'm running it on, > recopied all the jars to a single directory and ran it again, but I get the > same error. > > On Linux, I'm running: > $ java -version > java version "1.7.0_45" > Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) > 64-Bit Server VM (build 24.45-b08, mixed mode) > > On Windows, I'm compiling with: > $ java -version > java version "1.7.0_67" > Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) > 64-Bit Server VM (build 24.65-b04, mixed mode) > > > -Michael > > > -----Original Message----- > From: Terence Yim [mailto:[email protected]] > Sent: Saturday, August 09, 2014 2:33 AM > To: [email protected] > Subject: Re: Newbie: how to deploy/launch > > Hi Peterson, > > Welcome to Apache Twill !! > > The general principle to launch a Twill application in YARN is simply to > run a regular Java main class and launch the Twill application from there. > Twill takes care of finding the dependencies and building a "job jar" for > you. > > If you take a closer look at the HelloWorld example (source code in > http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a > HelloWorldRunnable, which basically is the class that will get executed in > a YARN container. The other piece is the main() method, which is a YARN > client and can run from any box that can talk to YARN RM (think of it as > where you submit a YARN job). > > To run the HelloWorld example, you can simply compile and run the class, > with both hadoop conf, hadoop jars and twill related jars in the classpath. > You can even run it from your IDE, by having the yarn configuration > available through the classpath or provided to the TwillRunnerService > directly. > > In the Twill example directory, we used the maven-shade-plugin to build a > fat jar that contains the HelloWorld class, as well as Twill classes and > its dependencies. So, to run it, you can simply do something like: > > CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP > org.apache.twill.example.yarn.HelloWorld > > You also mentioned OSGI bundle jar. In fact, that's another thing that > Twill supports. It allows you to run any Java application in twill, without > worrying about library versions conflict between your application and > twill/hadoop. You could simply write an application, build an OSGI bundle > jar (actually Twill only look for jar entries that are .class or .jar, > hence you can create a jar in that format using any tool you wanted), and > use the the bundle jar runner > (http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN > client and also has a generic TwillRunnable wrapper that will unjar the > bundle jar when running in a YARN container, create a ClassLoader using the > .class and .jar files from the bundle jar and launch the > main() method through that ClassLoader. > > If you face any issues while trying to run a Twill application, please > feel free to discuss in here :) > > Thanks, > Terence > > On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael < > [email protected]> wrote: > > Based on the source code for TwillLauncher, I tried to launch this on a > Hadoop cluster like so: > > > > $ java -cp > > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or > > iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example > > s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example > > s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH > > OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill- > > common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH > > OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub > > ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t > > will-zookeeper-0.4.0-incubating-SNAPSHOT.jar > > org.apache.twill.launcher.TwillLauncher > > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > > org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181 > > > > > > Or with the BundledJar: > > $ java -cp > > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or > > iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example > > s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example > > s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH > > OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill- > > common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH > > OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub > > ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t > > will-zookeeper-0.4.0-incubating-SNAPSHOT.jar > > org.apache.twill.launcher.TwillLauncher > > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > > org.apache.twill.example.yarn.BundledJarExample localhost:2181 > > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar . > > org.apache.twill.example.yarn.HelloWorld localhost:2181 > > > > But both fail as not being able to find the hadoop jars. I imagine you > have an easier way to launch this. Can we launch it via "hadoop jar" or > "yarn jar" instead? > > > > Which leads back to my questions about how to set up the classpath - a > fat (shaded) jar or some OSGi mechanism? > > > > -Michael > > > > -----Original Message----- > > From: Peterson, Michael [mailto:[email protected]] > > Sent: Friday, August 08, 2014 9:59 AM > > To: [email protected] > > Subject: Newbie: how to deploy/launch > > > > Hello, > > > > Just joined the list yesterday. I have read over the documentation I > could find and downloaded and built the code. I have a few questions: > > > > 1. How does one deploy and launch a Twill-based app? I am familiar > with Slider, so is it something like that, where you have to put your jars > into HDFS launch the jar with the YarnClient and point it to the jar you > put in HDFS? Any config or properties files required? > > > > 2. I see that the EchoMain example you provide uses OSGi, but it > looks like the other example "yarn" does not and uses a shaded jar instead. > Does that mean that running Twill with OSGi is optional or does it require > OSGi? > > > > > > BTW - the code builds on Window but at least one of the tests fails. > > > > Thank you, > > -Michael >
