Try to have guava 13 jar at the beginning if your classpath, that should solve the problem you mentioned.
Terence Sent from my iPhone > On Aug 11, 2014, at 1:17 PM, "Peterson, Michael" > <[email protected]> wrote: > > I did a little more research on the run time error. > > When I run it with -verbose: > $ java -verbose -cp $CP org.apache.twill.example.yarn.HelloWorld > localhost:2181 > > I see that the problem is with the guava libraries - the Service$Listener > class in particular. > > [... lots of lines removed ...] > [Loaded com.google.common.util.concurrent.Service$Listener from > file:/home/yarn/twill2/guava-16.0.1.jar] > [Loaded java.lang.ClassFormatError from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > Exception in thread "main" [Loaded java.lang.Throwable$PrintStreamOrWriter > from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.lang.Throwable$WrappedPrintStream from > /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.util.IdentityHashMap from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.util.IdentityHashMap$KeySet from > /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > java.lang.IncompatibleClassChangeError: Implementing class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61) > [Loaded java.lang.Shutdown from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > [Loaded java.lang.Shutdown$Lock from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar] > > > The version in my version of Hadoop is Guava 11.0.2: > > $ ls /usr/lib/hadoop/lib/guava* > /usr/lib/hadoop/lib/guava-11.0.2.jar > > > Has anyone hit this issue with Twill? And how should we resolve it? > > -Michael > > > -----Original Message----- > From: Peterson, Michael > Sent: Monday, August 11, 2014 11:16 AM > To: [email protected] > Subject: RE: Newbie: how to deploy/launch > > Hi Terence, > > Thanks for the detailed write up and explanation how to set up the classpath > to run a Twill app and around how Twill uses OSGi. > > Right now I'm trying to get the Hello example working without OSGi and still > having problems. > > Issue #1: > >> In the Twill example directory, we used the maven-shade-plugin to build a >> fat jar that contains the HelloWorld class, as well as Twill classes and its >> dependencies. > > When I build Twill on Windows (I have the latest code from > https://git-wip-us.apache.org/repos/asf/incubator-twill.git), it generates a > shaded jar in the yarn-example directory, but when I build it on Linux, it > does *not* create a shaded jar. > > But even the shaded jar on Windows doesn't have any of its dependencies in > it, so I don't think the pom is set up correctly? > > u070072@DUR-MPETER2-2W7 /d/eval/twill/incubator-twill (master) $ jar -tf > twill-examples/yarn/target/twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > META-INF/MANIFEST.MF > META-INF/ > META-INF/DEPENDENCIES > META-INF/LICENSE > META-INF/NOTICE > org/ > org/apache/ > org/apache/twill/ > org/apache/twill/example/ > org/apache/twill/example/yarn/ > org/apache/twill/example/yarn/BundledJarExample$1.class > org/apache/twill/example/yarn/BundledJarExample$ExampleBundledJarApp.class > org/apache/twill/example/yarn/BundledJarExample.class > org/apache/twill/example/yarn/HelloWorld$1.class > org/apache/twill/example/yarn/HelloWorld$HelloWorldRunnable.class > org/apache/twill/example/yarn/HelloWorld.class > META-INF/maven/ > META-INF/maven/org.apache.twill/ > META-INF/maven/org.apache.twill/twill-examples-yarn/ > META-INF/maven/org.apache.twill/twill-examples-yarn/pom.xml > META-INF/maven/org.apache.twill/twill-examples-yarn/pom.properties > > > So I had to copy all the binary jars to the Hadoop system and put them all > into a classpath env var, similar to how you suggested: > > $ echo $CP | tr ':' '\n' > guava-16.0.1.jar > twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar > original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar > twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar > slf4j-api-1.7.5.jar > twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar > twill-api-0.4.0-incubating-SNAPSHOT.jar > twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > twill-common-0.4.0-incubating-SNAPSHOT.jar > twill-ext-0.4.0-incubating-SNAPSHOT.jar > twill-core-0.4.0-incubating-SNAPSHOT.jar > twill-yarn-0.4.0-incubating-SNAPSHOT.jar > twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar > twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar > /etc/hadoop/conf > /usr/lib/hadoop/lib/* > /usr/lib/hadoop/.//* > /usr/lib/hadoop-hdfs/./ > /usr/lib/hadoop-hdfs/lib/* > /usr/lib/hadoop-hdfs/.//* > /usr/lib/hadoop-yarn/lib/* > /usr/lib/hadoop-yarn/.//* > /usr/lib/hadoop-mapreduce/lib/* > /usr/lib/hadoop-mapreduce/.//* > /usr/share/java/mysql-connector-java-5.1.17.jar > /usr/share/java/mysql-connector-java.jar > /usr/lib/hadoop-mapreduce/* > /usr/lib/tez/* > /usr/lib/tez/lib/* > /etc/tez/conf > > > > Issue #2: > > When I run the HelloWorld after setting up the classpath, I get an error: > > $ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181 > Exception in thread "main" java.lang.IncompatibleClassChangeError: > Implementing class > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at > java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61) > > > Researching this it looks like a binary incompatibility so I recompiled all > the Twill jars on the Linux Hadoop machine that I'm running it on, recopied > all the jars to a single directory and ran it again, but I get the same error. > > On Linux, I'm running: > $ java -version > java version "1.7.0_45" > Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit > Server VM (build 24.45-b08, mixed mode) > > On Windows, I'm compiling with: > $ java -version > java version "1.7.0_67" > Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit > Server VM (build 24.65-b04, mixed mode) > > > -Michael > > > -----Original Message----- > From: Terence Yim [mailto:[email protected]] > Sent: Saturday, August 09, 2014 2:33 AM > To: [email protected] > Subject: Re: Newbie: how to deploy/launch > > Hi Peterson, > > Welcome to Apache Twill !! > > The general principle to launch a Twill application in YARN is simply to run > a regular Java main class and launch the Twill application from there. Twill > takes care of finding the dependencies and building a "job jar" for you. > > If you take a closer look at the HelloWorld example (source code in > http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a > HelloWorldRunnable, which basically is the class that will get executed in a > YARN container. The other piece is the main() method, which is a YARN client > and can run from any box that can talk to YARN RM (think of it as where you > submit a YARN job). > > To run the HelloWorld example, you can simply compile and run the class, with > both hadoop conf, hadoop jars and twill related jars in the classpath. You > can even run it from your IDE, by having the yarn configuration available > through the classpath or provided to the TwillRunnerService directly. > > In the Twill example directory, we used the maven-shade-plugin to build a fat > jar that contains the HelloWorld class, as well as Twill classes and its > dependencies. So, to run it, you can simply do something like: > > CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP > org.apache.twill.example.yarn.HelloWorld > > You also mentioned OSGI bundle jar. In fact, that's another thing that Twill > supports. It allows you to run any Java application in twill, without > worrying about library versions conflict between your application and > twill/hadoop. You could simply write an application, build an OSGI bundle jar > (actually Twill only look for jar entries that are .class or .jar, hence you > can create a jar in that format using any tool you wanted), and use the the > bundle jar runner > (http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN client > and also has a generic TwillRunnable wrapper that will unjar the bundle jar > when running in a YARN container, create a ClassLoader using the .class and > .jar files from the bundle jar and launch the > main() method through that ClassLoader. > > If you face any issues while trying to run a Twill application, please feel > free to discuss in here :) > > Thanks, > Terence > >> On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael >> <[email protected]> wrote: >> Based on the source code for TwillLauncher, I tried to launch this on a >> Hadoop cluster like so: >> >> $ java -cp >> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or >> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example >> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example >> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH >> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill- >> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH >> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub >> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t >> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar >> org.apache.twill.launcher.TwillLauncher >> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar >> org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181 >> >> >> Or with the BundledJar: >> $ java -cp >> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or >> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example >> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example >> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH >> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill- >> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH >> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub >> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t >> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar >> org.apache.twill.launcher.TwillLauncher >> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar >> org.apache.twill.example.yarn.BundledJarExample localhost:2181 >> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar . >> org.apache.twill.example.yarn.HelloWorld localhost:2181 >> >> But both fail as not being able to find the hadoop jars. I imagine you have >> an easier way to launch this. Can we launch it via "hadoop jar" or "yarn >> jar" instead? >> >> Which leads back to my questions about how to set up the classpath - a fat >> (shaded) jar or some OSGi mechanism? >> >> -Michael >> >> -----Original Message----- >> From: Peterson, Michael [mailto:[email protected]] >> Sent: Friday, August 08, 2014 9:59 AM >> To: [email protected] >> Subject: Newbie: how to deploy/launch >> >> Hello, >> >> Just joined the list yesterday. I have read over the documentation I could >> find and downloaded and built the code. I have a few questions: >> >> 1. How does one deploy and launch a Twill-based app? I am familiar >> with Slider, so is it something like that, where you have to put your jars >> into HDFS launch the jar with the YarnClient and point it to the jar you put >> in HDFS? Any config or properties files required? >> >> 2. I see that the EchoMain example you provide uses OSGi, but it looks >> like the other example "yarn" does not and uses a shaded jar instead. Does >> that mean that running Twill with OSGi is optional or does it require OSGi? >> >> >> BTW - the code builds on Window but at least one of the tests fails. >> >> Thank you, >> -Michael
