Hi Terence, Thanks for the detailed write up and explanation how to set up the classpath to run a Twill app and around how Twill uses OSGi.
Right now I'm trying to get the Hello example working without OSGi and still having problems. Issue #1: > In the Twill example directory, we used the maven-shade-plugin to build a fat > jar that contains the HelloWorld class, as well as Twill classes and its > dependencies. When I build Twill on Windows (I have the latest code from https://git-wip-us.apache.org/repos/asf/incubator-twill.git), it generates a shaded jar in the yarn-example directory, but when I build it on Linux, it does *not* create a shaded jar. But even the shaded jar on Windows doesn't have any of its dependencies in it, so I don't think the pom is set up correctly? u070072@DUR-MPETER2-2W7 /d/eval/twill/incubator-twill (master) $ jar -tf twill-examples/yarn/target/twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar META-INF/MANIFEST.MF META-INF/ META-INF/DEPENDENCIES META-INF/LICENSE META-INF/NOTICE org/ org/apache/ org/apache/twill/ org/apache/twill/example/ org/apache/twill/example/yarn/ org/apache/twill/example/yarn/BundledJarExample$1.class org/apache/twill/example/yarn/BundledJarExample$ExampleBundledJarApp.class org/apache/twill/example/yarn/BundledJarExample.class org/apache/twill/example/yarn/HelloWorld$1.class org/apache/twill/example/yarn/HelloWorld$HelloWorldRunnable.class org/apache/twill/example/yarn/HelloWorld.class META-INF/maven/ META-INF/maven/org.apache.twill/ META-INF/maven/org.apache.twill/twill-examples-yarn/ META-INF/maven/org.apache.twill/twill-examples-yarn/pom.xml META-INF/maven/org.apache.twill/twill-examples-yarn/pom.properties So I had to copy all the binary jars to the Hadoop system and put them all into a classpath env var, similar to how you suggested: $ echo $CP | tr ':' '\n' guava-16.0.1.jar twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar slf4j-api-1.7.5.jar twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar twill-api-0.4.0-incubating-SNAPSHOT.jar twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar twill-common-0.4.0-incubating-SNAPSHOT.jar twill-ext-0.4.0-incubating-SNAPSHOT.jar twill-core-0.4.0-incubating-SNAPSHOT.jar twill-yarn-0.4.0-incubating-SNAPSHOT.jar twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar /etc/hadoop/conf /usr/lib/hadoop/lib/* /usr/lib/hadoop/.//* /usr/lib/hadoop-hdfs/./ /usr/lib/hadoop-hdfs/lib/* /usr/lib/hadoop-hdfs/.//* /usr/lib/hadoop-yarn/lib/* /usr/lib/hadoop-yarn/.//* /usr/lib/hadoop-mapreduce/lib/* /usr/lib/hadoop-mapreduce/.//* /usr/share/java/mysql-connector-java-5.1.17.jar /usr/share/java/mysql-connector-java.jar /usr/lib/hadoop-mapreduce/* /usr/lib/tez/* /usr/lib/tez/lib/* /etc/tez/conf Issue #2: When I run the HelloWorld after setting up the classpath, I get an error: $ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181 Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:800) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) at java.net.URLClassLoader.access$100(URLClassLoader.java:71) at java.net.URLClassLoader$1.run(URLClassLoader.java:361) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:425) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61) Researching this it looks like a binary incompatibility so I recompiled all the Twill jars on the Linux Hadoop machine that I'm running it on, recopied all the jars to a single directory and ran it again, but I get the same error. On Linux, I'm running: $ java -version java version "1.7.0_45" Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode) On Windows, I'm compiling with: $ java -version java version "1.7.0_67" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) -Michael -----Original Message----- From: Terence Yim [mailto:[email protected]] Sent: Saturday, August 09, 2014 2:33 AM To: [email protected] Subject: Re: Newbie: how to deploy/launch Hi Peterson, Welcome to Apache Twill !! The general principle to launch a Twill application in YARN is simply to run a regular Java main class and launch the Twill application from there. Twill takes care of finding the dependencies and building a "job jar" for you. If you take a closer look at the HelloWorld example (source code in http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a HelloWorldRunnable, which basically is the class that will get executed in a YARN container. The other piece is the main() method, which is a YARN client and can run from any box that can talk to YARN RM (think of it as where you submit a YARN job). To run the HelloWorld example, you can simply compile and run the class, with both hadoop conf, hadoop jars and twill related jars in the classpath. You can even run it from your IDE, by having the yarn configuration available through the classpath or provided to the TwillRunnerService directly. In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies. So, to run it, you can simply do something like: CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP org.apache.twill.example.yarn.HelloWorld You also mentioned OSGI bundle jar. In fact, that's another thing that Twill supports. It allows you to run any Java application in twill, without worrying about library versions conflict between your application and twill/hadoop. You could simply write an application, build an OSGI bundle jar (actually Twill only look for jar entries that are .class or .jar, hence you can create a jar in that format using any tool you wanted), and use the the bundle jar runner (http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN client and also has a generic TwillRunnable wrapper that will unjar the bundle jar when running in a YARN container, create a ClassLoader using the .class and .jar files from the bundle jar and launch the main() method through that ClassLoader. If you face any issues while trying to run a Twill application, please feel free to discuss in here :) Thanks, Terence On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael <[email protected]> wrote: > Based on the source code for TwillLauncher, I tried to launch this on a > Hadoop cluster like so: > > $ java -cp > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or > iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example > s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example > s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH > OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill- > common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH > OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub > ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t > will-zookeeper-0.4.0-incubating-SNAPSHOT.jar > org.apache.twill.launcher.TwillLauncher > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181 > > > Or with the BundledJar: > $ java -cp > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or > iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example > s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example > s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH > OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill- > common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH > OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub > ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t > will-zookeeper-0.4.0-incubating-SNAPSHOT.jar > org.apache.twill.launcher.TwillLauncher > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > org.apache.twill.example.yarn.BundledJarExample localhost:2181 > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar . > org.apache.twill.example.yarn.HelloWorld localhost:2181 > > But both fail as not being able to find the hadoop jars. I imagine you have > an easier way to launch this. Can we launch it via "hadoop jar" or "yarn > jar" instead? > > Which leads back to my questions about how to set up the classpath - a fat > (shaded) jar or some OSGi mechanism? > > -Michael > > -----Original Message----- > From: Peterson, Michael [mailto:[email protected]] > Sent: Friday, August 08, 2014 9:59 AM > To: [email protected] > Subject: Newbie: how to deploy/launch > > Hello, > > Just joined the list yesterday. I have read over the documentation I could > find and downloaded and built the code. I have a few questions: > > 1. How does one deploy and launch a Twill-based app? I am familiar with > Slider, so is it something like that, where you have to put your jars into > HDFS launch the jar with the YarnClient and point it to the jar you put in > HDFS? Any config or properties files required? > > 2. I see that the EchoMain example you provide uses OSGi, but it looks > like the other example "yarn" does not and uses a shaded jar instead. Does > that mean that running Twill with OSGi is optional or does it require OSGi? > > > BTW - the code builds on Window but at least one of the tests fails. > > Thank you, > -Michael
