Hi Peterson, Welcome to Apache Twill !!
The general principle to launch a Twill application in YARN is simply to run a regular Java main class and launch the Twill application from there. Twill takes care of finding the dependencies and building a "job jar" for you. If you take a closer look at the HelloWorld example (source code in http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a HelloWorldRunnable, which basically is the class that will get executed in a YARN container. The other piece is the main() method, which is a YARN client and can run from any box that can talk to YARN RM (think of it as where you submit a YARN job). To run the HelloWorld example, you can simply compile and run the class, with both hadoop conf, hadoop jars and twill related jars in the classpath. You can even run it from your IDE, by having the yarn configuration available through the classpath or provided to the TwillRunnerService directly. In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies. So, to run it, you can simply do something like: CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP org.apache.twill.example.yarn.HelloWorld You also mentioned OSGI bundle jar. In fact, that's another thing that Twill supports. It allows you to run any Java application in twill, without worrying about library versions conflict between your application and twill/hadoop. You could simply write an application, build an OSGI bundle jar (actually Twill only look for jar entries that are .class or .jar, hence you can create a jar in that format using any tool you wanted), and use the the bundle jar runner (http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN client and also has a generic TwillRunnable wrapper that will unjar the bundle jar when running in a YARN container, create a ClassLoader using the .class and .jar files from the bundle jar and launch the main() method through that ClassLoader. If you face any issues while trying to run a Twill application, please feel free to discuss in here :) Thanks, Terence On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael <[email protected]> wrote: > Based on the source code for TwillLauncher, I tried to launch this on a > Hadoop cluster like so: > > $ java -cp > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSHOT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSHOT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar > org.apache.twill.launcher.TwillLauncher > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181 > > > Or with the BundledJar: > $ java -cp > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSHOT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSHOT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar > org.apache.twill.launcher.TwillLauncher > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar > org.apache.twill.example.yarn.BundledJarExample localhost:2181 > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar . > org.apache.twill.example.yarn.HelloWorld localhost:2181 > > But both fail as not being able to find the hadoop jars. I imagine you have > an easier way to launch this. Can we launch it via "hadoop jar" or "yarn > jar" instead? > > Which leads back to my questions about how to set up the classpath - a fat > (shaded) jar or some OSGi mechanism? > > -Michael > > -----Original Message----- > From: Peterson, Michael [mailto:[email protected]] > Sent: Friday, August 08, 2014 9:59 AM > To: [email protected] > Subject: Newbie: how to deploy/launch > > Hello, > > Just joined the list yesterday. I have read over the documentation I could > find and downloaded and built the code. I have a few questions: > > 1. How does one deploy and launch a Twill-based app? I am familiar with > Slider, so is it something like that, where you have to put your jars into > HDFS launch the jar with the YarnClient and point it to the jar you put in > HDFS? Any config or properties files required? > > 2. I see that the EchoMain example you provide uses OSGi, but it looks > like the other example "yarn" does not and uses a shaded jar instead. Does > that mean that running Twill with OSGi is optional or does it require OSGi? > > > BTW - the code builds on Window but at least one of the tests fails. > > Thank you, > -Michael
