Thanks Alvin!!
I tried the solution, you suggested but application is throwing exception :
14/04/06 14:57:10 INFO internal.ServiceMain: Service
org.apache.twill.internal.appmaster.ApplicationMasterService@3c9c31
started.
14/04/06 14:57:10 INFO controller.ControllerEpochListener:
[ControllerEpochListener on 1]: Initialized controller epoch to 1 and
zk version 0
14/04/06 14:57:10 INFO appmaster.ApplicationMasterService: Request 1
container with capability <memory:512, vCores:1>
14/04/06 14:57:12 INFO appmaster.ApplicationMasterService: Got
container container_1396815710754_0007_01_000002
14/04/06 14:57:12 INFO appmaster.ApplicationMasterService: Starting
runnable BundledJarRunnable with
RunnableProcessLauncher{container=org.apache.twill.internal.yarn.Hadoop21YarnContainerInfo@1f30035}
14/04/06 14:57:12 INFO appmaster.RunnableProcessLauncher: Launching in
container container_1396815710754_0007_01_000002 at
localhost/127.0.0.1, [java -Djava.io.tmpdir=tmp
-Dyarn.container=$YARN_CONTAINER_ID
-Dtwill.runnable=$TWILL_APP_NAME.$TWILL_RUNNABLE_NAME -cp
launcher.jar:$HADOOP_CONF_DIR -Xmx359m
org.apache.twill.launcher.TwillLauncher container.jar
org.apache.twill.internal.container.TwillContainerMain true
1><LOG_DIR>/stdout 2><LOG_DIR>/stderr]
14/04/06 14:57:12 INFO impl.ContainerManagementProtocolProxy: Opening
proxy : localhost:55102
14/04/06 14:57:12 INFO appmaster.ApplicationMasterService: Runnable
BundledJarRunnable fully provisioned with 1 instances.
14/04/06 14:57:18 INFO appmaster.ApplicationMasterService: Container
container_1396815710754_0007_01_000002 completed with
COMPLETE:Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:662)
.
14/04/06 14:57:18 WARN appmaster.RunningContainers: Container
container_1396815710754_0007_01_000002 exited abnormally with state
COMPLETE, exit code 10.
14/04/06 14:57:18 INFO appmaster.ApplicationMasterService: All
containers completed. Shutting down application master.
14/04/06 14:57:18 INFO appmaster.ApplicationMasterService: Stop
application master with spec:
{"name":"ExampleBundledJarApp","runnables":{"BundledJarRunnable":{"name":"BundledJarRunnable","runnable":{"classname":"org.apache.twill.ext.BundledJarRunnable","name":"BundledJarRunnable","arguments":{}},"resources":{"cores":1,"memorySize":512,"instances":1,"uplink":-1,"downlink":-1,"hosts":[],"racks":[]},"files":[{"name":"job-web-service-1.0.jar","uri":"file:/twill/ExampleBundledJarApp/03f7c49a-6471-40b6-9992-af07905d44cc/job-web-service-1.0.jar.7e6aaa7d-fc66-462d-abeb-3b2fb3b7d812.jar","lastModified":1396821425000,"size":75347840,"archive":false,"pattern":null}]}},"orders":[{"names":["BundledJarRunnable"],"type":"STARTED"}],"handler":{"classname":"org.apache.twill.internal.LogOnlyEventHandler","configs":{}}}
14/04/06 14:57:18 INFO appmaster.RunningContainers: Stopping all
instances of BundledJarRunnable
Twill example code:
-------------------
//Example Application
private static class ExampleBundledJarApp implements TwillApplication {
77 private final String jarName;
78 private final URI jarURI;
79
80 public ExampleBundledJarApp(String jarName, URI jarURI) {
81 this.jarName = jarName;
82 this.jarURI = jarURI;
83 }
84
85 @Override
86 public TwillSpecification configure() {
87 return TwillSpecification.Builder.with()
88
.setName("ExampleBundedJarApp").withRunnable()
89 .add("BundledJarRunnable",
new BundledJarRunnable())
90
.withLocalFiles().add(jarName, jarURI, false).apply()
91 .anyOrder().build();
92 }
93 }
//calling twill runner
Arguments arguments = new Arguments.Builder()
204
.setJarFileName("lib/job-web-service-1.0.jar").setLibFolder("lib")
205
.setMainClassName(Main.class.getPackage().getName()).setMainArgs(new
String[]{"1"})
206 .createArguments();
207
208
209
210 LOG.info("arguments jar file:: "+
arguments.getJarFileName());
211 LOG.info("arguments lib folder:: "+
arguments.getLibFolder());
212 LOG.info("arguments main class:: "+
arguments.getMainClassName());
213 LOG.info("arguments main args:: "+
arguments.getMainArgs());
214
215 LOG.info("arguments:: "+
Arrays.asList(arguments.toArray()));
216
217
218 File jarFile = new File(arguments.getJarFileName());
219 Preconditions.checkArgument(jarFile != null);
220 Preconditions.checkState(jarFile.exists(), "jar file
does not exist: "+jarFile.getAbsolutePath());
221 Preconditions.checkState(jarFile.canRead());
222
223
224 TwillPreparer twillPreparer = runnerService
225 .prepare(new
ExampleBundledJarApp(jarFile.getName(), jarFile.toURI()))
226 .withArguments("BundledJarRunnable",
arguments.toArray())
227 .addLogHandler(
228 new
PrinterLogHandler(new PrintWriter(System.out)));
229
230 twillPreparer.start();
There are two problem I observed here:
1. if u dont set any main arg it throw an exception, need to be fixed by
BundledJarRunnable.Arguments by checking main args for being null
2. I see arguments are not populated in the json (check the container error
log above).
Thanks,
Aakash
On Tue, Apr 1, 2014 at 4:35 PM, Alvin Wang <[email protected]> wrote:
> I recently committed something that can serve as a workaround for this,
> when I was trying to get Presto running with Twill. Presto was using Guava
> 16 which was incompatible with Twill's Guava 13.
>
> You can take a look at BundledJarRunnable available in the 0.2.0-incubating
> release (sample
> usage<
> https://git-wip-us.apache.org/repos/asf?p=incubator-twill.git;a=blob;f=twill-examples/yarn/src/main/java/org/apache/twill/example/yarn/BundledJarExample.java;h=3cc88a1b7624270c2d870c8a249185d608b09613;hb=d4a352de4a62abdf4607af77d610597630e0ee1a
> >).
> It's currently very limited in terms of functionality, but you can use it
> to run an application without having to worry about Twill's classpath. You
> can specify a bundle jar file, folder containing the dependencies within
> the jar file, main class name, and arguments to send to the main method.
>
> The BundledJarRunnable will load the jars in the folder containing the
> dependencies and also the bundle jar into a clean classloader, so it won't
> conflict with Twill's dependencies. To use this, do the following:
>
> 1. Package your program into a bundle jar. I used the following in my
> pom.xml:
>
> <build>
> <plugins>
> <plugin>
> <groupId>org.apache.felix</groupId>
> <artifactId>maven-bundle-plugin</artifactId>
> <version>2.3.7</version>
> <extensions>true</extensions>
> <configuration>
> <instructions>
> <Embed-Dependency>*;inline=false</Embed-Dependency>
> <Embed-Transitive>true</Embed-Transitive>
> <Embed-Directory>lib</Embed-Directory>
> </instructions>
> </configuration>
> </plugin>
> </plugins>
> </build>
>
> 2. Configure your TwillApplication to include a BundledJarRunnable() and
> the jar file:
> ...withRunnable().add("Test123", new
> BundledJarRunnable()).withLocalFiles().add("sdf.jar", localJarURI, false)
>
> 3. Configure your TwillPreparer to pass runtime arguments (main class name,
> main method args, etc.) to your runnable:
>
> BundledJarRunner.Arguments arguments = new
> BundledJarRunner.Arguments.Builder()
> .set ...
> ...
> .build();
>
> final TwillController controller = twillRunner.prepare(
> new ExampleBundledJarApp()
> .withArguments("Test123", arguments.toArray())
> .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out,
> true)))
> .start();
>
> --
>
> Let me know if you have any questions.
>
> Alvin
>
>
> On Tue, Apr 1, 2014 at 3:01 AM, aakash aakash <[email protected]
> >wrote:
>
> > Hi,
> >
> > I am trying to start a rest web service on hadoop using Twill, but
> > unfortunately jersey use asm 3.1 but in Twill we use asm 4 and source is
> > also different. Which is now a big problem for me.
> >
> > Is there any workaround for this. Any help will be highly appreciated.
> >
> > Thanks,
> > Aakash
> >
>
