[
https://issues.apache.org/jira/browse/TWILL-63?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14950908#comment-14950908
]
ASF GitHub Bot commented on TWILL-63:
-------------------------------------
Github user chtyim commented on a diff in the pull request:
https://github.com/apache/incubator-twill/pull/66#discussion_r41661046
--- Diff:
twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillRunnerService.java ---
@@ -175,6 +187,36 @@ protected void shutDown() throws Exception {
};
}
+ public Location createTwillJar() throws IOException {
+ ApplicationBundler applicationBundler = new ApplicationBundler(new
ClassAcceptor());
+ List<Class<?>> classes = Lists.newArrayList();
+ classes.add(ApplicationMasterMain.class);
+ classes.add(TwillContainerMain.class);
+ // Stuck in the yarnAppClient class to make bundler being able to
pickup the right yarn-client version
+ classes.add(yarnAppClient.getClass());
+
+ twillDependencyClasses = getTwillDependencyClasses(classes);
+ File tempFile = File.createTempFile("twill", ".jar");
+ Location twillJar = new
LocalLocationFactory().create(tempFile.toURI());
+ applicationBundler.createBundle(twillJar, classes);
+ return twillJar;
+ }
+
+ private Set<String> getTwillDependencyClasses(List<Class<?>> classes)
throws IOException {
+ Iterable<String> classNames = Iterables.transform(classes, new
Function<Class<?>, String>() {
+ @Override
+ public String apply(Class<?> input) {
+ return input.getName();
+ }
+ });
+
+ ClassLoader classLoader =
Thread.currentThread().getContextClassLoader();
+ if (classLoader == null) {
+ classLoader = getClass().getClassLoader();
+ }
+ return Dependencies.getClassDependencies(classLoader, new
ClassAcceptor(), classNames);
--- End diff --
The `getClassDependencies` doesn't give you all twill classes. It only
gives you all classes in the dependencies. I think the better way is using all
classPathURL encountered during the creation of the twill jar to create a
`URLClassLoader`. To check if a class is inside the twill.jar, you can use
`ClassLoader.getResource()` on the `URLClassLoader`.
> Speed up application launch time
> --------------------------------
>
> Key: TWILL-63
> URL: https://issues.apache.org/jira/browse/TWILL-63
> Project: Apache Twill
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 0.2.0-incubating
> Reporter: Terence Yim
> Assignee: Shankar Selvam
> Fix For: 0.7.0-incubating
>
>
> Currently when launching an application, two new jars are always created
> locally, one for AM (appMaster.jar) and one for Container (container.jar) and
> copied to HDFS before submitting the application. The jar files could
> potentially be big and if it doesn't changed, it should require copying to
> HDFS again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
