[
https://issues.apache.org/jira/browse/TWILL-217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15881207#comment-15881207
]
Martin Serrano edited comment on TWILL-217 at 2/23/17 8:31 PM:
---------------------------------------------------------------
Seems then that we will need some way to get the list of all
org.apache.twill.api.* classes off of the classpath in order to do this
dependency walk:
* We could do this ourselves with some custom code.
* Guava 14 introduced such a utility:
[ClassPath|https://github.com/google/guava/wiki/ReflectionExplained#classpath].
What is the history around using guava 13 versus later libraries?
* Could sort of cheat, and do something like what {{YarnTwillRunnerService}}
does but restrict for the api package:
{code:language=java}
// Find all the classpaths for Twill classes. It is used for class
filtering when building application jar
// in the YarnTwillPreparer
Dependencies.findClassDependencies(getClass().getClassLoader(), new
ClassAcceptor() {
@Override
public boolean accept(String className, URL classUrl, URL classPathUrl) {
if (!className.startsWith("org.apache.twill.")) {
return false;
}
twillClassPaths.add(classPathUrl);
return true;
}
}, getClass().getName());
{code}
Thoughts?
was (Author: mserrano):
Seems then that we will need some way to get the list of all
org.apache.twill.api.* classes off of the classpath in order to do this
dependency walk:
* We could do this ourselves with some custom code.
* Guava 14 introduced such a utility:
[ClassPath|https://github.com/google/guava/wiki/ReflectionExplained#classpath].
What is the history around using guava 13 versus later libraries?
* Could sort of cheat, and do something like what {{YarnTwillRunnerService}}
does but just for the api package:
{code:language=java}
// Find all the classpaths for Twill classes. It is used for class
filtering when building application jar
// in the YarnTwillPreparer
Dependencies.findClassDependencies(getClass().getClassLoader(), new
ClassAcceptor() {
@Override
public boolean accept(String className, URL classUrl, URL classPathUrl) {
if (!className.startsWith("org.apache.twill.")) {
return false;
}
twillClassPaths.add(classPathUrl);
return true;
}
}, getClass().getName());
{code}
Thoughts?
> AppMaster launcher should include eventHandler dependencies and nothing else
> from application
> ---------------------------------------------------------------------------------------------
>
> Key: TWILL-217
> URL: https://issues.apache.org/jira/browse/TWILL-217
> Project: Apache Twill
> Issue Type: Improvement
> Components: yarn
> Affects Versions: 0.9.0
> Reporter: Martin Serrano
> Assignee: Martin Serrano
>
> Currently the launcher for the appmaster includes the application.jar
> libraries. This is to support user code that adds an EventHandler. The
> application may have many dependencies and including them in the appmaster
> classpath can lead to otherwise inaddressable incompatibilities.
> In my case, something in my application's large dependency graph was
> interfering with the Kafka server operation. I was not able to determine
> what it was but tweaking the appmaster loader to not include my application
> jars fixed the issue.
> Instead the bundler that creates the twill.jar should include the
> EventHandler extension (if any) as an explicit dependency. In this way, only
> the jars needed to support the event handler will be on the twill classpath.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
