Is there a standard way of handling run-time dependencies in Twill? I am
trying to run a Twill application that uses the Java ServiceLoader (
http://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html) to
load a class at run-time. Due to the class being instantiated at run-time,
Twill is not including its dependencies when
ApplicationBundler#findDependencies() is called and this causes exceptions
when the application is run in YARN. I tried to fix this by adding every
jar in my lib/ directory using TwillPreparer#withResources(). However,
this results in the exception below. It looks like jars are being added
twice (by findDependencies() and by me using withResources()):
Exception in thread " STARTING" java.lang.RuntimeException:
java.util.zip.ZipException: duplicate entry:
lib/jetty-util-9.0.7.v20131107.jar
at com.google.common.base.Throwables.propagate(Throwables.java:160)
at
org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:133)
at
org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:82)
at
org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:109)
at
com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.zip.ZipException: duplicate entry:
lib/jetty-util-9.0.7.v20131107.jar
at java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:215)
at java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109)
at
org.apache.twill.internal.ApplicationBundler.copyResource(ApplicationBundler.java:347)
at
org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:140)
at
org.apache.twill.yarn.YarnTwillPreparer.createContainerJar(YarnTwillPreparer.java:388)
at
org.apache.twill.yarn.YarnTwillPreparer.access$300(YarnTwillPreparer.java:106)
at
org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:264)
at
org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:253)
at
org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:98)
... 4 more
Rather than adding every jar in lib/, I did find a fix by just adding the
missing jars. However, I don't really like this as a long term solution as
I will need to maintain a list of missing jars. Is there a better way to
handle run-time dependencies? Would it make sense to not throw the
exception above if a duplicate entry is added? This would allow me to just
include all jars in my lib/ directory to insure all run-time dependencies
are present.
-Mike
