Hi Mike, For runtime dependencies that are not traceable by looking at the class (e.g. class loaded through Class.forName, which basically is the mechanism of ServiceLoader), you can use the "withDependencies()" method when preparing to launch the app (javadoc at http://s.apache.org/WYN). However, it only works if you know what that Class is and have access to that Class during compilation time. The other way, as you've figured out, is to use the "withResources" call.
I do agree that the library should be smarter in handling duplicated jar files (maybe ignoring the second one if the file name are the same). I've opened an issue to have that fixed (https://issues.apache.org/jira/browse/TWILL-108). Terence On Tue, Nov 4, 2014 at 11:25 AM, Mike Walch <[email protected]> wrote: > Is there a standard way of handling run-time dependencies in Twill? I am > trying to run a Twill application that uses the Java ServiceLoader ( > http://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html) to > load a class at run-time. Due to the class being instantiated at run-time, > Twill is not including its dependencies when > ApplicationBundler#findDependencies() is called and this causes exceptions > when the application is run in YARN. I tried to fix this by adding every > jar in my lib/ directory using TwillPreparer#withResources(). However, > this results in the exception below. It looks like jars are being added > twice (by findDependencies() and by me using withResources()): > > Exception in thread " STARTING" java.lang.RuntimeException: > java.util.zip.ZipException: duplicate entry: > lib/jetty-util-9.0.7.v20131107.jar > at com.google.common.base.Throwables.propagate(Throwables.java:160) > at > org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:133) > at > org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:82) > at > org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:109) > at > com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.util.zip.ZipException: duplicate entry: > lib/jetty-util-9.0.7.v20131107.jar > at java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:215) > at java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109) > at > org.apache.twill.internal.ApplicationBundler.copyResource(ApplicationBundler.java:347) > at > org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:140) > at > org.apache.twill.yarn.YarnTwillPreparer.createContainerJar(YarnTwillPreparer.java:388) > at > org.apache.twill.yarn.YarnTwillPreparer.access$300(YarnTwillPreparer.java:106) > at > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:264) > at > org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:253) > at > org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:98) > ... 4 more > > Rather than adding every jar in lib/, I did find a fix by just adding the > missing jars. However, I don't really like this as a long term solution as > I will need to maintain a list of missing jars. Is there a better way to > handle run-time dependencies? Would it make sense to not throw the > exception above if a duplicate entry is added? This would allow me to just > include all jars in my lib/ directory to insure all run-time dependencies > are present. > > -Mike
