Hi Mike, Yes, that’s the expected behavior. The idea of “withResources” was to add resources to the classpath one by one (like “resources” directory in maven project). I think it’s a good idea to support a withResources that have *.jar classpath expansion. You mind help opening a JIRA for that?
Terence > On Nov 5, 2014, at 10:47 AM, Mike Walch <[email protected]> wrote: > > Hi Terence, > > Thanks for the response! I looked at the "withDependencies" method and it > looks like it could work for my situation. However, when I add my class > using the method, some jar dependencies are still missing as they are > probably not traceable even with the class added. For my situation, it's > probably better anyways to add all of my jars using "withResources". > > I submitted a pull request for TWILL-108 on GitHub. While working on it, I > also saw that a directory could be provided to "withResources" without > causing the exception in TWILL-108. When I tried this out, Twill put all > of my jars in a "resources/" directory and added this directory to the > classpath of my container. However, it added the "resources/" directory > without an asterisk (i.e "resources/*"). By not indicating a classpath > wildcard, the jars in "resources/" where not found by my container. Is > this the expected behaviour? > > Mike > > On Wed, Nov 5, 2014 at 3:22 AM, Terence Yim <[email protected]> wrote: > >> Hi Mike, >> >> For runtime dependencies that are not traceable by looking at the >> class (e.g. class loaded through Class.forName, which basically is the >> mechanism of ServiceLoader), you can use the "withDependencies()" >> method when preparing to launch the app (javadoc at >> http://s.apache.org/WYN). However, it only works if you know what that >> Class is and have access to that Class during compilation time. The >> other way, as you've figured out, is to use the "withResources" call. >> >> I do agree that the library should be smarter in handling duplicated >> jar files (maybe ignoring the second one if the file name are the >> same). I've opened an issue to have that fixed >> (https://issues.apache.org/jira/browse/TWILL-108). >> >> Terence >> >> On Tue, Nov 4, 2014 at 11:25 AM, Mike Walch <[email protected]> wrote: >>> Is there a standard way of handling run-time dependencies in Twill? I am >>> trying to run a Twill application that uses the Java ServiceLoader ( >>> http://docs.oracle.com/javase/7/docs/api/java/util/ServiceLoader.html) >> to >>> load a class at run-time. Due to the class being instantiated at >> run-time, >>> Twill is not including its dependencies when >>> ApplicationBundler#findDependencies() is called and this causes >> exceptions >>> when the application is run in YARN. I tried to fix this by adding every >>> jar in my lib/ directory using TwillPreparer#withResources(). However, >>> this results in the exception below. It looks like jars are being added >>> twice (by findDependencies() and by me using withResources()): >>> >>> Exception in thread " STARTING" java.lang.RuntimeException: >>> java.util.zip.ZipException: duplicate entry: >>> lib/jetty-util-9.0.7.v20131107.jar >>> at com.google.common.base.Throwables.propagate(Throwables.java:160) >>> at >>> >> org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:133) >>> at >>> >> org.apache.twill.internal.AbstractZKServiceController.startUp(AbstractZKServiceController.java:82) >>> at >>> >> org.apache.twill.internal.AbstractExecutionServiceController$ServiceDelegate.startUp(AbstractExecutionServiceController.java:109) >>> at >>> >> com.google.common.util.concurrent.AbstractIdleService$1$1.run(AbstractIdleService.java:43) >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: java.util.zip.ZipException: duplicate entry: >>> lib/jetty-util-9.0.7.v20131107.jar >>> at >> java.util.zip.ZipOutputStream.putNextEntry(ZipOutputStream.java:215) >>> at >> java.util.jar.JarOutputStream.putNextEntry(JarOutputStream.java:109) >>> at >>> >> org.apache.twill.internal.ApplicationBundler.copyResource(ApplicationBundler.java:347) >>> at >>> >> org.apache.twill.internal.ApplicationBundler.createBundle(ApplicationBundler.java:140) >>> at >>> >> org.apache.twill.yarn.YarnTwillPreparer.createContainerJar(YarnTwillPreparer.java:388) >>> at >>> >> org.apache.twill.yarn.YarnTwillPreparer.access$300(YarnTwillPreparer.java:106) >>> at >>> >> org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:264) >>> at >>> >> org.apache.twill.yarn.YarnTwillPreparer$1.call(YarnTwillPreparer.java:253) >>> at >>> >> org.apache.twill.yarn.YarnTwillController.doStartUp(YarnTwillController.java:98) >>> ... 4 more >>> >>> Rather than adding every jar in lib/, I did find a fix by just adding the >>> missing jars. However, I don't really like this as a long term solution >> as >>> I will need to maintain a list of missing jars. Is there a better way to >>> handle run-time dependencies? Would it make sense to not throw the >>> exception above if a duplicate entry is added? This would allow me to >> just >>> include all jars in my lib/ directory to insure all run-time dependencies >>> are present. >>> >>> -Mike >>
