Can't answer your question exactly, but can let you know what I do. I build all dependencies into 1 jar, and by using Maven for my build environment, when I assemble my jar, I am 100% sure all my dependencies are collected together. This is working very nicely for me and I have used the same scripts for around 20 different jars that I run on EC2 - each had different dependencies which would have been a pain to manage seperately, but maven simplifies this massively.
Let me know if you want any of my maven config for assembly etc if you are a maven user... Cheers, Tim On Wed, Dec 3, 2008 at 3:19 PM, Scott Whitecross <[EMAIL PROTECTED]> wrote: > What's the best way to use third party libraries with Hadoop? For example, > I want to run a job with both a jar file containing the mob, and also extra > libraries. I noticed a couple solutions with a search, but I'm hoping for > something better: > > - Merge the third party jar libraries into the job jar > - Distribute the third party libraries across the cluster in the local boxes > classpath. > > What I'd really like is a way to add an extra option to the hadoop jar > command, such as hadoop/bin/hadoop jar myJar.jar myJobClass -classpath > thirdpartyjar1.jar:jar2.jar:etc args > > Anything exist like this? >
