Can't answer your question exactly, but can let you know what I do.

I build all dependencies into 1 jar, and by using Maven for my build
environment, when I assemble my jar, I am 100% sure all my
dependencies are collected together.  This is working very nicely for
me and I have used the same scripts for around 20 different jars that
I run on EC2 - each had different dependencies which would have been a
pain to manage seperately, but maven simplifies this massively.

Let me know if you want any of my maven config for assembly etc if you
are a maven user...

Cheers,

Tim


On Wed, Dec 3, 2008 at 3:19 PM, Scott Whitecross <[EMAIL PROTECTED]> wrote:
> What's the best way to use third party libraries with Hadoop?  For example,
> I want to run a job with both a jar file containing the mob, and also extra
> libraries.  I noticed a couple solutions with a search, but I'm hoping for
> something better:
>
> - Merge the third party jar libraries into the job jar
> - Distribute the third party libraries across the cluster in the local boxes
> classpath.
>
> What I'd really like is a way to add an extra option to the hadoop jar
> command, such as hadoop/bin/hadoop jar myJar.jar myJobClass -classpath
> thirdpartyjar1.jar:jar2.jar:etc  args
>
> Anything exist like this?
>

Reply via email to