thank you, Jake, Ken. Ok then "job" jar i!= shade plugin. It doesn't disassemble dependencies jars, that is. (I guess i still need to find more data about how it is used, but i get the idea).
I am good with this approach. I still slightly at odds with the fact that it has to be done at the release (build) time, but whatever. If we just needs to throw a pack of jars from driver side to classpath at backend, DistributedCache is the tool for it (imo.) On Mon, May 9, 2011 at 11:13 AM, Jake Mannix <[email protected]> wrote: > On Mon, May 9, 2011 at 10:53 AM, Dmitriy Lyubimov <[email protected]> wrote: > >> Ok then i am missing some fundamental knowledge here about the 'job >> jar'. It's probably a lame question, but i'll try to ask it anyway. >> What is a "job jar"? Is it a java spec or hadoop spec? >> > > It's not a spec at all. It's a hadoop convention. The jar you pass in to > the hadoop > shell script "hadoop jar mystuff.jar myclassname -myargs") has a MANIFEST > which appends to the classpath of the mappers and reducers the contents of > its own lib directory (inside the jar), where other jars reside. > > This is exactly analogous to the way servlet containers deal with .war files > (except that WAR files became an actual spec). > > People in hadoop-land call the "jar with manifest pointing to its own > internal > lib directory" as a "job" jar. > > -jake >
