This might be of use to you, as I had a related problem years ago: http://www.developertutorials.com/tutorials/java/simplify-application-delivery-one-jar-050422/page5.html
The trick is a one-jar solution allows you to embed jar files and load classes from them. Rolling up all your jar variants in a single jar and then selectively loading the desired version might provide you a useful solution. The source is there, so you could start with a working classloader and perhaps apply a few tweaks. On Apr 14, 2010, at 10:33 AM, Segel, Mike wrote: > Scott, > > While that may work for a quick fix. Its not a good long term solution and > you then run in to a problem where you upgrade your hadoop release and the > removed jar is replaced or if you replace the jar, it possible to get > overwritten. > > In this specific instance, the Jackson libraries are not that important and > they can be replaced. > But that doesn't mean that this issue won't come up again and its something > you can't easily pop out and replace. > > This is why I'm looking at custom class loading and trying to understand what > can be accomplished with the methods in the Configuration class. > > Thx > > -Mike > > > -----Original Message----- > From: Scott Carey [mailto:[email protected]] > Sent: Wednesday, April 14, 2010 12:02 PM > To: [email protected] > Subject: Re: Custom Class Loader for Hadoop M/R jobs? > > Depending on what the dependency is, you might be able to just remove it from > hadoop's lib directory on your cluster. > > For me, Hadoop's later versions has jackson-1.0.1 in its lib directory and > that breaks usage of Avro in a M/R job among other things. However, the > feature that uses this library is unimportant to me (configuration dump in > JSON format) so I just removed the jar. > > -Scott > > On Apr 14, 2010, at 6:39 AM, Segel, Mike wrote: > >> Hi, >> >> Ok, here's a bit of a bizarre issue... >> >> How do you handle class collisions between Hadoop and your m/r job which >> calls other 3rd party classes. >> >> An example: Hadoop has an older version of an open source jar in its /lib >> directory. You're interfacing with a 3rd party OS tool that uses a later >> release of the same jar. >> >> You can modify the classpath, and that might work. But the better way is to >> create a Custom Class Loader. (Non-trivial) >> >> Looking at the Configuration class, it looks like there are a couple of >> methods that deal with loading a class in to the configuration so that the >> m/r jobs can have access to them on each node. >> >> Is this the correct intended use, or am I missing something? >> Has anyone done something like this? >> >> Thx >> >> -Mike >> >> Michael Segel >> Architect, R&D >> NAVTEQ >> 425 West Randolph Street >> Chicago, IL 60606 >> (T) +1 312-780-3432 >> (C) +1 312-952-8175 >> www.navteq.com<http://www.navteq.com/> >> >> >> >> The information contained in this communication may be CONFIDENTIAL and is >> intended only for the use of the recipient(s) named above. If you are not >> the intended recipient, you are hereby notified that any dissemination, >> distribution, or copying of this communication, or any of its contents, is >> strictly prohibited. If you have received this communication in error, >> please notify the sender and delete/destroy the original message and any >> copy of it from your computer or paper files. > > > > The information contained in this communication may be CONFIDENTIAL and is > intended only for the use of the recipient(s) named above. If you are not > the intended recipient, you are hereby notified that any dissemination, > distribution, or copying of this communication, or any of its contents, is > strictly prohibited. If you have received this communication in error, > please notify the sender and delete/destroy the original message and any copy > of it from your computer or paper files.
