David, The only reason to roll your own is if you do this in a specific map reduce job and the functionality doesn't already exist. And no, I'm not suggesting that.
As to OSGi for a solution, sure, that would work. I think Chris was looking at something like Tomcat because they have also had to deal with this issue, so a solution exists. I wasn't aware of the OSGi implementations so that too is an option. The reason I started to look at this in terms of the Configuration class is that it looks like some of the pieces are already in place and I was wondering if the class already has enough of a solution in place to allow for custom class loads? Thanks to everyone! -Mike > Date: Fri, 16 Apr 2010 10:05:05 +0100 > Subject: Re: Custom Class Loader for Hadoop M/R jobs? > From: [email protected] > To: [email protected] > > Hi there, > > Many thanks to Caspar for the kind words. Yep we have had Hadoop > running in an OSGi environment in the past - though it was for a proof > of concept so I've not tracked the current Hadoop trunk to see if that > patch still applies. I'm more than happy to try to offer advice if > this is something you wish to pursue, though I'd also highly recommend > the Felix mailing lists ([email protected] or > [email protected]) as excellent sources of OSGi wisdom. > > Regarding licensing again IANAL but here's the list of projects at > apache are using OSGi (that I am aware of) so I'm pretty certain it's > compatible... > > http://felix.apache.org > http://aries.apache.org > http://sling.apache.org > http://tuscany.apache.org > http://servicemix.apache.org > http://activemq.apache.org > http://cxf.apache.org > > Also I know there's been work to make other library projects such as > Apache Commons OSGi compatible. > > Regarding custom classloaders I'd tend to advise checking if OSGi > solves your problems first before trying to role your own - the > benefits of using a common classloading model is that your components > can then interact with other frameworks - also if you run into complex > problems their are a wealth of experts out there who can help you - > where as with a custom approach you are on your own. > > Regards, > > Dave > > > On Thu, 15 Apr 2010 10:47, Caspar MacRae wrote: > > > > ServiceMix and ActiveMQ are also using OSGi in their core AFAIK. > > > > The way this (these) problems have been solved is more of an afterthought, > > and regardless is quite simplistic (thinking of classloader isolation in web > > containers, app servers, and the monolithic WARs produced as a result), > > however OSGi's approach to modularity provides a truly elegant solution. > > > > wrt to the ticket https://issues.apache.org/jira/browse/MAPREDUCE-243, Dave > > Savage is a nice chap who works for Paremus (an enterprise OSGi provider) I > > spoke with him at the recent OSGi devcon in London and he did quite a bit of > > work to get this working for private clients - I'm sure with his expertise > > it would not be a huge effort to get this rolling again. > > > > > On 15 April 2010 18:23, Segel, Mike <[email protected]> wrote: > > > > > > Ok, so are you using OSGi's code, or just writing your own code to > > > implement a published API/specification? > > > > > > I realize this is getting off topic, so you may just want to e-mail me > > > directly. > > > > > > Thx > > > > > > -Mike > > > > > > -----Original Message----- > > > From: Marcel May [mailto:[email protected]] > > > Sent: Thursday, April 15, 2010 11:35 AM > > > To: [email protected] > > > Subject: Re: Custom Class Loader for Hadoop M/R jobs? > > > > > > > > > On Apr 15, 2010, at 6:27 PM, Segel, Mike wrote: > > > > > > > Caspar, > > > > > > > > While IANAL, I did a first blush read of OSGi's Specification License, > > > Version 1.0. on the website link: > > > http://www.osgi.org/Specifications/Licensing as well as the license on the > > > download page: http://www.osgi.org/Download/Release4V42?info=nothanks. > > > > > > > > In a nutshell, OSGi's Specification License, Version 1.0. is > > > > incompatible > > > with Apache's. > > > > > > > > That alone can explain why Apache projects won't touch OSGi. > > > > > > > > > > http://felix.apache.org/ > > > > > > > From a more practical issue, this problem is solved in other Apache > > > projects, so why not borrow from another Apache project like Tomcat? > > > > > > > > Does that make sense? It appears OSGi is a non-starter. > > > > > > > > HTH > > > > > > > > -Mike > > > ... > > > > > > Cheers, > > > Marcel > > > > > > > > > The information contained in this communication may be CONFIDENTIAL and is > > > intended only for the use of the recipient(s) named above. If you are not > > > the intended recipient, you are hereby notified that any dissemination, > > > distribution, or copying of this communication, or any of its contents, is > > > strictly prohibited. If you have received this communication in error, > > > please notify the sender and delete/destroy the original message and any > > > copy of it from your computer or paper files. > > > _________________________________________________________________ The New Busy is not the old busy. Search, chat and e-mail from your inbox. http://www.windowslive.com/campaign/thenewbusy?ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_3
