Re: linux containers with Hadoop

[Edward Capriolo]

On Fri, Sep 30, 2011 at 9:03 AM, bikash sharma sharmabiks...@gmail.comwrote:

 Hi,
 Does anyone knows if Linux containers (which are like kernel supported
 virtualization technique for providing resource isolation across
 process/appication) have ever been used with Hadoop to provide resource
 isolation for map/reduce tasks?
 If yes, what could be the up/down sides of such approach and how feasible
 it
 is in the context of Hadoop?
 Any pointers if any in terms of papers, etc would be useful.

 Thanks,
 Bikash


Previously hadoop launched map reduce tasks as a single user, now with
security tasks can launch as different users in the same OS/VM. I would say
the closest you can to that isolation is the work done with mesos .
http://www.mesosproject.org/

Re: linux containers with Hadoop

[bikash sharma]

Thanks Edward, so mostly the linux containers are used in Hadoop for
ensuring isolation in terms of providing security across mapreduce jobs from
different users (even mesos seem to leverage the same) not for resource
fairness?

On Fri, Sep 30, 2011 at 1:39 PM, Edward Capriolo edlinuxg...@gmail.comwrote:

 On Fri, Sep 30, 2011 at 9:03 AM, bikash sharma sharmabiks...@gmail.com
 wrote:

  Hi,
  Does anyone knows if Linux containers (which are like kernel supported
  virtualization technique for providing resource isolation across
  process/appication) have ever been used with Hadoop to provide resource
  isolation for map/reduce tasks?
  If yes, what could be the up/down sides of such approach and how feasible
  it
  is in the context of Hadoop?
  Any pointers if any in terms of papers, etc would be useful.
 
  Thanks,
  Bikash
 

 Previously hadoop launched map reduce tasks as a single user, now with
 security tasks can launch as different users in the same OS/VM. I would say
 the closest you can to that isolation is the work done with mesos .
 http://www.mesosproject.org/