I am interested in investigating the idea of YARN on top of Mesos. One of the benefits I can think of is that we can get rid of the static resource allocation between YARN and Mesos clusters. In that way, Mesos can allocate those resources that are not used by YARN to other Mesos frameworks like Aurora, Marathon, etc, to increase the resource utilization of the entire data center. Also, we could avoid running each MRv2 job as a framework which I think might cause some maintenance complexity (e.g. for framework rate limiting, etc). Finally, YARN currently does not have a good isolation support. It only supports cpu isolation right now (using cgroups). By porting YARN on top of Mesos, we might be able to leverage the existing Mesos containerizer strategy to provide better isolation between tasks. Maxime, I am curious why do you think it does not make sense to run YARN over Mesos? Since I am not super familar with YARN, I might be missing something.
I have been thinking of making ResourceManager in YARN a Mesos framework and making NodeManager a Mesos executor. The NodeManager will launch containers using primitives provided by Mesos so that we have a consistent containerizer layer. I haven't fully figured out how this could be done yet (e.g., nested containers, communication between NodeManager and ResourceManager, etc.), but I would love to explore this direction. I would like to hear about any feedback/suggestions you guys have about this direction. Thanks, - Jie On Fri, Jul 25, 2014 at 1:39 PM, Maxime Brugidou <maxime.brugi...@gmail.com> wrote: > We run both mesos and yarn in prod and it does not make sense to run yarn > over mesos. > > However it would be interesting to find a way to run MRv2 jobs on mesos > with some custom layer to swap yarn with mesos. Not sure how to start > though... MRv2 contains a yarn application master that needs to be > rewritten as a mesos framework scheduler. This is probably doable. However > with MRv2 every map reduce job would be mapped as a new framework in Mesos. > Not sure how many frameworks mesos can run and scale up to. Especially > short lived frameworks. > On Jul 25, 2014 8:54 PM, "Tom Arnfeld" <t...@duedil.com> wrote: > >> Hey Luyi, >> >> That's correct, the Hadoop framework currently only supports Hadoop 2 >> MRv1. It also doesn't have great support for the HA jobtracker available in >> newer versions of Hadoop, but I've been working on that the past few weeks. >> >> I'm not sure how Hadoop 2 would play with Mesos, but very interested to >> find out more. Am I correct in thinking MRv2 will only run on top of YARN? >> >> I wonder if anyone else on the mailing list is running YARN on top of >> Mesos... >> >> Tom. >> >> On Friday, 25 July 2014, Luyi Wang <wangluyi1...@gmail.com> wrote: >> >>> Checked the mesos github(https://github.com/mesos/hadoop). It listed >>> support for MapReduce V1 >>> >>> How about the MR V2? >>> >>> Right now we are using cloudera to manage hadoop clusters where uses >>> MRV2. We are planning to migrate all our services to mesos(still in the >>> initial investigating stage). Good suggestions, advice and experiences are >>> welcomed. >>> >>> Thanks a lot! >>> >>> >>> -Luyi. >>> >>> >>> >>>
