I've not seen any issues pertaining to running many short lived frameworks, but that's not near the number of frameworks you'd see if each job was a framework.
We've been pushing all our work on MRv1 High Availability JT upstream on the github.com/mesos/hadoop repo, though there hasn't been much to it. There's some outstanding work in regards to framework failover that needs to be done (there's an issue on GH for this, right now if the JT fails over the mesos framework will re-register and all task trackers relaunched meaning running jobs restart from the beginning) and a couple of small bugs we've found in relation to memory limits that we haven't debugged. Maxime, it'd be cool to hear more about how possible it would be to do port the MRv2 framework equivalent to Mesos. I'm not very familiar with the internals of YARN itself. Tom. On Friday, 25 July 2014, Maxime Brugidou <maxime.brugi...@gmail.com> wrote: > We run both mesos and yarn in prod and it does not make sense to run yarn > over mesos. > > However it would be interesting to find a way to run MRv2 jobs on mesos > with some custom layer to swap yarn with mesos. Not sure how to start > though... MRv2 contains a yarn application master that needs to be > rewritten as a mesos framework scheduler. This is probably doable. However > with MRv2 every map reduce job would be mapped as a new framework in Mesos. > Not sure how many frameworks mesos can run and scale up to. Especially > short lived frameworks. > On Jul 25, 2014 8:54 PM, "Tom Arnfeld" <t...@duedil.com > <javascript:_e(%7B%7D,'cvml','t...@duedil.com');>> wrote: > >> Hey Luyi, >> >> That's correct, the Hadoop framework currently only supports Hadoop 2 >> MRv1. It also doesn't have great support for the HA jobtracker available in >> newer versions of Hadoop, but I've been working on that the past few weeks. >> >> I'm not sure how Hadoop 2 would play with Mesos, but very interested to >> find out more. Am I correct in thinking MRv2 will only run on top of YARN? >> >> I wonder if anyone else on the mailing list is running YARN on top of >> Mesos... >> >> Tom. >> >> On Friday, 25 July 2014, Luyi Wang <wangluyi1...@gmail.com >> <javascript:_e(%7B%7D,'cvml','wangluyi1...@gmail.com');>> wrote: >> >>> Checked the mesos github(https://github.com/mesos/hadoop). It listed >>> support for MapReduce V1 >>> >>> How about the MR V2? >>> >>> Right now we are using cloudera to manage hadoop clusters where uses >>> MRV2. We are planning to migrate all our services to mesos(still in the >>> initial investigating stage). Good suggestions, advice and experiences are >>> welcomed. >>> >>> Thanks a lot! >>> >>> >>> -Luyi. >>> >>> >>> >>>
