Gotcha so this is just a better mesos framework for Hadoop and has nothing to do with the Myriad / Yarn stuff, which similarly, prevents you from having to statically setup a Hadoop cluster.
https://mesosphere.com/2015/02/11/yarn-on-mesos-big-data/ Nice stuff On Saturday, March 28, 2015, Tom Arnfeld <t...@duedil.com> wrote: > To follow up, this is also a decent solution to a nasty problem in the > current framework detailed here, https://github.com/mesos/hadoop/issues/32 > . > > > -- > > Tom Arnfeld > Developer // DueDil > > > On Sat, Mar 28, 2015 at 2:40 PM, Jeff Schroeder < > jeffschroe...@computer.org > <javascript:_e(%7B%7D,'cvml','jeffschroe...@computer.org');>> wrote: > >> Does this have any pros / cons over Myriad, which runs Yarn on Mesos? >> Other than not requiring Yarn :) >> >> On Saturday, March 28, 2015, Tom Arnfeld <t...@duedil.com >> <javascript:_e(%7B%7D,'cvml','t...@duedil.com');>> wrote: >> >>> Hey everyone, >>> >>> I thought it best to send an email to the list before merging and >>> tagging a 0.1.0 release for the Hadoop on Mesos framework. This release is >>> for a new feature we've been working on for quite some time, which allows >>> Hadoop TaskTrackers to be semi-terminated when they are idle, without >>> destroying any map output they may need to retain for running reduce tasks. >>> >>> Essentially this means that over the lifetime of a job (one with more >>> map/reduce tasks than the size of the cluster) the ratio of map and reduce >>> slots can change, resulting in significantly better resource utilization, >>> because the map slots can be freed up after they have finished doing work. >>> >>> If anyone is running Hadoop on Mesos or would be kind enough to >>> contribute to reviewing the code in the diff, or giving the branch a go on >>> their cluster, that would be very much appreciated! We've been running the >>> patch in production for several months and have seen some quite significant >>> performance gains with our type of workload. >>> >>> The pull request is here https://github.com/mesos/hadoop/pull/33. >>> >>> Feel free to get in touch if you have any questions! Thanks! >>> >>> -- >>> >>> Tom Arnfeld >>> Developer // DueDil >>> >> >> >> -- >> Text by Jeff, typos by iPhone >> > > -- Text by Jeff, typos by iPhone
