To follow up, this is also a decent solution to a nasty problem in the current framework detailed here, https://github.com/mesos/hadoop/issues/32.
-- Tom Arnfeld Developer // DueDil On Sat, Mar 28, 2015 at 2:40 PM, Jeff Schroeder <jeffschroe...@computer.org> wrote: > Does this have any pros / cons over Myriad, which runs Yarn on Mesos? Other > than not requiring Yarn :) > On Saturday, March 28, 2015, Tom Arnfeld <t...@duedil.com> wrote: >> Hey everyone, >> >> I thought it best to send an email to the list before merging and tagging >> a 0.1.0 release for the Hadoop on Mesos framework. This release is for a >> new feature we've been working on for quite some time, which allows Hadoop >> TaskTrackers to be semi-terminated when they are idle, without destroying >> any map output they may need to retain for running reduce tasks. >> >> Essentially this means that over the lifetime of a job (one with more >> map/reduce tasks than the size of the cluster) the ratio of map and reduce >> slots can change, resulting in significantly better resource utilization, >> because the map slots can be freed up after they have finished doing work. >> >> If anyone is running Hadoop on Mesos or would be kind enough to contribute >> to reviewing the code in the diff, or giving the branch a go on their >> cluster, that would be very much appreciated! We've been running the patch >> in production for several months and have seen some quite significant >> performance gains with our type of workload. >> >> The pull request is here https://github.com/mesos/hadoop/pull/33. >> >> Feel free to get in touch if you have any questions! Thanks! >> >> -- >> >> Tom Arnfeld >> Developer // DueDil >> > -- > Text by Jeff, typos by iPhone
