We're running the framework to support our legacy jobs written in Hadoop MRv1. Essentially this is a feature that moves further towards getting Hadoop to play nicely on a shared cluster.
The Hadoop on Mesos framework is pretty greedy at the moment, and it can be quite problematic if you're trying to pack a multi-tenant cluster to the max. -- Tom Arnfeld Developer // DueDil On Saturday, Mar 28, 2015 at 2:40 pm, Jeff Schroeder <jeffschroe...@computer.org>, wrote: Does this have any pros / cons over Myriad, which runs Yarn on Mesos? Other than not requiring Yarn :) On Saturday, March 28, 2015, Tom Arnfeld <t...@duedil.com> wrote: Hey everyone, I thought it best to send an email to the list before merging and tagging a 0.1.0 release for the Hadoop on Mesos framework. This release is for a new feature we've been working on for quite some time, which allows Hadoop TaskTrackers to be semi-terminated when they are idle, without destroying any map output they may need to retain for running reduce tasks. Essentially this means that over the lifetime of a job (one with more map/reduce tasks than the size of the cluster) the ratio of map and reduce slots can change, resulting in significantly better resource utilization, because the map slots can be freed up after they have finished doing work. If anyone is running Hadoop on Mesos or would be kind enough to contribute to reviewing the code in the diff, or giving the branch a go on their cluster, that would be very much appreciated! We've been running the patch in production for several months and have seen some quite significant performance gains with our type of workload. The pull request is here https://github.com/mesos/hadoop/pull/33. Feel free to get in touch if you have any questions! Thanks! -- Tom Arnfeld Developer // DueDil -- Text by Jeff, typos by iPhone
