To follow up, this is also a decent solution to a nasty problem in the current 
framework detailed here, https://github.com/mesos/hadoop/issues/32.




--


Tom Arnfeld

Developer // DueDil

On Sat, Mar 28, 2015 at 2:40 PM, Jeff Schroeder
<jeffschroe...@computer.org> wrote:

> Does this have any pros / cons over Myriad, which runs Yarn on Mesos? Other
> than not requiring Yarn :)
> On Saturday, March 28, 2015, Tom Arnfeld <t...@duedil.com> wrote:
>>  Hey everyone,
>>
>> I thought it best to send an email to the list before merging and tagging
>> a 0.1.0 release for the Hadoop on Mesos framework. This release is for a
>> new feature we've been working on for quite some time, which allows Hadoop
>> TaskTrackers to be semi-terminated when they are idle, without destroying
>> any map output they may need to retain for running reduce tasks.
>>
>> Essentially this means that over the lifetime of a job (one with more
>> map/reduce tasks than the size of the cluster) the ratio of map and reduce
>> slots can change, resulting in significantly better resource utilization,
>> because the map slots can be freed up after they have finished doing work.
>>
>> If anyone is running Hadoop on Mesos or would be kind enough to contribute
>> to reviewing the code in the diff, or giving the branch a go on their
>> cluster, that would be very much appreciated! We've been running the patch
>> in production for several months and have seen some quite significant
>> performance gains with our type of workload.
>>
>> The pull request is here https://github.com/mesos/hadoop/pull/33.
>>
>> Feel free to get in touch if you have any questions! Thanks!
>>
>>  --
>>
>> Tom Arnfeld
>> Developer // DueDil
>>
> -- 
> Text by Jeff, typos by iPhone

Reply via email to