Hey everyone,
I thought it best to send an email to the list before merging and tagging a 0.1.0 release for the Hadoop on Mesos framework. This release is for a new feature we've been working on for quite some time, which allows Hadoop TaskTrackers to be semi-terminated when they are idle, without destroying any map output they may need to retain for running reduce tasks. Essentially this means that over the lifetime of a job (one with more map/reduce tasks than the size of the cluster) the ratio of map and reduce slots can change, resulting in significantly better resource utilization, because the map slots can be freed up after they have finished doing work. If anyone is running Hadoop on Mesos or would be kind enough to contribute to reviewing the code in the diff, or giving the branch a go on their cluster, that would be very much appreciated! We've been running the patch in production for several months and have seen some quite significant performance gains with our type of workload. The pull request is here https://github.com/mesos/hadoop/pull/33. Feel free to get in touch if you have any questions! Thanks! -- Tom Arnfeld Developer // DueDil
