+1 Sounds great, thanks for making the improvements. -Elizabeth
On Saturday, March 28, 2015, Brenden Matthews <[email protected]> wrote: > This is great, thanks Tom. > > On Sat, Mar 28, 2015 at 6:52 AM, Tom Arnfeld <[email protected] > <javascript:;>> wrote: > > > Hey everyone, > > > > > > I thought it best to send an email to the list before merging and tagging > > a 0.1.0 release for the Hadoop on Mesos framework. This release is for a > > new feature we've been working on for quite some time, which allows > Hadoop > > TaskTrackers to be semi-terminated when they are idle, without destroying > > any map output they may need to retain for running reduce tasks. > > > > > > Essentially this means that over the lifetime of a job (one with more > > map/reduce tasks than the size of the cluster) the ratio of map and > reduce > > slots can change, resulting in significantly better resource utilization, > > because the map slots can be freed up after they have finished doing > work. > > > > > > If anyone is running Hadoop on Mesos or would be kind enough to > contribute > > to reviewing the code in the diff, or giving the branch a go on their > > cluster, that would be very much appreciated! We've been running the > patch > > in production for several months and have seen some quite significant > > performance gains with our type of workload. > > > > > > The pull request is here https://github.com/mesos/hadoop/pull/33. > > > > > > Feel free to get in touch if you have any questions! Thanks! > > > > > > -- > > > > Tom Arnfeld > > Developer // DueDil >
