Hey everyone, I've been working on a potential extension to Hadoop on Mesos which allows the framework to potentially release allocated (but idle) TaskTracker slots if they are doing nothing. This helps release resources hadoop is allocated but not using, to increase overall cluster utilisation when multiple frameworks are involved.
This scenario most commonly appears when you have a large job with an expensive reduce phase. While the reducers are running the map slots are completely idle, and therefore are unable to be offered to other frameworks that could make use of the resources. However, there are various intricacies of doing this, and it's quite hacky, largely because it requires we change the number of available slots on a TaskTracker while it's running. I'd be interested to hear anyones thoughts on the idea... Especially those that worked on the hadoop framework early on! Pull request is here https://github.com/mesos/hadoop/pull/33 and another issue related to this solution can be found here https://github.com/mesos/hadoop/issues/32. Cheers, Tom.
