Hi Subru, Thanks for starting the discussion. We are targeting merging YARN-5734 (API-based scheduler configuration) to branch-2 before the release of 2.9.0, since the feature is close to complete. Regarding the requirements for merge,
1. API compatibility - this feature adds new APIs, does not modify any existing ones. 2. Turning feature off - using the feature is configurable and is turned off by default. 3. Stability/testing - this is an RM-only change, so we plan on deploying this feature to a test RM and verifying configuration changes for capacity scheduler. (Right now fair scheduler is not supported.) 4. Deployment - we want to get this feature in to 2.9.0 since we want to use this feature and 2.9 version in our next upgrade. 5. Timeline - we have one main blocker which we are planning to resolve by end of week. The rest of the month will be testing then a merge vote on the last week of Sept. Please let me know if you have any concerns. Thanks! Jonathan Hung On Wed, Jul 26, 2017 at 11:23 AM, J. Rottinghuis <jrottingh...@gmail.com> wrote: > Thanks Vrushali for being entirely open as to the current status of ATSv2. > I appreciate that we want to ensure things are tested at scale, and as you > said we are working on that right now on our clusters. > We have tested the feature to demonstrate it works at what we consider > moderate scale. > > I think the criteria for including this feature in the 2.9 release should > be if it can be safely turned off and not cause impact to anybody not using > the new feature. The confidence for this is high for timeline service v2. > > Therefore, I think timeline service v2 should definitely be part of 2.9. > That is the big draw for us to work on stabilizing a 2.9 release rather > than just going to 2.8 and back-porting things ourselves. > > Thanks, > > Joep > > On Tue, Jul 25, 2017 at 11:39 PM, Vrushali Channapattan < > vrushalic2...@gmail.com> wrote: > > > Thanks Subru for initiating this discussion. > > > > Wanted to share some thoughts in the context of Timeline Service v2. The > > current status of this module is that we are ramping up for a second > merge > > to trunk. We still have a few merge blocker jiras outstanding, which we > > think we will finish soon. > > > > While we have done some testing, we are yet to test at scale. Given all > > this, we were thinking of initially targeting a beta release vehicle > rather > > than a stable release. > > > > As such, timeline service v2 has branch-2 branch called as > > YARN-5355-branch-2 in case anyone wants to try it out. Timeline service > v2 > > can be turned off and should not affect the cluster. > > > > thanks > > Vrushali > > > > > > > > > > > > On Mon, Jul 24, 2017 at 1:26 PM, Subru Krishnan <su...@apache.org> > wrote: > > > > > Folks, > > > > > > With the release for 2.8, we would like to look ahead to 2.9 release as > > > there are many features/improvements in branch-2 (about 1062 commits), > > that > > > are in need of a release vechile. > > > > > > Here's our first cut of the proposal from the YARN side: > > > > > > 1. Scheduler improvements (decoupling allocation from node > heartbeat, > > > allocation ID, concurrency fixes, LightResource etc). > > > 2. Timeline Service v2 > > > 3. Opportunistic containers > > > 4. Federation > > > > > > We would like to hear a formal list from HDFS & Hadoop (& MapReduce if > > any) > > > and will update the Roadmap wiki accordingly. > > > > > > Considering our familiarity with the above mentioned YARN features, we > > > would like to volunteer as the co-RMs for 2.9.0. > > > > > > We want to keep the timeline at 8-12 weeks to keep the release > pragmatic. > > > > > > Feedback? > > > > > > -Subru/Arun > > > > > >
