Hi Subru, We are also discussing the merge of HDFS-10467 (Router-based federation) and we would like to target 2.9 to do a full release together with YARN federation. Chris Douglas already arranged the integration into trunk for 3.0.0 GA.
Regarding the points to cover: 1. API compatibility: we just extend ClientProtocol so no changes in the API. 2. Turning feature off: if the Router is not started, the feature is disabled completely. 3. Stability/testing: the internal version is heavily tested. We will start testing the OSS version soon. In any case, the feature is isolated and minor bugs will not affect anybody else other than the users of the feature. 4. Deployment: we are currently using 2.7.1 and we would like to switch to 2.9 when available. 5. Timeline: finishing the UI and the security JIRAs in HDFS-10467 should give us a ready to use version. There will be small features added but nothing major. There are a couple minor issues with the merge (e.g., HDFS-12384) but should be worked out soon. Thanks, Inigo On Tue, Sep 5, 2017 at 4:26 PM, Jonathan Hung <jyhung2...@gmail.com> wrote: > Hi Subru, > > Thanks for starting the discussion. We are targeting merging YARN-5734 > (API-based scheduler configuration) to branch-2 before the release of > 2.9.0, since the feature is close to complete. Regarding the requirements > for merge, > > 1. API compatibility - this feature adds new APIs, does not modify any > existing ones. > 2. Turning feature off - using the feature is configurable and is turned > off by default. > 3. Stability/testing - this is an RM-only change, so we plan on deploying > this feature to a test RM and verifying configuration changes for capacity > scheduler. (Right now fair scheduler is not supported.) > 4. Deployment - we want to get this feature in to 2.9.0 since we want to > use this feature and 2.9 version in our next upgrade. > 5. Timeline - we have one main blocker which we are planning to resolve by > end of week. The rest of the month will be testing then a merge vote on the > last week of Sept. > > Please let me know if you have any concerns. Thanks! > > > Jonathan Hung > > On Wed, Jul 26, 2017 at 11:23 AM, J. Rottinghuis <jrottingh...@gmail.com> > wrote: > > > Thanks Vrushali for being entirely open as to the current status of > ATSv2. > > I appreciate that we want to ensure things are tested at scale, and as > you > > said we are working on that right now on our clusters. > > We have tested the feature to demonstrate it works at what we consider > > moderate scale. > > > > I think the criteria for including this feature in the 2.9 release should > > be if it can be safely turned off and not cause impact to anybody not > using > > the new feature. The confidence for this is high for timeline service v2. > > > > Therefore, I think timeline service v2 should definitely be part of 2.9. > > That is the big draw for us to work on stabilizing a 2.9 release rather > > than just going to 2.8 and back-porting things ourselves. > > > > Thanks, > > > > Joep > > > > On Tue, Jul 25, 2017 at 11:39 PM, Vrushali Channapattan < > > vrushalic2...@gmail.com> wrote: > > > > > Thanks Subru for initiating this discussion. > > > > > > Wanted to share some thoughts in the context of Timeline Service v2. > The > > > current status of this module is that we are ramping up for a second > > merge > > > to trunk. We still have a few merge blocker jiras outstanding, which we > > > think we will finish soon. > > > > > > While we have done some testing, we are yet to test at scale. Given all > > > this, we were thinking of initially targeting a beta release vehicle > > rather > > > than a stable release. > > > > > > As such, timeline service v2 has branch-2 branch called as > > > YARN-5355-branch-2 in case anyone wants to try it out. Timeline service > > v2 > > > can be turned off and should not affect the cluster. > > > > > > thanks > > > Vrushali > > > > > > > > > > > > > > > > > > On Mon, Jul 24, 2017 at 1:26 PM, Subru Krishnan <su...@apache.org> > > wrote: > > > > > > > Folks, > > > > > > > > With the release for 2.8, we would like to look ahead to 2.9 release > as > > > > there are many features/improvements in branch-2 (about 1062 > commits), > > > that > > > > are in need of a release vechile. > > > > > > > > Here's our first cut of the proposal from the YARN side: > > > > > > > > 1. Scheduler improvements (decoupling allocation from node > > heartbeat, > > > > allocation ID, concurrency fixes, LightResource etc). > > > > 2. Timeline Service v2 > > > > 3. Opportunistic containers > > > > 4. Federation > > > > > > > > We would like to hear a formal list from HDFS & Hadoop (& MapReduce > if > > > any) > > > > and will update the Roadmap wiki accordingly. > > > > > > > > Considering our familiarity with the above mentioned YARN features, > we > > > > would like to volunteer as the co-RMs for 2.9.0. > > > > > > > > We want to keep the timeline at 8-12 weeks to keep the release > > pragmatic. > > > > > > > > Feedback? > > > > > > > > -Subru/Arun > > > > > > > > > >
