Hi Sangjin Thanks for bringing this point. We did similar exercise with current YARN-5355 branch today. Tests are validated against default configuration and timeline server v.1.5 as well.
All YARN major features such as RM HA/Restart/work-preserving-restart, NM restart, scheduling, sample MR/distributes shell jobs run are validated. Everything looks fine. This testing will also be done once YARN-5355 branch code freezed completely as well in couple of days. Thanks & Regards Rohith Sharma K S On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote: > Kudos to Vrushali and the team for getting ready for this large and > important feature! I know a huge team effort went into this. I look forward > to seeing this merged. > > I'd like to ask one piece of due diligence. Could you please inspect > rigorously to ensure that when disabled Timeline Service v.2 does not > impact other features in any way? We did a similar exercise when we had the > first drop, and it would be good to repeat that... Thanks! > > Sangjin > > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <andrew.w...@cloudera.com> > wrote: > > > Great, thanks Vrushali! Sounds good to me. > > > > I have a few procedural release notes comments I'll put on YARN-5355, to > > make sure we advertise this to our users appropriately. > > > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan < > > vrushal...@gmail.com> wrote: > > > > > Hi Andrew, > > > > > > Thanks for your response! > > > > > > There have been no changes to existing APIs since alpha1. > > > > > > We at Twitter have tested the feature to demonstrate it works at what > we > > > consider moderate scale but this did not include the security related > > > testing. The security testing is in progress at present by Timeline > > Service > > > V2 team in the community and we think we will have more details on this > > > very soon. > > > > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we > > > think of as "merge-blockers". The issues being targeted for merge are > in > > > [link1] below. There are about 59 jiras of which 56 are completed. > > > > > > We plan to make a new umbrella jira after the merge to trunk. We will > > then > > > create a new branch with the new jira name and move these open jiras > > under > > > YARN-5355 as subtasks of that new umbrella jira. > > > > > > thanks > > > Vrushali > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991 > > > > > > > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang < > andrew.w...@cloudera.com> > > > wrote: > > > > > >> Hi Vrushali, > > >> > > >> Glad to hear this major dev milestone is nearing completion! > > >> > > >> Repeating my request on other merge [DISCUSS] threads, could you > comment > > >> on testing and API stability of this merge? Our timeline for beta1 is > > about > > >> a month out, so there's not much time to fix things beforehand. > > >> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should > > most > > >> of these be moved out to a new umbrella? I'm wondering what needs to > be > > >> completed before sending the merge vote. > > >> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex > the > > >> beta1 release date for this feature than others. Hopefully that won't > be > > >> necessary though :) > > >> > > >> Best, > > >> Andrew > > >> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan < > > >> vrushalic2...@gmail.com> wrote: > > >> > > >>> Looks like some of the hyperlinks appear messed up, my apologies, > > >>> resending > > >>> the same email with hopefully better looking content: > > >>> > > >>> Hi All, > > >>> > > >>> I'd like to open a discussion for merging Timeline Service v2 > > (YARN-5355) > > >>> to trunk in a few weeks. > > >>> > > >>> We have previously completed one merge onto trunk [1] and Timeline > > >>> Service > > >>> v2 has been part of Hadoop release 3.0.0-alpha1. > > >>> > > >>> Since then, we have been working on extending the capabilities of > > >>> Timeline > > >>> Service v2 in a feature branch [2]. There are a few related issues > > >>> pending > > >>> that are being actively worked upon and tested. As soon as they are > > >>> resolved, we plan on starting a merge vote within the next two weeks. > > The > > >>> goal is to get this into hadoop3 beta. > > >>> > > >>> We have paid close attention to ensure that once disabled Timeline > > >>> Service > > >>> v2 does not impact existing functionality when disabled (by default). > > >>> > > >>> At a high level, following are the key features that have been > > >>> implemented > > >>> since 3.0.0-alpha1: > > >>> - Security (via Kerberos Authentication & delegation tokens) > > [YARN-3053] > > >>> - Timeline server usability improvements [timeline-server > > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c > > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv > > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC% > > >>> 2C%20created%20ASC> > > >>> ] > > >>> - HBase specific improvements [atsv2-hbase > > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l > > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C% > > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC> > > >>> ] > > >>> - REST API additions and improvements [timeline-reader > > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c > > >>> omponent%20in%20(timelineclient%2C%20timelinereader)% > > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C% > 20created%20ASC> > > >>> ] > > >>> - Reader side simple authorization via whitelist [YARN-6820] > > >>> > > >>> We would love to get your thoughts on this before we open a real > voting > > >>> thread. > > >>> > > >>> Special thanks to a team of folks who worked hard and contributed > > towards > > >>> this effort via patches, reviews and guidance: Rohith Sharma K S, > Varun > > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep > > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack. > > >>> > > >>> Thanks > > >>> Vrushali > > >>> [1] Merge to trunk: http://www.mail-archive.com/ > yarn-dev@hadoop.apache > > . > > >>> org/msg23897.html > > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/ > msg23897.html> > > >>> [2] feature branch YARN-5355 commits: https://github.com/ap > > >>> ache/hadoop/commits/YARN-5355 > > >>> > > >>> <https://github.com/apache/hadoop/commits/YARN-5355> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> > > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan < > > >>> vrushal...@gmail.com> wrote: > > >>> > > >>> > Hi All > > >>> > > > >>> > I’d like to open a discussion for merging Timeline Service v.2 > > >>> (YARN-5355) > > >>> > to trunk in a few weeks. > > >>> > > > >>> > We have previously completed one merge onto trunk [1] and Timeline > > >>> Service > > >>> > v2 has been part of Hadoop release 3.0.0-alpha1. > > >>> > > > >>> > Since then, we have been working on extending the capabilities of > > >>> Timeline > > >>> > Service v2 in a feature branch [2]. There are a few related issues > > >>> pending > > >>> > that are being actively worked upon & tested. As soon as they are > > >>> resolved, > > >>> > we plan on starting a merge vote within the next 2 weeks. The goal > is > > >>> to > > >>> > get this in for Hadoop3 beta. > > >>> > > > >>> > We have paid close attention to ensure that once disabled Timeline > > >>> Service > > >>> > v.2 does not impact existing functionality when disabled (by > > default). > > >>> > > > >>> > At a high level, following are the key features that have been > > >>> implemented > > >>> > since 3.0.0-alpha1: > > >>> > - Security (via Kerberos Authentication & delegation tokens) at the > > >>> writer > > >>> > [YARN-3053] > > >>> > - Timeline server usability improvements [timeline-server > > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql= > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN- > > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND% > > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated% > > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC> > > >>> > ] > > >>> > - HBase specific improvements [atsv2-hbase > > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql= > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN- > > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat > > >>> ed%20DESC%2C% > > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC> > > >>> > ] > > >>> > - REST API additions & improvements [timeline-reader > > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql= > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN- > > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C% > > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori > > >>> ty%20DESC%2C% > > >>> > 20created%20ASC> > > >>> > ] > > >>> > - Reader side simple authorization via whitelist [YARN-6820] > > >>> > > > >>> > We would love to get your thoughts on these and more before we > open a > > >>> real > > >>> > voting thread. > > >>> > > > >>> > Special thanks to a team of folks who worked hard and contributed > > >>> towards > > >>> > this effort with patches, reviews and guidance: Rohith Sharma K S, > > >>> Varun > > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, > Joep > > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack. > > >>> > > > >>> > Thanks! > > >>> > Vrushali > > >>> > > > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop. > > >>> > apache.org/msg23897.html > > >>> > [2] feature branch YARN-5355 commits: https://github.com/ > > >>> > apache/hadoop/commits/YARN-5355 > > >>> > > > >>> > > >> > > >> > > > > > >
