Hey guys, Looks like we are done with blockers for Apache Hadoop 2.7.4 release. https://issues.apache.org/jira/issues/?filter=12340814
I just committed HDFS-11896 <https://issues.apache.org/jira/browse/HDFS-11896>, and decided not to wait for HDFS-11576 <https://issues.apache.org/jira/browse/HDFS-11576>, see Jira comment. Thanks Vinod for pointing out HDFS-11742 <https://issues.apache.org/jira/browse/HDFS-11742>. It was thoroughly tested on a small cluster and Kihwal committed it last week, thanks. Will start building initial RC. Please refrain from committing to branch-2.7 for some time. Thank you everybody for contributing. --Konst On Thu, Jul 20, 2017 at 12:07 PM, Vinod Kumar Vavilapalli < vino...@apache.org> wrote: > Thanks for taking 2.7.4 over Konstantin! > > Regarding rolling RC next week, I still see that there are 4 blocker / > critical tickets targeted for 2.7.4: https://issues.apache. > org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE% > 2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C% > 20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20% > 22Target%20Version%2Fs%22%20%3D%202.7.4 > <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4> > . > > We should get closure on them. https://issues.apache. > org/jira/browse/HDFS-11742 definitely was something that was deemed a > blocker for 2.8.2, not sure about 2.7.4. > > I’m ‘back’ - let me know if you need any help. > > Thanks > +Vinod > > On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <shv.had...@gmail.com> > wrote: > > Hi everybody. > > We have been doing some internal testing of Hadoop 2.7.4. The testing is > going well. > Did not find any major issues on our workloads. > Used an internal tool called Dynamometer to check NameNode performance on > real cluster traces. Good. > Overall test cluster performance looks good. > Some more testing is still going on. > > I plan to build an RC next week. If there are no objection. > > Thanks, > --Konst > > On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.had...@gmail.com > > > wrote: > > Hey guys. > > An update on 2.7.4 progress. > We are down to 4 blockers. There is some work remaining on those. > https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814 > Would be good if people could follow up on review comments. > > I looked through nightly Jenkins build results for 2.7.4 both on Apache > Jenkins and internal. > Some test fail intermittently, but there no consistent failures. I filed > HDFS-11985 to track some of them. > https://issues.apache.org/jira/browse/HDFS-11985 > I do not currently consider these failures as blockers. LMK if some of > them are. > > We started internal testing of branch-2.7 on one of our smallish (100+ > nodes) test clusters. > Will update on the results. > > There is a plan to enable BigTop for 2.7.4 testing. > > Akira, Brahma thank you for setting up a wiki page for 2.7.4 release. > Thank you everybody for contributing to this effort. > > Regards, > --Konstantin > > > On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org> > wrote: > > Sure. > If you want to edit the wiki, please tell me your ASF confluence account. > > -Akira > > On 2017/05/30 15:31, Rohith Sharma K S wrote: > > Couple of more JIRAs need to be back ported for 2.7.4 release. These will > solve RM HA unstability issues. > https://issues.apache.org/jira/browse/YARN-5333 > https://issues.apache.org/jira/browse/YARN-5988 > https://issues.apache.org/jira/browse/YARN-6304 > > I will raise a JIRAs to back port it. > > @Akira , could you help to add these JIRAs into wiki? > > Thanks & Regards > Rohith Sharma K S > > On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote: > > Created a page for 2.7.4 release. > > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4 > > If you want to edit this wiki, please ping me. > > Regards, > Akira > > > On 2017/05/23 4:42, Brahma Reddy Battula wrote: > > Hi Konstantin Shvachko > > > > how about creating a wiki page for 2.7.4 release status like 2.8 and > trunk in following link.?? > > > https://cwiki.apache.org/confluence/display/HADOOP > > > ________________________________ > From: Konstantin Shvachko <shv.had...@gmail.com> > Sent: Saturday, May 13, 2017 3:58 AM > To: Akira Ajisaka > Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; > yarn-...@hadoop.apache.org > Subject: Re: About 2.7.4 Release > > Latest update on the links and filters. Here is the correct link for > the > filter: > https://issues.apache.org/jira/secure/IssueNavigator.jspa? > requestId=12340814 > > Also updated: https://s.apache.org/Dzg4 > > Had to do some Jira debugging. Sorry for confusion. > > Thanks, > --Konstantin > > On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko < > shv.had...@gmail.com> > wrote: > > Hey Akira, > > > I didn't have private filters. Most probably Jira caches something. > Your filter is in the right direction, but for some reason it lists > only > 22 issues, while mine has 29. > It misses e.g. YARN-5543 <https://issues.apache.org/jir > a/browse/YARN-5543> > . > > Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers", > shared it with "everybody", and updated my link to point to that > filter. > So > you can use any of the three methods below to get the correct list: > 1. Go to https://s.apache.org/Dzg4 > 2. Go to the filter via > https://issues.apache.org/jira/issues?filter=12340814 > or by finding "Hadoop 2.7.4 release blockers" filter in the jira > 3. On Advanced issues search page paste this: > project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = > release-blocker > AND "Target Version/s" = 2.7.4 > > Hope this solves the confusion for which issues are included. > Please LMK if it doesn't, as it is important. > > Thanks, > --Konstantin > > On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> > wrote: > > Hi Konstantin, > > > Thank you for volunteering as release manager! > > Actually the original link works fine: https://s.apache.org/Dzg4 > > > I couldn't see the link. Maybe is it private filter? > > > Here is a link I generated: https://s.apache.org/ehKy > This filter includes resolved issue and excludes fixversion == 2.7.4 > > Thanks and Regards, > Akira > > On 2017/05/08 19:20, Konstantin Shvachko wrote: > > Hi Brahma Reddy Battula, > > > Actually the original link works fine: https://s.apache.org/Dzg4 > Your link excludes closed and resolved issues, which needs > backporting, > and > which we cannot reopen, as discussed in this thread earlier. > > Looked through the issues you proposed: > > HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311> > Seems like a new feature. It helps failover to standby node when > primary > is > under heavy load, but it introduces new APIs, addresses, config > parameters. > And needs at least one follow up jira. > Looks like a backward compatible change, though. > Did you have a chance to run it in production? > > +1 on > HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987> > > [HDFS-10987] Make Decommission less expensive when lot of ...< > > > https://issues.apache.org/jira/browse/HDFS-10987> > > issues.apache.org > When user want to decommission a node which having 50M blocks ,it could > hold the namesystem lock for long time.We've seen it is taking 36 sec. > As > we knew during this ... > > > > HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902> > > > [HDFS-9902] Support different values of dfs.datanode.du ...< > > > https://issues.apache.org/jira/browse/HDFS-9902> > > issues.apache.org > Now Hadoop support different storage type for DISK, SSD, ARCHIVE and > RAM_DISK, but they share one configuration dfs.datanode.du.reserved. > The > DISK size may be several ... > > > > HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312> > > > Trash does not descent into child directories to check for ...< > > > https://issues.apache.org/jira/browse/HDFS-8312> > > issues.apache.org > HDFS trash does not descent into child directory to check if user has > permission to delete files. For example: Run the following command to > initialize directory ... > > > > HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100> > > > Upgrade Jsch jar to latest version to fix vulnerability in ...< > > > https://issues.apache.org/jira/browse/HADOOP-14100> > > issues.apache.org > Recently there was on vulnerability reported on jsch library. Its fixed > in latest 0.1.54 version before CVE was made public. > https://cve.mitre.org/cgi-bin/cvename.cgi ... > > > > > Added them to 2.7.4 release. You should see them via the above link > > now. > Would be good if you could attach backport patches for some of them? > > Appreciate your help, > --Konstantin > > On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula < > brahmareddy.batt...@huawei.com> wrote: > > > Looks following link is not correct.. > > > https://s.apache.org/Dzg4 > > It should be like following..? > > https://s.apache.org/wi3U > > > Apart from Konstantin mentioned,Following also good to go..? let me > know > your thoughts on this. > > For Large Cluster: > ============= > > https://issues.apache.org/jira/browse/HDFS-9311===Life Line > Protocol > https://issues.apache.org/jira/browse/HDFS-10987===Deecommission > Expensive when lot's of blocks are present > > https://issues.apache.org/jira/browse/HDFS-9902=== > "dfs.datanode.du.reserved" per Storage Type > > For Security: > ========= > https://issues.apache.org/jira/browse/HDFS-8312===Trash does not > descent > into child directories to check for permission > https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch > jar > to > latest version to fix vulnerability in old versions > > > > Regards > Brahma Reddy Battula > > -----Original Message----- > From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID] > Sent: 06 May 2017 02:40 > To: Konstantin Shvachko > Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; > mapreduce-...@hadoop.apache.org > ; > yarn-...@hadoop.apache.org > Subject: Re: About 2.7.4 Release > > List LGTM Konstantin! > > Let's say that we will only create a new tracking JIRA for patches > which > do not backport cleanly, to avoid having too many lying around. > Otherwise > we can directly attach to old ticket. If a clean backport does > happen > to > break a test the nightly build will help us catch it. > > Erik > > On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko < > shv.had...@gmail.com> > wrote: > > Great Zhe. Let's monitor the build. > > > I marked all jiras I knew of for inclusion into 2.7.4 as I > described > before. > Target Version/s: 2.7.4 > Label: release-blocker > > Here is the link to the list: https://s.apache.org/Dzg4 Please > let > me > know if I missed anything. > And feel free to pick up any. Most of backports are pretty > straightforward, but not all. > > We can create tracking jiras for backporting if you need to run > Jenkins on the patch (and since Allen does not allow reopening > them). > But I think the final patch should be attached to the original > jira. > Otherwise history will be hard to follow. > > Thanks, > --Konstantin > > On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <z...@apache.org> wrote: > > Thanks for volunteering as RM Konstantin! The plan LGTM. > > > I've created a nightly Jenkins job for branch-2.7 (unit tests): > https://builds.apache.org/job/Hadoop-branch2.7-nightly/ > > On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko < > > shv.had...@gmail.com> > > > wrote: > > > Hey guys, > > > I and a few of my colleagues would like to help here and move > 2.7.4 > release forward. A few points in this regard. > > 1. Reading through this thread since March 1 I see that Vinod > hinted on managing the release. Vinod, if you still want the > job / > have bandwidth will be happy to work with you. > Otherwise I am glad to volunteer as the release manager. > > 2. In addition to current blockers and criticals, I would like > to > > propose > > > > a > > > few issues to be included in the release, see the list below. > > Those > are mostly bug fixes and optimizations, which we already have in > our > > internal > > > > branch and run in production. Plus one minor feature "node > > > labeling", which we found very handy, when you have heterogeneous > > environments and mixed workloads, like MR and Spark. > > 3. For marking issues for the release I propose to > - set the target version to 2.7.4, and > - add a new label "release-blocker" > That way we will know issues targeted for the release without > reopening them for backports. > > 4. I see quite a few people are interested in the release. With > all > the help I think we can target to release by the end of May. > > Other things include fixing CHANGES.txt and fixing Jenkins build > for > > 2.7.4 > > > > branch. > > > > Thanks, > --Konstantin > > ========== List of issue for 2.7.4 =========== > ------ Backports > HADOOP-12975 <https://issues.apache.org/jir > a/browse/HADOOP-12975>. > Add > > du > > > > jitters > > > HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR > > > batching > > > > HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. > NPE > > > when applying AvailableSpaceBlockPlacementPolicy > > HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. > fsck > > removal > > > > of dot printing > > > HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>. > > space-balanced > policy for balancer > HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>. > abort > balancer if upgrade in progress > HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>. > skip > small blocks in getBlocks > > YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. > SLS > simulator > YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. > SLS > YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. > SLS > YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. > SLS > > ----- Node labeling > MAPREDUCE-6304 > <https://issues.apache.org/jira/browse/MAPREDUCE-6304> > YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943> > YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109> > YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140> > YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250> > YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925> > > -- > > Zhe Zhang > > Apache Hadoop Committer > http://zhe-thoughts.github.io/about/ | @oldcap > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > > > >
