Thanks for taking 2.7.4 over Konstantin! Regarding rolling RC next week, I still see that there are 4 blocker / critical tickets targeted for 2.7.4: https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20MAPREDUCE%2C%20HADOOP%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%202.7.4 <https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS,%20MAPREDUCE,%20HADOOP,%20YARN)%20AND%20priority%20in%20(Blocker,%20Critical)%20AND%20resolution%20=%20Unresolved%20AND%20%22Target%20Version/s%22%20=%202.7.4>.
We should get closure on them. https://issues.apache.org/jira/browse/HDFS-11742 <https://issues.apache.org/jira/browse/HDFS-11742> definitely was something that was deemed a blocker for 2.8.2, not sure about 2.7.4. I’m ‘back’ - let me know if you need any help. Thanks +Vinod > On Jul 13, 2017, at 5:45 PM, Konstantin Shvachko <shv.had...@gmail.com> wrote: > > Hi everybody. > > We have been doing some internal testing of Hadoop 2.7.4. The testing is > going well. > Did not find any major issues on our workloads. > Used an internal tool called Dynamometer to check NameNode performance on > real cluster traces. Good. > Overall test cluster performance looks good. > Some more testing is still going on. > > I plan to build an RC next week. If there are no objection. > > Thanks, > --Konst > > On Thu, Jun 15, 2017 at 4:42 PM, Konstantin Shvachko <shv.had...@gmail.com> > wrote: > >> Hey guys. >> >> An update on 2.7.4 progress. >> We are down to 4 blockers. There is some work remaining on those. >> https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814 >> Would be good if people could follow up on review comments. >> >> I looked through nightly Jenkins build results for 2.7.4 both on Apache >> Jenkins and internal. >> Some test fail intermittently, but there no consistent failures. I filed >> HDFS-11985 to track some of them. >> https://issues.apache.org/jira/browse/HDFS-11985 >> I do not currently consider these failures as blockers. LMK if some of >> them are. >> >> We started internal testing of branch-2.7 on one of our smallish (100+ >> nodes) test clusters. >> Will update on the results. >> >> There is a plan to enable BigTop for 2.7.4 testing. >> >> Akira, Brahma thank you for setting up a wiki page for 2.7.4 release. >> Thank you everybody for contributing to this effort. >> >> Regards, >> --Konstantin >> >> >> On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org> >> wrote: >> >>> Sure. >>> If you want to edit the wiki, please tell me your ASF confluence account. >>> >>> -Akira >>> >>> On 2017/05/30 15:31, Rohith Sharma K S wrote: >>> >>>> Couple of more JIRAs need to be back ported for 2.7.4 release. These will >>>> solve RM HA unstability issues. >>>> https://issues.apache.org/jira/browse/YARN-5333 >>>> https://issues.apache.org/jira/browse/YARN-5988 >>>> https://issues.apache.org/jira/browse/YARN-6304 >>>> >>>> I will raise a JIRAs to back port it. >>>> >>>> @Akira , could you help to add these JIRAs into wiki? >>>> >>>> Thanks & Regards >>>> Rohith Sharma K S >>>> >>>> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote: >>>> >>>> Created a page for 2.7.4 release. >>>>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4 >>>>> >>>>> If you want to edit this wiki, please ping me. >>>>> >>>>> Regards, >>>>> Akira >>>>> >>>>> >>>>> On 2017/05/23 4:42, Brahma Reddy Battula wrote: >>>>> >>>>> Hi Konstantin Shvachko >>>>>> >>>>>> >>>>>> how about creating a wiki page for 2.7.4 release status like 2.8 and >>>>>> trunk in following link.?? >>>>>> >>>>>> >>>>>> https://cwiki.apache.org/confluence/display/HADOOP >>>>>> >>>>>> >>>>>> ________________________________ >>>>>> From: Konstantin Shvachko <shv.had...@gmail.com> >>>>>> Sent: Saturday, May 13, 2017 3:58 AM >>>>>> To: Akira Ajisaka >>>>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; >>>>>> yarn-...@hadoop.apache.org >>>>>> Subject: Re: About 2.7.4 Release >>>>>> >>>>>> Latest update on the links and filters. Here is the correct link for >>>>>> the >>>>>> filter: >>>>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa? >>>>>> requestId=12340814 >>>>>> >>>>>> Also updated: https://s.apache.org/Dzg4 >>>>>> >>>>>> Had to do some Jira debugging. Sorry for confusion. >>>>>> >>>>>> Thanks, >>>>>> --Konstantin >>>>>> >>>>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko < >>>>>> shv.had...@gmail.com> >>>>>> wrote: >>>>>> >>>>>> Hey Akira, >>>>>> >>>>>>> >>>>>>> I didn't have private filters. Most probably Jira caches something. >>>>>>> Your filter is in the right direction, but for some reason it lists >>>>>>> only >>>>>>> 22 issues, while mine has 29. >>>>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir >>>>>>> a/browse/YARN-5543> >>>>>>> . >>>>>>> >>>>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers", >>>>>>> shared it with "everybody", and updated my link to point to that >>>>>>> filter. >>>>>>> So >>>>>>> you can use any of the three methods below to get the correct list: >>>>>>> 1. Go to https://s.apache.org/Dzg4 >>>>>>> 2. Go to the filter via >>>>>>> https://issues.apache.org/jira/issues?filter=12340814 >>>>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira >>>>>>> 3. On Advanced issues search page paste this: >>>>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = >>>>>>> release-blocker >>>>>>> AND "Target Version/s" = 2.7.4 >>>>>>> >>>>>>> Hope this solves the confusion for which issues are included. >>>>>>> Please LMK if it doesn't, as it is important. >>>>>>> >>>>>>> Thanks, >>>>>>> --Konstantin >>>>>>> >>>>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> >>>>>>> wrote: >>>>>>> >>>>>>> Hi Konstantin, >>>>>>> >>>>>>>> >>>>>>>> Thank you for volunteering as release manager! >>>>>>>> >>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4 >>>>>>>> >>>>>>>>> >>>>>>>>> I couldn't see the link. Maybe is it private filter? >>>>>>>> >>>>>>>> Here is a link I generated: https://s.apache.org/ehKy >>>>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4 >>>>>>>> >>>>>>>> Thanks and Regards, >>>>>>>> Akira >>>>>>>> >>>>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote: >>>>>>>> >>>>>>>> Hi Brahma Reddy Battula, >>>>>>>> >>>>>>>>> >>>>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4 >>>>>>>>> Your link excludes closed and resolved issues, which needs >>>>>>>>> backporting, >>>>>>>>> and >>>>>>>>> which we cannot reopen, as discussed in this thread earlier. >>>>>>>>> >>>>>>>>> Looked through the issues you proposed: >>>>>>>>> >>>>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311> >>>>>>>>> Seems like a new feature. It helps failover to standby node when >>>>>>>>> primary >>>>>>>>> is >>>>>>>>> under heavy load, but it introduces new APIs, addresses, config >>>>>>>>> parameters. >>>>>>>>> And needs at least one follow up jira. >>>>>>>>> Looks like a backward compatible change, though. >>>>>>>>> Did you have a chance to run it in production? >>>>>>>>> >>>>>>>>> +1 on >>>>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987> >>>>>>>>> >>>>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...< >>>>>>>> >>>>>>> https://issues.apache.org/jira/browse/HDFS-10987> >>>>>> issues.apache.org >>>>>> When user want to decommission a node which having 50M blocks ,it could >>>>>> hold the namesystem lock for long time.We've seen it is taking 36 sec. >>>>>> As >>>>>> we knew during this ... >>>>>> >>>>>> >>>>>> >>>>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902> >>>>>> >>>>>>> >>>>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...< >>>>>>>> >>>>>>> https://issues.apache.org/jira/browse/HDFS-9902> >>>>>> issues.apache.org >>>>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and >>>>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved. >>>>>> The >>>>>> DISK size may be several ... >>>>>> >>>>>> >>>>>> >>>>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312> >>>>>> >>>>>>> >>>>>>>>> Trash does not descent into child directories to check for ...< >>>>>>>> >>>>>>> https://issues.apache.org/jira/browse/HDFS-8312> >>>>>> issues.apache.org >>>>>> HDFS trash does not descent into child directory to check if user has >>>>>> permission to delete files. For example: Run the following command to >>>>>> initialize directory ... >>>>>> >>>>>> >>>>>> >>>>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100> >>>>>> >>>>>>> >>>>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...< >>>>>>>> >>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100> >>>>>> issues.apache.org >>>>>> Recently there was on vulnerability reported on jsch library. Its fixed >>>>>> in latest 0.1.54 version before CVE was made public. >>>>>> https://cve.mitre.org/cgi-bin/cvename.cgi ... >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Added them to 2.7.4 release. You should see them via the above link >>>>>>>>> now. >>>>>>>>> Would be good if you could attach backport patches for some of them? >>>>>>>>> >>>>>>>>> Appreciate your help, >>>>>>>>> --Konstantin >>>>>>>>> >>>>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula < >>>>>>>>> brahmareddy.batt...@huawei.com> wrote: >>>>>>>>> >>>>>>>>> >>>>>>>>> Looks following link is not correct.. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> https://s.apache.org/Dzg4 >>>>>>>>>> >>>>>>>>>> It should be like following..? >>>>>>>>>> >>>>>>>>>> https://s.apache.org/wi3U >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me >>>>>>>>>> know >>>>>>>>>> your thoughts on this. >>>>>>>>>> >>>>>>>>>> For Large Cluster: >>>>>>>>>> ============= >>>>>>>>>> >>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line >>>>>>>>>> Protocol >>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission >>>>>>>>>> Expensive when lot's of blocks are present >>>>>>>>>> >>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902=== >>>>>>>>>> "dfs.datanode.du.reserved" per Storage Type >>>>>>>>>> >>>>>>>>>> For Security: >>>>>>>>>> ========= >>>>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not >>>>>>>>>> descent >>>>>>>>>> into child directories to check for permission >>>>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch >>>>>>>>>> jar >>>>>>>>>> to >>>>>>>>>> latest version to fix vulnerability in old versions >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Regards >>>>>>>>>> Brahma Reddy Battula >>>>>>>>>> >>>>>>>>>> -----Original Message----- >>>>>>>>>> From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID] >>>>>>>>>> Sent: 06 May 2017 02:40 >>>>>>>>>> To: Konstantin Shvachko >>>>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; >>>>>>>>>> mapreduce-...@hadoop.apache.org >>>>>>>>>> ; >>>>>>>>>> yarn-...@hadoop.apache.org >>>>>>>>>> Subject: Re: About 2.7.4 Release >>>>>>>>>> >>>>>>>>>> List LGTM Konstantin! >>>>>>>>>> >>>>>>>>>> Let's say that we will only create a new tracking JIRA for patches >>>>>>>>>> which >>>>>>>>>> do not backport cleanly, to avoid having too many lying around. >>>>>>>>>> Otherwise >>>>>>>>>> we can directly attach to old ticket. If a clean backport does >>>>>>>>>> happen >>>>>>>>>> to >>>>>>>>>> break a test the nightly build will help us catch it. >>>>>>>>>> >>>>>>>>>> Erik >>>>>>>>>> >>>>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko < >>>>>>>>>> shv.had...@gmail.com> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Great Zhe. Let's monitor the build. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I >>>>>>>>>>> described >>>>>>>>>>> before. >>>>>>>>>>> Target Version/s: 2.7.4 >>>>>>>>>>> Label: release-blocker >>>>>>>>>>> >>>>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please >>>>>>>>>>> let >>>>>>>>>>> me >>>>>>>>>>> know if I missed anything. >>>>>>>>>>> And feel free to pick up any. Most of backports are pretty >>>>>>>>>>> straightforward, but not all. >>>>>>>>>>> >>>>>>>>>>> We can create tracking jiras for backporting if you need to run >>>>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening >>>>>>>>>>> them). >>>>>>>>>>> But I think the final patch should be attached to the original >>>>>>>>>>> jira. >>>>>>>>>>> Otherwise history will be hard to follow. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> --Konstantin >>>>>>>>>>> >>>>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <z...@apache.org> wrote: >>>>>>>>>>> >>>>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests): >>>>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/ >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko < >>>>>>>>>>>> >>>>>>>>>>>> shv.had...@gmail.com> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Hey guys, >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> I and a few of my colleagues would like to help here and move >>>>>>>>>>>>> 2.7.4 >>>>>>>>>>>>> release forward. A few points in this regard. >>>>>>>>>>>>> >>>>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod >>>>>>>>>>>>> hinted on managing the release. Vinod, if you still want the >>>>>>>>>>>>> job / >>>>>>>>>>>>> have bandwidth will be happy to work with you. >>>>>>>>>>>>> Otherwise I am glad to volunteer as the release manager. >>>>>>>>>>>>> >>>>>>>>>>>>> 2. In addition to current blockers and criticals, I would like >>>>>>>>>>>>> to >>>>>>>>>>>>> >>>>>>>>>>>>> propose >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> a >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> few issues to be included in the release, see the list below. >>>>>>>>>>>>> Those >>>>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in >>>>>>>>>>>>> our >>>>>>>>>>>>> >>>>>>>>>>>>> internal >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> branch and run in production. Plus one minor feature "node >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> labeling", which we found very handy, when you have heterogeneous >>>>>>>>>>>>> environments and mixed workloads, like MR and Spark. >>>>>>>>>>>>> >>>>>>>>>>>>> 3. For marking issues for the release I propose to >>>>>>>>>>>>> - set the target version to 2.7.4, and >>>>>>>>>>>>> - add a new label "release-blocker" >>>>>>>>>>>>> That way we will know issues targeted for the release without >>>>>>>>>>>>> reopening them for backports. >>>>>>>>>>>>> >>>>>>>>>>>>> 4. I see quite a few people are interested in the release. With >>>>>>>>>>>>> all >>>>>>>>>>>>> the help I think we can target to release by the end of May. >>>>>>>>>>>>> >>>>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build >>>>>>>>>>>>> for >>>>>>>>>>>>> >>>>>>>>>>>>> 2.7.4 >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> branch. >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Thanks, >>>>>>>>>>>>> --Konstantin >>>>>>>>>>>>> >>>>>>>>>>>>> ========== List of issue for 2.7.4 =========== >>>>>>>>>>>>> ------ Backports >>>>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jir >>>>>>>>>>>>> a/browse/HADOOP-12975>. >>>>>>>>>>>>> Add >>>>>>>>>>>>> >>>>>>>>>>>>> du >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> jitters >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR >>>>>>>>>>>>> >>>>>>>>>>>>> batching >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. >>>>>>>>>>> NPE >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy >>>>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. >>>>>>>>>>>>> fsck >>>>>>>>>>>>> >>>>>>>>>>>>> removal >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> of dot printing >>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>. >>>>>>>>>>>>> space-balanced >>>>>>>>>>>>> policy for balancer >>>>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>. >>>>>>>>>>>>> abort >>>>>>>>>>>>> balancer if upgrade in progress >>>>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>. >>>>>>>>>>>>> skip >>>>>>>>>>>>> small blocks in getBlocks >>>>>>>>>>>>> >>>>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. >>>>>>>>>>>>> SLS >>>>>>>>>>>>> simulator >>>>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. >>>>>>>>>>>>> SLS >>>>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. >>>>>>>>>>>>> SLS >>>>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. >>>>>>>>>>>>> SLS >>>>>>>>>>>>> >>>>>>>>>>>>> ----- Node labeling >>>>>>>>>>>>> MAPREDUCE-6304 >>>>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304> >>>>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943> >>>>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109> >>>>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140> >>>>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250> >>>>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925> >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> >>>>>>>>>>>>> Zhe Zhang >>>>>>>>>>>> Apache Hadoop Committer >>>>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >>>>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >>>>> >>>>> >>>>> >>>> >>
