Hey guys. An update on 2.7.4 progress. We are down to 4 blockers. There is some work remaining on those. https://issues.apache.org/jira/browse/HDFS-11896?filter=12340814 Would be good if people could follow up on review comments.
I looked through nightly Jenkins build results for 2.7.4 both on Apache Jenkins and internal. Some test fail intermittently, but there no consistent failures. I filed HDFS-11985 to track some of them. https://issues.apache.org/jira/browse/HDFS-11985 I do not currently consider these failures as blockers. LMK if some of them are. We started internal testing of branch-2.7 on one of our smallish (100+ nodes) test clusters. Will update on the results. There is a plan to enable BigTop for 2.7.4 testing. Akira, Brahma thank you for setting up a wiki page for 2.7.4 release. Thank you everybody for contributing to this effort. Regards, --Konstantin On Tue, May 30, 2017 at 12:08 AM, Akira Ajisaka <aajis...@apache.org> wrote: > Sure. > If you want to edit the wiki, please tell me your ASF confluence account. > > -Akira > > On 2017/05/30 15:31, Rohith Sharma K S wrote: > >> Couple of more JIRAs need to be back ported for 2.7.4 release. These will >> solve RM HA unstability issues. >> https://issues.apache.org/jira/browse/YARN-5333 >> https://issues.apache.org/jira/browse/YARN-5988 >> https://issues.apache.org/jira/browse/YARN-6304 >> >> I will raise a JIRAs to back port it. >> >> @Akira , could you help to add these JIRAs into wiki? >> >> Thanks & Regards >> Rohith Sharma K S >> >> On 29 May 2017 at 12:19, Akira Ajisaka <aajis...@apache.org> wrote: >> >> Created a page for 2.7.4 release. >>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.7.4 >>> >>> If you want to edit this wiki, please ping me. >>> >>> Regards, >>> Akira >>> >>> >>> On 2017/05/23 4:42, Brahma Reddy Battula wrote: >>> >>> Hi Konstantin Shvachko >>>> >>>> >>>> how about creating a wiki page for 2.7.4 release status like 2.8 and >>>> trunk in following link.?? >>>> >>>> >>>> https://cwiki.apache.org/confluence/display/HADOOP >>>> >>>> >>>> ________________________________ >>>> From: Konstantin Shvachko <shv.had...@gmail.com> >>>> Sent: Saturday, May 13, 2017 3:58 AM >>>> To: Akira Ajisaka >>>> Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; >>>> yarn-...@hadoop.apache.org >>>> Subject: Re: About 2.7.4 Release >>>> >>>> Latest update on the links and filters. Here is the correct link for the >>>> filter: >>>> https://issues.apache.org/jira/secure/IssueNavigator.jspa? >>>> requestId=12340814 >>>> >>>> Also updated: https://s.apache.org/Dzg4 >>>> >>>> Had to do some Jira debugging. Sorry for confusion. >>>> >>>> Thanks, >>>> --Konstantin >>>> >>>> On Wed, May 10, 2017 at 2:30 PM, Konstantin Shvachko < >>>> shv.had...@gmail.com> >>>> wrote: >>>> >>>> Hey Akira, >>>> >>>>> >>>>> I didn't have private filters. Most probably Jira caches something. >>>>> Your filter is in the right direction, but for some reason it lists >>>>> only >>>>> 22 issues, while mine has 29. >>>>> It misses e.g. YARN-5543 <https://issues.apache.org/jir >>>>> a/browse/YARN-5543> >>>>> . >>>>> >>>>> Anyways, I created a Jira filter now "Hadoop 2.7.4 release blockers", >>>>> shared it with "everybody", and updated my link to point to that >>>>> filter. >>>>> So >>>>> you can use any of the three methods below to get the correct list: >>>>> 1. Go to https://s.apache.org/Dzg4 >>>>> 2. Go to the filter via >>>>> https://issues.apache.org/jira/issues?filter=12340814 >>>>> or by finding "Hadoop 2.7.4 release blockers" filter in the jira >>>>> 3. On Advanced issues search page paste this: >>>>> project in (HDFS, HADOOP, YARN, MAPREDUCE) AND labels = release-blocker >>>>> AND "Target Version/s" = 2.7.4 >>>>> >>>>> Hope this solves the confusion for which issues are included. >>>>> Please LMK if it doesn't, as it is important. >>>>> >>>>> Thanks, >>>>> --Konstantin >>>>> >>>>> On Tue, May 9, 2017 at 9:58 AM, Akira Ajisaka <aajis...@apache.org> >>>>> wrote: >>>>> >>>>> Hi Konstantin, >>>>> >>>>>> >>>>>> Thank you for volunteering as release manager! >>>>>> >>>>>> Actually the original link works fine: https://s.apache.org/Dzg4 >>>>>> >>>>>>> >>>>>>> I couldn't see the link. Maybe is it private filter? >>>>>> >>>>>> Here is a link I generated: https://s.apache.org/ehKy >>>>>> This filter includes resolved issue and excludes fixversion == 2.7.4 >>>>>> >>>>>> Thanks and Regards, >>>>>> Akira >>>>>> >>>>>> On 2017/05/08 19:20, Konstantin Shvachko wrote: >>>>>> >>>>>> Hi Brahma Reddy Battula, >>>>>> >>>>>>> >>>>>>> Actually the original link works fine: https://s.apache.org/Dzg4 >>>>>>> Your link excludes closed and resolved issues, which needs >>>>>>> backporting, >>>>>>> and >>>>>>> which we cannot reopen, as discussed in this thread earlier. >>>>>>> >>>>>>> Looked through the issues you proposed: >>>>>>> >>>>>>> HDFS-9311 <https://issues.apache.org/jira/browse/HDFS-9311> >>>>>>> Seems like a new feature. It helps failover to standby node when >>>>>>> primary >>>>>>> is >>>>>>> under heavy load, but it introduces new APIs, addresses, config >>>>>>> parameters. >>>>>>> And needs at least one follow up jira. >>>>>>> Looks like a backward compatible change, though. >>>>>>> Did you have a chance to run it in production? >>>>>>> >>>>>>> +1 on >>>>>>> HDFS-10987 <https://issues.apache.org/jira/browse/HDFS-10987> >>>>>>> >>>>>>> [HDFS-10987] Make Decommission less expensive when lot of ...< >>>>>> >>>>> https://issues.apache.org/jira/browse/HDFS-10987> >>>> issues.apache.org >>>> When user want to decommission a node which having 50M blocks ,it could >>>> hold the namesystem lock for long time.We've seen it is taking 36 sec. >>>> As >>>> we knew during this ... >>>> >>>> >>>> >>>> HDFS-9902 <https://issues.apache.org/jira/browse/HDFS-9902> >>>> >>>>> >>>>>>> [HDFS-9902] Support different values of dfs.datanode.du ...< >>>>>> >>>>> https://issues.apache.org/jira/browse/HDFS-9902> >>>> issues.apache.org >>>> Now Hadoop support different storage type for DISK, SSD, ARCHIVE and >>>> RAM_DISK, but they share one configuration dfs.datanode.du.reserved. The >>>> DISK size may be several ... >>>> >>>> >>>> >>>> HDFS-8312 <https://issues.apache.org/jira/browse/HDFS-8312> >>>> >>>>> >>>>>>> Trash does not descent into child directories to check for ...< >>>>>> >>>>> https://issues.apache.org/jira/browse/HDFS-8312> >>>> issues.apache.org >>>> HDFS trash does not descent into child directory to check if user has >>>> permission to delete files. For example: Run the following command to >>>> initialize directory ... >>>> >>>> >>>> >>>> HADOOP-14100 <https://issues.apache.org/jira/browse/HADOOP-14100> >>>> >>>>> >>>>>>> Upgrade Jsch jar to latest version to fix vulnerability in ...< >>>>>> >>>>> https://issues.apache.org/jira/browse/HADOOP-14100> >>>> issues.apache.org >>>> Recently there was on vulnerability reported on jsch library. Its fixed >>>> in latest 0.1.54 version before CVE was made public. >>>> https://cve.mitre.org/cgi-bin/cvename.cgi ... >>>> >>>> >>>> >>>> >>>> Added them to 2.7.4 release. You should see them via the above link >>>>>>> now. >>>>>>> Would be good if you could attach backport patches for some of them? >>>>>>> >>>>>>> Appreciate your help, >>>>>>> --Konstantin >>>>>>> >>>>>>> On Mon, May 8, 2017 at 8:39 AM, Brahma Reddy Battula < >>>>>>> brahmareddy.batt...@huawei.com> wrote: >>>>>>> >>>>>>> >>>>>>> Looks following link is not correct.. >>>>>>> >>>>>>>> >>>>>>>> https://s.apache.org/Dzg4 >>>>>>>> >>>>>>>> It should be like following..? >>>>>>>> >>>>>>>> https://s.apache.org/wi3U >>>>>>>> >>>>>>>> >>>>>>>> Apart from Konstantin mentioned,Following also good to go..? let me >>>>>>>> know >>>>>>>> your thoughts on this. >>>>>>>> >>>>>>>> For Large Cluster: >>>>>>>> ============= >>>>>>>> >>>>>>>> https://issues.apache.org/jira/browse/HDFS-9311===Life Line >>>>>>>> Protocol >>>>>>>> https://issues.apache.org/jira/browse/HDFS-10987===Deecommission >>>>>>>> Expensive when lot's of blocks are present >>>>>>>> >>>>>>>> https://issues.apache.org/jira/browse/HDFS-9902=== >>>>>>>> "dfs.datanode.du.reserved" per Storage Type >>>>>>>> >>>>>>>> For Security: >>>>>>>> ========= >>>>>>>> https://issues.apache.org/jira/browse/HDFS-8312===Trash does not >>>>>>>> descent >>>>>>>> into child directories to check for permission >>>>>>>> https://issues.apache.org/jira/browse/HADOOP-14100===Upgrade Jsch >>>>>>>> jar >>>>>>>> to >>>>>>>> latest version to fix vulnerability in old versions >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Regards >>>>>>>> Brahma Reddy Battula >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Erik Krogen [mailto:ekro...@linkedin.com.INVALID] >>>>>>>> Sent: 06 May 2017 02:40 >>>>>>>> To: Konstantin Shvachko >>>>>>>> Cc: Zhe Zhang; Hadoop Common; Hdfs-dev; >>>>>>>> mapreduce-...@hadoop.apache.org >>>>>>>> ; >>>>>>>> yarn-...@hadoop.apache.org >>>>>>>> Subject: Re: About 2.7.4 Release >>>>>>>> >>>>>>>> List LGTM Konstantin! >>>>>>>> >>>>>>>> Let's say that we will only create a new tracking JIRA for patches >>>>>>>> which >>>>>>>> do not backport cleanly, to avoid having too many lying around. >>>>>>>> Otherwise >>>>>>>> we can directly attach to old ticket. If a clean backport does >>>>>>>> happen >>>>>>>> to >>>>>>>> break a test the nightly build will help us catch it. >>>>>>>> >>>>>>>> Erik >>>>>>>> >>>>>>>> On Thu, May 4, 2017 at 7:21 PM, Konstantin Shvachko < >>>>>>>> shv.had...@gmail.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>> Great Zhe. Let's monitor the build. >>>>>>>> >>>>>>>> >>>>>>>>> I marked all jiras I knew of for inclusion into 2.7.4 as I >>>>>>>>> described >>>>>>>>> before. >>>>>>>>> Target Version/s: 2.7.4 >>>>>>>>> Label: release-blocker >>>>>>>>> >>>>>>>>> Here is the link to the list: https://s.apache.org/Dzg4 Please let >>>>>>>>> me >>>>>>>>> know if I missed anything. >>>>>>>>> And feel free to pick up any. Most of backports are pretty >>>>>>>>> straightforward, but not all. >>>>>>>>> >>>>>>>>> We can create tracking jiras for backporting if you need to run >>>>>>>>> Jenkins on the patch (and since Allen does not allow reopening >>>>>>>>> them). >>>>>>>>> But I think the final patch should be attached to the original >>>>>>>>> jira. >>>>>>>>> Otherwise history will be hard to follow. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> --Konstantin >>>>>>>>> >>>>>>>>> On Wed, May 3, 2017 at 4:53 PM, Zhe Zhang <z...@apache.org> wrote: >>>>>>>>> >>>>>>>>> Thanks for volunteering as RM Konstantin! The plan LGTM. >>>>>>>>> >>>>>>>>> >>>>>>>>>> I've created a nightly Jenkins job for branch-2.7 (unit tests): >>>>>>>>>> https://builds.apache.org/job/Hadoop-branch2.7-nightly/ >>>>>>>>>> >>>>>>>>>> On Wed, May 3, 2017 at 12:42 AM Konstantin Shvachko < >>>>>>>>>> >>>>>>>>>> shv.had...@gmail.com> >>>>>>>>>> >>>>>>>>> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Hey guys, >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> I and a few of my colleagues would like to help here and move >>>>>>>>>>> 2.7.4 >>>>>>>>>>> release forward. A few points in this regard. >>>>>>>>>>> >>>>>>>>>>> 1. Reading through this thread since March 1 I see that Vinod >>>>>>>>>>> hinted on managing the release. Vinod, if you still want the job >>>>>>>>>>> / >>>>>>>>>>> have bandwidth will be happy to work with you. >>>>>>>>>>> Otherwise I am glad to volunteer as the release manager. >>>>>>>>>>> >>>>>>>>>>> 2. In addition to current blockers and criticals, I would like to >>>>>>>>>>> >>>>>>>>>>> propose >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> a >>>>>>>>> >>>>>>>>>> >>>>>>>>>> few issues to be included in the release, see the list below. >>>>>>>>>>> Those >>>>>>>>>>> are mostly bug fixes and optimizations, which we already have in >>>>>>>>>>> our >>>>>>>>>>> >>>>>>>>>>> internal >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> branch and run in production. Plus one minor feature "node >>>>>>>>> >>>>>>>>>> >>>>>>>>>> labeling", which we found very handy, when you have heterogeneous >>>>>>>>>>> environments and mixed workloads, like MR and Spark. >>>>>>>>>>> >>>>>>>>>>> 3. For marking issues for the release I propose to >>>>>>>>>>> - set the target version to 2.7.4, and >>>>>>>>>>> - add a new label "release-blocker" >>>>>>>>>>> That way we will know issues targeted for the release without >>>>>>>>>>> reopening them for backports. >>>>>>>>>>> >>>>>>>>>>> 4. I see quite a few people are interested in the release. With >>>>>>>>>>> all >>>>>>>>>>> the help I think we can target to release by the end of May. >>>>>>>>>>> >>>>>>>>>>> Other things include fixing CHANGES.txt and fixing Jenkins build >>>>>>>>>>> for >>>>>>>>>>> >>>>>>>>>>> 2.7.4 >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> branch. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> --Konstantin >>>>>>>>>>> >>>>>>>>>>> ========== List of issue for 2.7.4 =========== >>>>>>>>>>> ------ Backports >>>>>>>>>>> HADOOP-12975 <https://issues.apache.org/jira/browse/HADOOP-12975 >>>>>>>>>>> >. >>>>>>>>>>> Add >>>>>>>>>>> >>>>>>>>>>> du >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> jitters >>>>>>>>> >>>>>>>>>> >>>>>>>>>> HDFS-9710 <https://issues.apache.org/jira/browse/HDFS-9710>. IBR >>>>>>>>>>> >>>>>>>>>>> batching >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> HDFS-10715 <https://issues.apache.org/jira/browse/HDFS-10715>. NPE >>>>>>>>> >>>>>>>>>> >>>>>>>>>> when applying AvailableSpaceBlockPlacementPolicy >>>>>>>>>>> HDFS-2538 <https://issues.apache.org/jira/browse/HDFS-2538>. >>>>>>>>>>> fsck >>>>>>>>>>> >>>>>>>>>>> removal >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> of dot printing >>>>>>>>> >>>>>>>>>> >>>>>>>>>> HDFS-8131 <https://issues.apache.org/jira/browse/HDFS-8131>. >>>>>>>>>>> space-balanced >>>>>>>>>>> policy for balancer >>>>>>>>>>> HDFS-8549 <https://issues.apache.org/jira/browse/HDFS-8549>. >>>>>>>>>>> abort >>>>>>>>>>> balancer if upgrade in progress >>>>>>>>>>> HDFS-9412 <https://issues.apache.org/jira/browse/HDFS-9412>. >>>>>>>>>>> skip >>>>>>>>>>> small blocks in getBlocks >>>>>>>>>>> >>>>>>>>>>> YARN-1471 <https://issues.apache.org/jira/browse/YARN-1471>. SLS >>>>>>>>>>> simulator >>>>>>>>>>> YARN-4302 <https://issues.apache.org/jira/browse/YARN-4302>. SLS >>>>>>>>>>> YARN-4367 <https://issues.apache.org/jira/browse/YARN-4367>. SLS >>>>>>>>>>> YARN-4612 <https://issues.apache.org/jira/browse/YARN-4612>. SLS >>>>>>>>>>> >>>>>>>>>>> ----- Node labeling >>>>>>>>>>> MAPREDUCE-6304 >>>>>>>>>>> <https://issues.apache.org/jira/browse/MAPREDUCE-6304> >>>>>>>>>>> YARN-2943 <https://issues.apache.org/jira/browse/YARN-2943> >>>>>>>>>>> YARN-4109 <https://issues.apache.org/jira/browse/YARN-4109> >>>>>>>>>>> YARN-4140 <https://issues.apache.org/jira/browse/YARN-4140> >>>>>>>>>>> YARN-4250 <https://issues.apache.org/jira/browse/YARN-4250> >>>>>>>>>>> YARN-4925 <https://issues.apache.org/jira/browse/YARN-4925> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Zhe Zhang >>>>>>>>>> Apache Hadoop Committer >>>>>>>>>> http://zhe-thoughts.github.io/about/ | @oldcap >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>> >>>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >>> >>> >>> >>
