[jira] [Created] (HADOOP-19148) Update solr from 8.11.2 to 8.11.3 to address CVE-2023-50298
Brahma Reddy Battula created HADOOP-19148: - Summary: Update solr from 8.11.2 to 8.11.3 to address CVE-2023-50298 Key: HADOOP-19148 URL: https://issues.apache.org/jira/browse/HADOOP-19148 Project: Hadoop Common Issue Type: Improvement Components: common Reporter: Brahma Reddy Battula Update solr from 8.11.2 to 8.11.3 to address CVE-2023-50298 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Query : Hadoop Cluster OS upgrade
Hi All, Does anybody tried out/share learnings ,using maintenance state or upgrade domains for big data cluster OS upgrades? Regards, Brahma
Re: [INFO] Hadoop-3.4 Release Update
Thanks for driving this. One query: Does all the jiras from 3.3 are part of the 3.4? On Mon, 22 Jan 2024 at 11:50 AM, slfan1989 wrote: > Thank you very much for specifying the frozen time for the > Branch-3.4.0. @Xiaoqiao > He > > If there is a need to backport any PRs to branch-3.4/branch-3.4.0, please > feel free to contact me. > > Best Regards, > Shilun Fan. > > On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He wrote: > > > Hi All, > > > > Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024). > > If there is any blocker/critical PR please backport to branch-3.4 and > > branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan. > > > > If any reported issues are not ready in time please wait for the next > > release. > > > > Thanks all. > > > > Best Regards, > > - He Xiaoqiao > > > > > > > > On Mon, Jan 22, 2024 at 10:43 AM slfan1989 wrote: > > > >> Thank you for the feedback! > >> > >> I apologize for updating partial information in the Hadoop module. > >> > >> Today, I will complete the rollback of the modified JIRA and add back > the > >> 3.4.0 version. > >> > >> Best > >> > >> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He > >> wrote: > >> > >>> Thanks all for your input. Connected to Shilun offline yesterday, and > he > >>> will > >>> update or recover the fix version tag before 3.4.0-RC1. > >>> > >>> Best Regards, > >>> - He Xiaoqiao > >>> > >>> > >>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki > > >>> wrote: > >>> > >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should > >>> still > >>> > be > >>> > > included in the Fix versions, > >>> > > even if it was already released in 3.3.x. If I understand > correctly, > >>> this > >>> > > is usually our practice. > >>> > > >>> > +1. > >>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are > >>> > maintaining both branch-3.4 and branch-3.3. > >>> > > >>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma < > tasan...@apache.org> > >>> > wrote: > >>> > > > >>> > > Hi Shilun, > >>> > > > >>> > > Thank you for leading the 3.4.0 release. > >>> > > > >>> > > > 2. In JIRA issues where other release versions have been > released, > >>> > remove > >>> > > > the 3.4.0 entry from the fix version. > >>> > > > >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should > >>> still > >>> > be > >>> > > included in the Fix versions, > >>> > > even if it was already released in 3.3.x. If I understand > correctly, > >>> this > >>> > > is usually our practice. > >>> > > > >>> > > Thanks, > >>> > > - Takanobu > >>> > > > >>> > > 2024年1月21日(日) 13:56 slfan1989 : > >>> > > > >>> > > > Hi All, > >>> > > > > >>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to > >>> > perform > >>> > > > some operations on the JIRA issues related to HADOOP, HDFS, YARN, > >>> and > >>> > > > MAPREDUCE. Specifically, the tasks include: > >>> > > > > >>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked as > >>> > > > non-blocker, update the target version to 3.5.0. > >>> > > > > >>> > > > 2. In JIRA issues where other release versions have been > released, > >>> > remove > >>> > > > the 3.4.0 entry from the fix version. > >>> > > > > >>> > > > 3. For JIRA issues with target version set to 3.4.0, fix version > >>> set to > >>> > > > 3.4.0, and status set to > >>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are > >>> not > >>> > set, > >>> > > > complete the information. > >>> > > > > >>> > > > Best Regards, > >>> > > > - Shilun Fan. > >>> > > > > >>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He > > >>> > wrote: > >>> > > > > >>> > > > > Hi All, > >>> > > > > > >>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is > >>> > created[1]. > >>> > > > > > >>> > > > > Please set the proper fix version while committing jira, 3.5.0 > >>> should > >>> > > > > be priority. If there is any blocker/critical please also > >>> backport to > >>> > > > > branch-3.4 or sync to me or RM Shilun Fan. > >>> > > > > > >>> > > > > Shilun will start 3.4.0 RC1 voting in the next few days. > >>> > > > > > >>> > > > > Thanks. > >>> > > > > > >>> > > > > Best Regards, > >>> > > > > - He Xiaoqiao > >>> > > > > > >>> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4 > >>> > > > > > >>> > > > > >>> > > >>> > - > >>> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org > >>> > For additional commands, e-mail: private-h...@hadoop.apache.org > >>> > > >>> > > >>> > >> >
Re: Resources for understanding Hadoop
Please go through the following https://github.com/apache/hadoop/blob/trunk/BUILDING.txt and a specific command to generate the distribution which can be run after your changes. mvn package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true Hope this helps. On Thu, Jun 23, 2022 at 9:41 PM Rahul Bhardwaj wrote: > I am following this wiki > <https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html> > to > build and run hadoop locally in pseudo-dsitributed mode. But I am unable to > figure out how to build my changes and generate similar binaries so that I > can test my changes locally. Is there some documentation on how to do this? > > On Mon, 13 Jun 2022 at 00:26, Brahma Reddy Battula > wrote: > >> Hi Rahul, >> >> Welcome to hadoop world. >> >> Apart from the gautham mentioned, you can check the following also. >> https://livebook.manning.com/book/hadoop-in-action/part-1/ >> >> Go through the following wiki for contributions >> https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute >> >> >> Please subscribe to the hadoop mailing list[1], and shoot your queries >> there from next time. >> >> >> >> >> >> 1. https://hadoop.apache.org/mailing_lists.html >> >> On Sun, Jun 12, 2022 at 10:42 PM Gautham Banasandra >> wrote: >> >>> Hi Rahul, >>> >>> I was looking for something more detailed and low-level like how the code >>> > for the various services in HDFS is organized, entrypoints etc. >>> >>> I found this book useful to get a good idea of Hadoop in general - Apache >>> Hadoop™ YARN: Moving beyond MapReduce and Batch Processing with Apache >>> Hadoop™ 2 [Book] (oreilly.com) >>> < >>> https://www.oreilly.com/library/view/apache-hadooptm-yarn/9780133441925/ >>> >. >>> >>> In my opinion, you get into Open Source contributions by just doing so. >>> You >>> don't have to know HDFS in detail to start contributing to it. Now that >>> you've gone through the Hadoop documentation, try setting up Hadoop in >>> pseudo-distributed mode. If you notice any glitch, try fixing it and send >>> out a PR. You never know what issue you'll find. I ran into this when I >>> tried compiling Hadoop on Windows - [HDFS-15385] Upgrade boost library to >>> 1.72 - ASF JIRA (apache.org) >>> <https://issues.apache.org/jira/browse/HDFS-15385> (And yes, this was my >>> first PR to Hadoop). Then use Docker and set up the Hadoop cluster with >>> multiple nodes. Once you're able to do this, try browsing >>> issues.apache.org >>> and you'll find tons of issues that you can work on. There's always so >>> much >>> work to do in Open Source and the thing that I like the most is that >>> "there's no deadline on anything" :) So, you can really work on some >>> awesome stuff, own it, perfect it and share it with the world. >>> >>> Best of luck. >>> >>> Thanks, >>> --Gautham >>> >>> On Sun, 12 Jun 2022 at 16:34, Rahul Bhardwaj >>> wrote: >>> >>> > Hi all, >>> > I am a newbie wanting to start contributing to the hadoop ecosystem. I >>> want >>> > to start by contributing to HDFS and was looking for resources to >>> > understand the architecture and I just found this - >>> > >>> > >>> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html >>> > which is a fairly high level documentation. I was looking for something >>> > more detailed and low-level like how the code for the various services >>> in >>> > HDFS is organized, entrypoints etc. Can someone point me to such >>> resources? >>> > Also is there a slack workspace for such discussions? Not sure if this >>> > mailing list is the right forum for such doubts. >>> > >>> >> >> >> -- >> >> >> >> --Brahma Reddy Battula >> > -- --Brahma Reddy Battula
Re: Resources for understanding Hadoop
Hi Rahul, Welcome to hadoop world. Apart from the gautham mentioned, you can check the following also. https://livebook.manning.com/book/hadoop-in-action/part-1/ Go through the following wiki for contributions https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute Please subscribe to the hadoop mailing list[1], and shoot your queries there from next time. 1. https://hadoop.apache.org/mailing_lists.html On Sun, Jun 12, 2022 at 10:42 PM Gautham Banasandra wrote: > Hi Rahul, > > I was looking for something more detailed and low-level like how the code > > for the various services in HDFS is organized, entrypoints etc. > > I found this book useful to get a good idea of Hadoop in general - Apache > Hadoop™ YARN: Moving beyond MapReduce and Batch Processing with Apache > Hadoop™ 2 [Book] (oreilly.com) > <https://www.oreilly.com/library/view/apache-hadooptm-yarn/9780133441925/ > >. > > In my opinion, you get into Open Source contributions by just doing so. You > don't have to know HDFS in detail to start contributing to it. Now that > you've gone through the Hadoop documentation, try setting up Hadoop in > pseudo-distributed mode. If you notice any glitch, try fixing it and send > out a PR. You never know what issue you'll find. I ran into this when I > tried compiling Hadoop on Windows - [HDFS-15385] Upgrade boost library to > 1.72 - ASF JIRA (apache.org) > <https://issues.apache.org/jira/browse/HDFS-15385> (And yes, this was my > first PR to Hadoop). Then use Docker and set up the Hadoop cluster with > multiple nodes. Once you're able to do this, try browsing > issues.apache.org > and you'll find tons of issues that you can work on. There's always so much > work to do in Open Source and the thing that I like the most is that > "there's no deadline on anything" :) So, you can really work on some > awesome stuff, own it, perfect it and share it with the world. > > Best of luck. > > Thanks, > --Gautham > > On Sun, 12 Jun 2022 at 16:34, Rahul Bhardwaj > wrote: > > > Hi all, > > I am a newbie wanting to start contributing to the hadoop ecosystem. I > want > > to start by contributing to HDFS and was looking for resources to > > understand the architecture and I just found this - > > > > > https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > > which is a fairly high level documentation. I was looking for something > > more detailed and low-level like how the code for the various services in > > HDFS is organized, entrypoints etc. Can someone point me to such > resources? > > Also is there a slack workspace for such discussions? Not sure if this > > mailing list is the right forum for such doubts. > > > -- --Brahma Reddy Battula
Re: Hadoop-3.2.3 Release Update
Sorry, I was looking for some CVE’s fixes.will make it progress on this… On Tue, 11 Jan 2022 at 5:58 PM, Wei-Chiu Chuang wrote: > Is this still making progress? > > On Tue, Oct 5, 2021 at 8:45 PM Brahma Reddy Battula > wrote: > > > Hi Akira, > > > > Thanks for your email!! > > > > I am evaluating the CVE’s which needs to go for this release.. > > > > Will update soon!! > > > > > > On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka > wrote: > > > > > Hi Brahma, > > > > > > What is the release process going on? Is there any blocker for the RC? > > > > > > -Akira > > > > > > On Wed, Sep 22, 2021 at 7:37 PM Xiaoqiao He > wrote: > > > > > > > Hi Brahma, > > > > > > > > The feature 'BPServiceActor processes commands from NameNode > > > > asynchronously' has been ready for both branch-3.2 and branch-3.2.3. > > > While > > > > cherry-picking there is only minor conflict, So I checked in > directly. > > > BTW, > > > > run some unit tests and build pseudo cluster to verify, it seems to > > work > > > > fine. > > > > FYI. > > > > > > > > Regards, > > > > - He Xiaoqiao > > > > > > > > On Thu, Sep 16, 2021 at 10:52 PM Brahma Reddy Battula < > > bra...@apache.org > > > > > > > > wrote: > > > > > > > >> Please go ahead. Let me know any help required on review. > > > >> > > > >> On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He > > > wrote: > > > >> > > > >>> Hi Brahma, > > > >>> > > > >>> I plan to involve HDFS-14997 and related JIRAs if possible. I have > > > >>> resolved the conflict and verified them locally. > > > >>> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113. > > > >>> I would like to hear some more response that if we have enough time > > to > > > >>> wait for it to be ready. > > > >>> Thanks. > > > >>> > > > >>> Best Regards, > > > >>> - He Xiaoqiao > > > >>> > > > >>> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He > > > wrote: > > > >>> > > > >>>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. > FYI. > > > >>>> > > > >>>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula < > > > bra...@apache.org> > > > >>>> wrote: > > > >>>> > > > >>>>> Hi All, > > > >>>>> > > > >>>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly > > > this > > > >>>>> can > > > >>>>> be done by this week,then I will try to create the RC next if > there > > > is > > > >>>>> no > > > >>>>> objection. > > > >>>>> > > > >>>>> https://issues.apache.org/jira/browse/HDFS-15160 > > > >>>>> > > > >>>>> > > > >>>>> > > > >>>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula < > > > >>>>> bra...@apache.org> > > > >>>>> wrote: > > > >>>>> > > > >>>>> > @Akira Ajisaka and @Masatake Iwasaki > > > >>>>> > > > > >>>>> > Looks all are build related issues when you try with bigtop. We > > can > > > >>>>> > discuss and prioritize this.. Will connect with you guys. > > > >>>>> > > > > >>>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki < > > > >>>>> > iwasak...@oss.nttdata.co.jp> wrote: > > > >>>>> > > > > >>>>> >> >> - > > > >>>>> >> > > > >>>>> > > > > > > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff > > > >>>>> >> > > > > >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14. > > > >>>>> >> > we do n
Re: Hadoop-3.2.3 Release Update
Hi Akira, Thanks for your email!! I am evaluating the CVE’s which needs to go for this release.. Will update soon!! On Tue, 5 Oct 2021 at 1:46 PM, Akira Ajisaka wrote: > Hi Brahma, > > What is the release process going on? Is there any blocker for the RC? > > -Akira > > On Wed, Sep 22, 2021 at 7:37 PM Xiaoqiao He wrote: > > > Hi Brahma, > > > > The feature 'BPServiceActor processes commands from NameNode > > asynchronously' has been ready for both branch-3.2 and branch-3.2.3. > While > > cherry-picking there is only minor conflict, So I checked in directly. > BTW, > > run some unit tests and build pseudo cluster to verify, it seems to work > > fine. > > FYI. > > > > Regards, > > - He Xiaoqiao > > > > On Thu, Sep 16, 2021 at 10:52 PM Brahma Reddy Battula > > > wrote: > > > >> Please go ahead. Let me know any help required on review. > >> > >> On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He > wrote: > >> > >>> Hi Brahma, > >>> > >>> I plan to involve HDFS-14997 and related JIRAs if possible. I have > >>> resolved the conflict and verified them locally. > >>> It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113. > >>> I would like to hear some more response that if we have enough time to > >>> wait for it to be ready. > >>> Thanks. > >>> > >>> Best Regards, > >>> - He Xiaoqiao > >>> > >>> On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He > wrote: > >>> > >>>> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. FYI. > >>>> > >>>> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula < > bra...@apache.org> > >>>> wrote: > >>>> > >>>>> Hi All, > >>>>> > >>>>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly > this > >>>>> can > >>>>> be done by this week,then I will try to create the RC next if there > is > >>>>> no > >>>>> objection. > >>>>> > >>>>> https://issues.apache.org/jira/browse/HDFS-15160 > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula < > >>>>> bra...@apache.org> > >>>>> wrote: > >>>>> > >>>>> > @Akira Ajisaka and @Masatake Iwasaki > >>>>> > > >>>>> > Looks all are build related issues when you try with bigtop. We can > >>>>> > discuss and prioritize this.. Will connect with you guys. > >>>>> > > >>>>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki < > >>>>> > iwasak...@oss.nttdata.co.jp> wrote: > >>>>> > > >>>>> >> >> - > >>>>> >> > >>>>> > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff > >>>>> >> > > >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14. > >>>>> >> > we do not see the issue usually since branch-3.2 uses > >>>>> zooekeper-3.4.13, > >>>>> >> > while it would be harmless to add the exclusion even for > >>>>> >> zooekeeper-3.4.13. > >>>>> >> > >>>>> >> I filed HADOOP-17849 for this. > >>>>> >> > >>>>> >> On 2021/08/16 12:02, Masatake Iwasaki wrote: > >>>>> >> > Thanks for bringing this up, Akira. Let me explain some > >>>>> background. > >>>>> >> > > >>>>> >> > > >>>>> >> >> - > >>>>> >> > >>>>> > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff > >>>>> >> > > >>>>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14. > >>>>> >> > we do not see the issue usually since branch-3.2 uses > >>>>> zooekeper-3.4.13, > >>>>> >> > while it would be harmless to add the exclusion even for > >>>>> >> zooekee
[jira] [Created] (HADOOP-17917) Backport HADOOP-15993 to branch-3.2 which Address CVE-2014-4611
Brahma Reddy Battula created HADOOP-17917: - Summary: Backport HADOOP-15993 to branch-3.2 which Address CVE-2014-4611 Key: HADOOP-17917 URL: https://issues.apache.org/jira/browse/HADOOP-17917 Project: Hadoop Common Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Now the version is 0.8.2.1 and it has net.jpountz.lz4:lz4:1.2.0 dependency, which is vulnerable. ([https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2014-4611]) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Hadoop-3.2.3 Release Update
Please go ahead. Let me know any help required on review. On Tue, Sep 14, 2021 at 6:57 PM Xiaoqiao He wrote: > Hi Brahma, > > I plan to involve HDFS-14997 and related JIRAs if possible. I have > resolved the conflict and verified them locally. > It will include: HDFS-14997 HDFS-15075 HDFS-15651 HDFS-15113. > I would like to hear some more response that if we have enough time to > wait for it to be ready. > Thanks. > > Best Regards, > - He Xiaoqiao > > On Tue, Sep 14, 2021 at 3:39 PM Xiaoqiao He wrote: > >> Hi Brahma, HDFS-15160 has checked in branch-3.2 & branch-3.2.3. FYI. >> >> On Tue, Sep 14, 2021 at 3:52 AM Brahma Reddy Battula >> wrote: >> >>> Hi All, >>> >>> Waiting for the following jira to commit to hadoop-3.2.3 , mostly this >>> can >>> be done by this week,then I will try to create the RC next if there is no >>> objection. >>> >>> https://issues.apache.org/jira/browse/HDFS-15160 >>> >>> >>> >>> On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula >>> wrote: >>> >>> > @Akira Ajisaka and @Masatake Iwasaki >>> > >>> > Looks all are build related issues when you try with bigtop. We can >>> > discuss and prioritize this.. Will connect with you guys. >>> > >>> > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki < >>> > iwasak...@oss.nttdata.co.jp> wrote: >>> > >>> >> >> - >>> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff >>> >> > >>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14. >>> >> > we do not see the issue usually since branch-3.2 uses >>> zooekeper-3.4.13, >>> >> > while it would be harmless to add the exclusion even for >>> >> zooekeeper-3.4.13. >>> >> >>> >> I filed HADOOP-17849 for this. >>> >> >>> >> On 2021/08/16 12:02, Masatake Iwasaki wrote: >>> >> > Thanks for bringing this up, Akira. Let me explain some background. >>> >> > >>> >> > >>> >> >> - >>> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff >>> >> > >>> >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14. >>> >> > we do not see the issue usually since branch-3.2 uses >>> zooekeper-3.4.13, >>> >> > while it would be harmless to add the exclusion even for >>> >> zooekeeper-3.4.13. >>> >> > >>> >> > >>> >> >> - >>> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff >>> >> >> - >>> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff >>> >> >> - >>> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff >>> >> > >>> >> > These are relevant to directory structure used by Bigtop package. >>> >> > If the fix does not break the tarball dist, >>> >> > it would be nice to have these on Hadoop too. >>> >> > >>> >> > >>> >> >> - >>> >> >>> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff >>> >> > >>> >> > This is for aarch64 and ppe64le lacking required phantomjs. >>> >> > It is only acceptable for Bigtop not running tests of YARN-UI2 on >>> >> packaging. >>> >> > Hadoop needs the phantomjs for testing YARN-UI2. >>> >> > >>> >> > >>> >> > On 2021/08/16 2:59, Brahma Reddy Battula wrote: >>> >> >> Thanks, I'll check them out. >>> >> >> >>> >> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka < >>> aajis...@apache.org> >>> >> wrote: >>> >> >> >>> >> >>> Thanks Brahma for cutting branch-3.2.3. >>> >> >>> >>> >> >>> In Apache Bigtop, there are some patches ap
Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker
Thanks all for attending the session, Hope we can have some more Good sessions.. Please find the recordings. https://drive.google.com/drive/u/1/folders/1mtIEUfnEUtil65U0nGGeJNAUYVsPpktt On Wed, Sep 8, 2021 at 7:37 PM Brahma Reddy Battula wrote: > Hi Sean, > > Pacific Time 9th, Sept 9:30 AM….. > > Please check the following link relevant time zone.. > > Time zone :: https://t.co/JmTN6zK8Ga?amp=1 > > > > On Wed, 8 Sep 2021 at 7:12 PM, Sean Busbey wrote: > >> Hi Brahma! >> >> Thanks for organizing this. What’s the timezone for the 10p - midnight? >> Pacific Time? >> >> > On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula >> wrote: >> > >> > Hi All, >> > >> > Updated the meeting to record the session.. Please use the following >> link >> > to attend the conference tomorrow. >> > >> > >> > Uber's story on running Apache Hadoop deployment in Docker >> > September 9, 2021, 10:00pm – September 10, 2021, 12:00am · >> > Google Meet joining info >> > *Video call link: https://meet.google.com/few-wppc-xoa >> > <https://meet.google.com/few-wppc-xoa>* >> > Or dial: (US) +1 443-424-3811 PIN: 384 713 518# >> > More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915 >> > >> > On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula < >> bra...@apache.org> >> > wrote: >> > >> >> Hi All, >> >> >> >> Happy to announce the following conference. Block your calendar and >> make >> >> yourself available. >> >> Thanks Mithun and team for accepting. >> >> >> >> >> >> *Topic:*Uber's story on running Apache Hadoop deployment in Docker >> >> >> >> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time* >> >> >> >> *Meeting Link : *Google Meet joining info >> >> Video call link: >> >> https://meet.google.com/akk-qmzy-qsu >> >> >> >> * Note: *Created Teams meeting also as a >> backup >> >> [2]. >> >> >> >> *High level Agenda: * >> >> >> >> - Importance of Hadoop Cluster Management (3-5 min) >> >> - Introduction about problem space >> >> - Challenges in this space and why it is important to solve them >> >> - Evolution of Uber Hadoop Cluster Management (10-15 min) >> >> - How our strategy evolved over time since Hadoop was set up at Uber >> >> - Key learnings from the evolution over the years >> >> - Our current approach today (15-20min) >> >> - Key learnings from this major overhaul >> >> - Current challenges that we are working on >> >> - Q&A (20 min) >> >> >> >> >> >> Aiming for 20 min Q&A assuming that there will be more questions based >> on >> >> the blog that was published(1) + presentation at the session. >> >> >> >> >> >> >> >> >> >> >> --- >> >> 1) https://eng.uber.com/hadoop-container-blog/ >> >> < >> https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/&sa=D&source=calendar&usd=2&usg=AOvVaw3OutuZVQBQnkhUd4fxaM22 >> > >> >> 2) >> >> >> >> *Join on your computer or mobile app* >> >> >> >> Click here to join the meeting >> >> < >> https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d >> > >> >> >> >> *Join with a video conferencing device* >> >> >> >> 967904...@t.plcm.vc >> >> >> >> Video Conference ID: 117 464 626 6 >> >> >> >> Alternate VTC instructions >> >> <https://dialin.plcm.vc/teams/?key=967904974&conf=1174646266> >> >> >> >> *Or call in (audio only)* >> >> >> >> +1 213-204-8714,,273569658# <+12132048714,,273569658#> United States, >> >> Los Angeles >> >> >> >> (833) 827-4491,,273569658# <8338274491,,273569658#> United States >> >> (Toll-free) >> >> >> >> Phone Conference ID: 273 569 658# >> >> >> >> Find a local number >> >> < >> https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658 >> > >> >> | Reset PIN <https://mysettings.lync.com/pstnconferencing> >> >> >> >> >> >> >> >> -- Brahma Reddy Battula >> >> >> > >> > >> > -- >> > >> > >> > >> > --Brahma Reddy Battula >> >> >> -- > > > > --Brahma Reddy Battula > -- --Brahma Reddy Battula
Re: Hadoop-3.2.3 Release Update
Hi All, Waiting for the following jira to commit to hadoop-3.2.3 , mostly this can be done by this week,then I will try to create the RC next if there is no objection. https://issues.apache.org/jira/browse/HDFS-15160 On Mon, Aug 16, 2021 at 2:22 PM Brahma Reddy Battula wrote: > @Akira Ajisaka and @Masatake Iwasaki > > Looks all are build related issues when you try with bigtop. We can > discuss and prioritize this.. Will connect with you guys. > > On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki < > iwasak...@oss.nttdata.co.jp> wrote: > >> >> - >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff >> > >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14. >> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13, >> > while it would be harmless to add the exclusion even for >> zooekeeper-3.4.13. >> >> I filed HADOOP-17849 for this. >> >> On 2021/08/16 12:02, Masatake Iwasaki wrote: >> > Thanks for bringing this up, Akira. Let me explain some background. >> > >> > >> >> - >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff >> > >> > This is for building hadoop-3.2.2 against zookeeper-3.4.14. >> > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13, >> > while it would be harmless to add the exclusion even for >> zooekeeper-3.4.13. >> > >> > >> >> - >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff >> >> - >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff >> >> - >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff >> > >> > These are relevant to directory structure used by Bigtop package. >> > If the fix does not break the tarball dist, >> > it would be nice to have these on Hadoop too. >> > >> > >> >> - >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff >> > >> > This is for aarch64 and ppe64le lacking required phantomjs. >> > It is only acceptable for Bigtop not running tests of YARN-UI2 on >> packaging. >> > Hadoop needs the phantomjs for testing YARN-UI2. >> > >> > >> > On 2021/08/16 2:59, Brahma Reddy Battula wrote: >> >> Thanks, I'll check them out. >> >> >> >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka >> wrote: >> >> >> >>> Thanks Brahma for cutting branch-3.2.3. >> >>> >> >>> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2. >> >>> >> >>> >> https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop >> >>> >> >>> In these patches, how about backporting the following issues to >> >>> branch-3.2 and branch-3.2.3? >> >>> - HADOOP-14922 >> >>> - HADOOP-15939 >> >>> - HADOOP-17569 >> >>> >> >>> In addition, there are some patches that don't have JIRA issue ID. >> >>> Maybe we need to create JIRAs and fix those. >> >>> - >> >>> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff >> >>> - >> >>> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff >> >>> - >> >>> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff >> >>> - >> >>> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff >> >>> - >> >>> >> https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff >> >>> >> >>> Thanks and regards, >> >>> Akira >> >>> >> >>> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He >> >>> wrote: >> >>>> >> >>>> Thanks Brahma for initiating this and making hadoop-3.2.3 release >> happen. >> >>>> >> >>>>
Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker
Hi Sean, Pacific Time 9th, Sept 9:30 AM….. Please check the following link relevant time zone.. Time zone :: https://t.co/JmTN6zK8Ga?amp=1 On Wed, 8 Sep 2021 at 7:12 PM, Sean Busbey wrote: > Hi Brahma! > > Thanks for organizing this. What’s the timezone for the 10p - midnight? > Pacific Time? > > > On Sep 8, 2021, at 1:17 AM, Brahma Reddy Battula > wrote: > > > > Hi All, > > > > Updated the meeting to record the session.. Please use the following link > > to attend the conference tomorrow. > > > > > > Uber's story on running Apache Hadoop deployment in Docker > > September 9, 2021, 10:00pm – September 10, 2021, 12:00am · > > Google Meet joining info > > *Video call link: https://meet.google.com/few-wppc-xoa > > <https://meet.google.com/few-wppc-xoa>* > > Or dial: (US) +1 443-424-3811 PIN: 384 713 518# > > More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915 > > > > On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula > > > wrote: > > > >> Hi All, > >> > >> Happy to announce the following conference. Block your calendar and make > >> yourself available. > >> Thanks Mithun and team for accepting. > >> > >> > >> *Topic:*Uber's story on running Apache Hadoop deployment in Docker > >> > >> *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time* > >> > >> *Meeting Link : *Google Meet joining info > >> Video call link: > >> https://meet.google.com/akk-qmzy-qsu > >> > >> * Note: *Created Teams meeting also as a backup > >> [2]. > >> > >> *High level Agenda: * > >> > >> - Importance of Hadoop Cluster Management (3-5 min) > >> - Introduction about problem space > >> - Challenges in this space and why it is important to solve them > >> - Evolution of Uber Hadoop Cluster Management (10-15 min) > >> - How our strategy evolved over time since Hadoop was set up at Uber > >> - Key learnings from the evolution over the years > >> - Our current approach today (15-20min) > >> - Key learnings from this major overhaul > >> - Current challenges that we are working on > >> - Q&A (20 min) > >> > >> > >> Aiming for 20 min Q&A assuming that there will be more questions based > on > >> the blog that was published(1) + presentation at the session. > >> > >> > >> > >> > >> > --- > >> 1) https://eng.uber.com/hadoop-container-blog/ > >> < > https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/&sa=D&source=calendar&usd=2&usg=AOvVaw3OutuZVQBQnkhUd4fxaM22 > > > >> 2) > >> > >> *Join on your computer or mobile app* > >> > >> Click here to join the meeting > >> < > https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d > > > >> > >> *Join with a video conferencing device* > >> > >> 967904...@t.plcm.vc > >> > >> Video Conference ID: 117 464 626 6 > >> > >> Alternate VTC instructions > >> <https://dialin.plcm.vc/teams/?key=967904974&conf=1174646266> > >> > >> *Or call in (audio only)* > >> > >> +1 213-204-8714,,273569658# <+12132048714,,273569658#> United States, > >> Los Angeles > >> > >> (833) 827-4491,,273569658# <8338274491,,273569658#> United States > >> (Toll-free) > >> > >> Phone Conference ID: 273 569 658# > >> > >> Find a local number > >> < > https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658 > > > >> | Reset PIN <https://mysettings.lync.com/pstnconferencing> > >> > >> > >> > >> -- Brahma Reddy Battula > >> > > > > > > -- > > > > > > > > --Brahma Reddy Battula > > > -- --Brahma Reddy Battula
Re: [Conference] Uber's story on running Apache Hadoop deployment in Docker
Hi All, Updated the meeting to record the session.. Please use the following link to attend the conference tomorrow. Uber's story on running Apache Hadoop deployment in Docker September 9, 2021, 10:00pm – September 10, 2021, 12:00am · Google Meet joining info *Video call link: https://meet.google.com/few-wppc-xoa <https://meet.google.com/few-wppc-xoa>* Or dial: (US) +1 443-424-3811 PIN: 384 713 518# More phone numbers: https://tel.meet/few-wppc-xoa?pin=7430296860915 On Fri, Aug 27, 2021 at 12:43 PM Brahma Reddy Battula wrote: > Hi All, > > Happy to announce the following conference. Block your calendar and make > yourself available. > Thanks Mithun and team for accepting. > > > *Topic:*Uber's story on running Apache Hadoop deployment in Docker > > *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time* > > *Meeting Link : *Google Meet joining info >Video call link: > https://meet.google.com/akk-qmzy-qsu > > * Note: *Created Teams meeting also as a backup > [2]. > > *High level Agenda: * > >- Importance of Hadoop Cluster Management (3-5 min) >- Introduction about problem space >- Challenges in this space and why it is important to solve them >- Evolution of Uber Hadoop Cluster Management (10-15 min) >- How our strategy evolved over time since Hadoop was set up at Uber >- Key learnings from the evolution over the years >- Our current approach today (15-20min) >- Key learnings from this major overhaul >- Current challenges that we are working on >- Q&A (20 min) > > > Aiming for 20 min Q&A assuming that there will be more questions based on > the blog that was published(1) + presentation at the session. > > > > > --- > 1) https://eng.uber.com/hadoop-container-blog/ > <https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/&sa=D&source=calendar&usd=2&usg=AOvVaw3OutuZVQBQnkhUd4fxaM22> > 2) > > *Join on your computer or mobile app* > > Click here to join the meeting > <https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d> > > *Join with a video conferencing device* > > 967904...@t.plcm.vc > > Video Conference ID: 117 464 626 6 > > Alternate VTC instructions > <https://dialin.plcm.vc/teams/?key=967904974&conf=1174646266> > > *Or call in (audio only)* > > +1 213-204-8714,,273569658# <+12132048714,,273569658#> United States, > Los Angeles > > (833) 827-4491,,273569658# <8338274491,,273569658#> United States > (Toll-free) > > Phone Conference ID: 273 569 658# > > Find a local number > <https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658> > | Reset PIN <https://mysettings.lync.com/pstnconferencing> > > > > -- Brahma Reddy Battula > -- --Brahma Reddy Battula
[Conference] Uber's story on running Apache Hadoop deployment in Docker
Hi All, Happy to announce the following conference. Block your calendar and make yourself available. Thanks Mithun and team for accepting. *Topic:*Uber's story on running Apache Hadoop deployment in Docker *Date : *Thu 2021-09-09 09:30 – 11:30 *Pacific Time* *Meeting Link : *Google Meet joining info Video call link: https://meet.google.com/akk-qmzy-qsu * Note: *Created Teams meeting also as a backup [2]. *High level Agenda: * - Importance of Hadoop Cluster Management (3-5 min) - Introduction about problem space - Challenges in this space and why it is important to solve them - Evolution of Uber Hadoop Cluster Management (10-15 min) - How our strategy evolved over time since Hadoop was set up at Uber - Key learnings from the evolution over the years - Our current approach today (15-20min) - Key learnings from this major overhaul - Current challenges that we are working on - Q&A (20 min) Aiming for 20 min Q&A assuming that there will be more questions based on the blog that was published(1) + presentation at the session. --- 1) https://eng.uber.com/hadoop-container-blog/ <https://www.google.com/url?q=https://eng.uber.com/hadoop-container-blog/&sa=D&source=calendar&usd=2&usg=AOvVaw3OutuZVQBQnkhUd4fxaM22> 2) *Join on your computer or mobile app* Click here to join the meeting <https://teams.microsoft.com/l/meetup-join/19%3ameeting_NmI2NTNhNGItMTRmZS00ZTJhLWFjNGYtMTg0M2M0ZGRkMGMz%40thread.v2/0?context=%7b%22Tid%22%3a%2238305e12-e15d-4ee8-88b9-c4db1c477d76%22%2c%22Oid%22%3a%229620d90d-11aa-4a4c-b54e-9f3efd907308%22%7d> *Join with a video conferencing device* 967904...@t.plcm.vc Video Conference ID: 117 464 626 6 Alternate VTC instructions <https://dialin.plcm.vc/teams/?key=967904974&conf=1174646266> *Or call in (audio only)* +1 213-204-8714,,273569658# <+12132048714,,273569658#> United States, Los Angeles (833) 827-4491,,273569658# <8338274491,,273569658#> United States (Toll-free) Phone Conference ID: 273 569 658# Find a local number <https://dialin.teams.microsoft.com/354a6938-11bf-45f7-97f9-0a77a564c966?id=273569658> | Reset PIN <https://mysettings.lync.com/pstnconferencing> -- Brahma Reddy Battula
Re: Hadoop-3.2.3 Release Update
@Akira Ajisaka and @Masatake Iwasaki Looks all are build related issues when you try with bigtop. We can discuss and prioritize this.. Will connect with you guys. On Mon, Aug 16, 2021 at 1:43 PM Masatake Iwasaki < iwasak...@oss.nttdata.co.jp> wrote: > >> - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff > > > > This is for building hadoop-3.2.2 against zookeeper-3.4.14. > > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13, > > while it would be harmless to add the exclusion even for > zooekeeper-3.4.13. > > I filed HADOOP-17849 for this. > > On 2021/08/16 12:02, Masatake Iwasaki wrote: > > Thanks for bringing this up, Akira. Let me explain some background. > > > > > >> - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff > > > > This is for building hadoop-3.2.2 against zookeeper-3.4.14. > > we do not see the issue usually since branch-3.2 uses zooekeper-3.4.13, > > while it would be harmless to add the exclusion even for > zooekeeper-3.4.13. > > > > > >> - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff > >> - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff > >> - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff > > > > These are relevant to directory structure used by Bigtop package. > > If the fix does not break the tarball dist, > > it would be nice to have these on Hadoop too. > > > > > >> - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff > > > > This is for aarch64 and ppe64le lacking required phantomjs. > > It is only acceptable for Bigtop not running tests of YARN-UI2 on > packaging. > > Hadoop needs the phantomjs for testing YARN-UI2. > > > > > > On 2021/08/16 2:59, Brahma Reddy Battula wrote: > >> Thanks, I'll check them out. > >> > >> On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka > wrote: > >> > >>> Thanks Brahma for cutting branch-3.2.3. > >>> > >>> In Apache Bigtop, there are some patches applied to Hadoop 3.2.2. > >>> > >>> > https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop > >>> > >>> In these patches, how about backporting the following issues to > >>> branch-3.2 and branch-3.2.3? > >>> - HADOOP-14922 > >>> - HADOOP-15939 > >>> - HADOOP-17569 > >>> > >>> In addition, there are some patches that don't have JIRA issue ID. > >>> Maybe we need to create JIRAs and fix those. > >>> - > >>> > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff > >>> - > >>> > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff > >>> - > >>> > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff > >>> - > >>> > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff > >>> - > >>> > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff > >>> > >>> Thanks and regards, > >>> Akira > >>> > >>> On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He > >>> wrote: > >>>> > >>>> Thanks Brahma for initiating this and making hadoop-3.2.3 release > happen. > >>>> > >>>> I would like to validate the HBase project (both the latest release > and > >>>> trunk branch). > >>>> Chao Sun will validate the Spark Project (Got in touch with Chao > >>> already). > >>>> once RC is out. > >>>> > >>>> Thanks and Regards, > >>>> - He Xiaoqiao > >>>> > >>>> > >>>> On Tue, Aug 10, 2021 at 5:54 PM Brahma Reddy Battula < > bra...@apache.org> > >>>> wrote: > >>>> > >>>>> Hi All, &g
Re: Hadoop-3.2.3 Release Update
Thanks, I'll check them out. On Sun, 15 Aug 2021 at 11:22 PM, Akira Ajisaka wrote: > Thanks Brahma for cutting branch-3.2.3. > > In Apache Bigtop, there are some patches applied to Hadoop 3.2.2. > > https://github.com/apache/bigtop/tree/master/bigtop-packages/src/common/hadoop > > In these patches, how about backporting the following issues to > branch-3.2 and branch-3.2.3? > - HADOOP-14922 > - HADOOP-15939 > - HADOOP-17569 > > In addition, there are some patches that don't have JIRA issue ID. > Maybe we need to create JIRAs and fix those. > - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch2-exclude-spotbugs-annotations.diff > - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch3-fix-broken-dir-detection.diff > - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch5-fix-kms-shellprofile.diff > - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch6-fix-httpfs-sh.diff > - > https://github.com/apache/bigtop/blob/master/bigtop-packages/src/common/hadoop/patch7-remove-phantomjs-in-yarn-ui.diff > > Thanks and regards, > Akira > > On Wed, Aug 11, 2021 at 12:31 PM Xiaoqiao He > wrote: > > > > Thanks Brahma for initiating this and making hadoop-3.2.3 release happen. > > > > I would like to validate the HBase project (both the latest release and > > trunk branch). > > Chao Sun will validate the Spark Project (Got in touch with Chao > already). > > once RC is out. > > > > Thanks and Regards, > > - He Xiaoqiao > > > > > > On Tue, Aug 10, 2021 at 5:54 PM Brahma Reddy Battula > > wrote: > > > > > Hi All, > > > > > > I cut branch-3.2.3 and it is ready for release. Please commit to > > > branch-3.2.3 if any critical/blocker issues need to go. > > > > > > *This time I am thinking of having downstream projects and companies' > > > voices,let's know how this can go. * > > > > > >- Planning to check with downstream projects like Spark,HBase and > Hive > > >if they can help on validation(Or running their UT on this branch) > > >- Collecting info from Companies who are already deployed and using > the > > >branch-3.2 > > > > > > > > > so that we can make a more stable release on 3.2 (so that features > released > > > on this branch impact can be known) ,please let me know anybody from > these > > > communities who can help on this. > > > > > > > > > Planning to create RC by this month end. Any suggestions are welcome. > > > > > > > > > > > > --Brahma Reddy Battula > > > > -- --Brahma Reddy Battula
Hadoop-3.2.3 Release Update
Hi All, I cut branch-3.2.3 and it is ready for release. Please commit to branch-3.2.3 if any critical/blocker issues need to go. *This time I am thinking of having downstream projects and companies' voices,let's know how this can go. * - Planning to check with downstream projects like Spark,HBase and Hive if they can help on validation(Or running their UT on this branch) - Collecting info from Companies who are already deployed and using the branch-3.2 so that we can make a more stable release on 3.2 (so that features released on this branch impact can be known) ,please let me know anybody from these communities who can help on this. Planning to create RC by this month end. Any suggestions are welcome. --Brahma Reddy Battula
Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?
@Wei-Chiu Chuang looks this is not concluded yet... Can we move forward..? On Thu, Jul 15, 2021 at 11:09 PM Brahma Reddy Battula wrote: > > I agree with Ahmed Hussein…Jira should not be used for number generation.. > > We can always revisit the jira to see useful discussion at one place… > > @wei-chu, +1 on proposal for cleaning the PR’s.. > > > On Thu, 15 Jul 2021 at 9:15 PM, epa...@apache.org > wrote: > >> > I usually use PR comments to discuss about the patch submitted. >> My concern is that still leaves multiple places to look in order to get a >> full picture of an issue. >> -Eric >> >> On Wednesday, July 14, 2021, 7:07:30 PM CDT, Masatake Iwasaki < >> iwasak...@oss.nttdata.co.jp> wrote: >> >> > - recently, JIRA became some sort of a "number generator" with >> insufficient >> > description/details as the >> >developers and the reviewers spending more time discussing in the PR. >> >> JIRA issues contain useful information in the fields. >> We are leveraging them in development and release process. >> >> * https://yetus.apache.org/documentation/0.13.0/releasedocmaker/ >> * >> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336122 >> >> I usually use PR comments to discuss about the patch submitted. >> JIRA comments are used for background or design discussion before and >> after submitting PR. >> There would be no problem having no comment in minor/trivial JIRA issues. >> >> >> On 2021/07/14 23:50, Ahmed Hussein wrote: >> > Do you consider migrating Jira issues to Github issues? >> > >> > I am a little bit concerned that there are some committers who still >> prefer >> > Jira-precommits over GitHub PR >> > (P.S. I am not a committer). >> > >> > Their point is that Github-PR confuses them with discussions/comments >> being >> > in two places rather than one. >> > >> > Personally, I found several Github-PRs comments discussing the validity >> of >> > the feature/bug. >> > As a result: >> > - recently, JIRA became some sort of a "number generator" with >> insufficient >> > description/details as the >> >developers and the reviewers spending more time discussing in the PR. >> > - the relation between a single Jira and Github-PR is 1-to-M. In order >> to >> > find related discussions, the user may >> >need to visit every PR (that may include closed ones) >> > >> > >> > >> > On Wed, Jul 14, 2021 at 8:46 AM Steve Loughran >> >> > wrote: >> > >> >> not sure about stale PR closing; when you've a patch which is still >> pending >> >> review it's not that fun to have it closed. >> >> >> >> maybe better to have review sessions. I recall many, many years ago >> >> attempts to try and catch up with all outstanding patch reviews. >> >> >> >> >> >> >> >> >> >> On Wed, 14 Jul 2021 at 03:00, Akira Ajisaka >> wrote: >> >> >> >>> Thank you Wei-Chiu for starting the discussion, >> >>> >> >>>> 3. JIRA security >> >>> I'm +1 to use private JIRA issues to handle vulnerabilities. >> >>> >> >>>> 5. Doc update >> >>> +1, I build the document daily and it helps me fixing documents: >> >>> https://aajisaka.github.io/hadoop-document/ It's great if the latest >> >>> document is built and published by the Apache Hadoop community. >> >>> >> >>> My idea related to GitHub PR: >> >>> 1. Disable the precommit jobs for JIRA, always use GitHub PR. It saves >> >>> costs to configure and debug the precommit jobs. >> >>> https://issues.apache.org/jira/browse/HADOOP-17798 >> >>> 2. Improve the pull request template for the contributors >> >>> https://issues.apache.org/jira/browse/HADOOP-17799 >> >>> >> >>> Regards, >> >>> Akira >> >>> >> >>> On Tue, Jul 13, 2021 at 12:35 PM Wei-Chiu Chuang >> >>> wrote: >> >>>> >> >>>> I work on multiple projects and learned a bunch from those >> >> projects.There >> >>>> are nice add-ons that help with productivity. There are things we can >> >> do >> >>> to >> >>>> help us manage the proje
Re: [External Sender] Re: [DISCUSS] Hadoop 3.2.3 release
**Important update** Hadoop-3.2.3 Release progress. I cut branch-3.2.3 and it is ready for release. Please commit to branch-3.2.3 if any critical/blocker issues need to go. As previously communicated,Planning to create RC by this month end. Any suggestions are welcome. On Thu, Aug 5, 2021 at 10:15 PM Jason Wen wrote: > Hi Steve, > > Which aws-java-sdk-bundle version did you test and encounter the following > test failures? > > Thanks, > Jason > > On 8/4/21, 7:51 AM, "Steve Loughran" wrote: > > it didn't take, so don't worry about it. Test failures > > java.lang.IllegalArgumentException: Input is expected to be encoded in > multiple of 4 bytes but found: 75 > at com.amazonaws.util.Base64Codec.decode(Base64Codec.java:198) > at com.amazonaws.util.Base64.decode(Base64.java:116) > at > > com.amazonaws.services.s3.AmazonS3Client.populateSSE_C(AmazonS3Client.java:4620) > at > > com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1806) > at > > org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1603) > at > > org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$13(S3AFileSystem.java:2828) > at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109) > at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265) > at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322) > at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261) > at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236) > at > > org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2826) > at > > org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2801) > at > > org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2146) > at > org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2079) > at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2304) > at > > org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338) > at > > org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193) > > On Tue, 3 Aug 2021 at 19:41, Brahma Reddy Battula > wrote: > > > Hi Steve, > > > > Is there any jira for this ?? > > > > > > > > On Tue, 3 Aug 2021 at 4:05 PM, Steve Loughran > > > > > wrote: > > > > > I'm testing how well a backport of the latest AWS SDK goes; if all > is > > > docile there then I'll merge that. Due diligence more than anything > > > elseone of the shaded netty JARs (never used by our code) has > a CVE > > in > > > older versions > > > > > > On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka > wrote: > > > > > > > Hi Steven, > > > > > > > > Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I > > > > opened a PR to backport YARN-8990: > > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_hadoop_pull_3254&d=DwIBaQ&c=DS6PUFBBr_KiLo7Sjt3ljp5jaW5k2i9ijVXllEdOozc&r=UflFQf1BWcrVtfjfN1LUqWWh-UBP5XtRGMdcDC-0P7o&m=dXWDhhZW6SZQt-9oQONYtGPMFTrUL4225rMKK-494YA&s=tLml0Gisl-xPsd4B67BNBZtPsz1T0qBPppHYRzLLl_0&e= > > > > > > > > Thanks, > > > > Akira > > > > > > > > On Thu, Jul 29, 2021 at 10:36 AM Steven Rand < > stevenjr...@gmail.com> > > > > wrote: > > > > > > > > > > I think it would be helpful if we could include YARN-8990 and > > YARN-8992 > > > > in the 3.2.3 release. Both are important fixes which were > included in > > > > 3.2.0, but never made their way to branch-3.2, so were omitted > from > > both > > > > 3.2.1 and 3.2.2. > > > > > > > > > > Best, > > > > > Steve > > > > > > > > > > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He < > xq.he2...@gmail.com> > > > wrote: > > > > >> > > > > >> cc @dev mail-list. > > > > >> > > > > >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He < > xq.he2...@gmail.com> > > > > wrote: > > > > >> > > > > >> > Hi Brahma, > > > > >> > >
[jira] [Resolved] (HADOOP-17840) Backport HADOOP-17837 to branch-3.2
[ https://issues.apache.org/jira/browse/HADOOP-17840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula resolved HADOOP-17840. --- Fix Version/s: 3.2.3 Hadoop Flags: Reviewed Resolution: Fixed Committed to branch-3.2.3...[~bbeaudreault] thanks for your contribution. > Backport HADOOP-17837 to branch-3.2 > --- > > Key: HADOOP-17840 > URL: https://issues.apache.org/jira/browse/HADOOP-17840 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Fix For: 3.2.3 > > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-17837) Make it easier to debug UnknownHostExceptions from NetUtils.connect
[ https://issues.apache.org/jira/browse/HADOOP-17837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brahma Reddy Battula resolved HADOOP-17837. --- Hadoop Flags: Reviewed Resolution: Fixed [~bbeaudreault] thanks raising the PR. Committed to trunk and branch-3.3.. As this only test assertion, ran locally and pushed. > Make it easier to debug UnknownHostExceptions from NetUtils.connect > --- > > Key: HADOOP-17837 > URL: https://issues.apache.org/jira/browse/HADOOP-17837 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Bryan Beaudreault >Assignee: Bryan Beaudreault >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Most UnknownHostExceptions thrown throughout hadoop include a useful message, > either the hostname that was not found or some other descriptor of the > problem. The UnknownHostException thrown from NetUtils.connect only includes > the [message of the underlying > UnresolvedAddressException|https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/net/NetUtils.java#L592]. > If you take a look at the source for UnresolvedAddressException, [it only > has a no-args > constructor|https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/nio/channels/UnresolvedAddressException.html] > (java11, but same is true in other versions). So it never has a message, > meaning the UnknownHostException message is empty. > We should include the endpoint.toString() in the UnknownHostException thrown > by NetUtils.connect -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [DISCUSS] Hadoop 3.2.3 release
Hi Steve, Is there any jira for this ?? On Tue, 3 Aug 2021 at 4:05 PM, Steve Loughran wrote: > I'm testing how well a backport of the latest AWS SDK goes; if all is > docile there then I'll merge that. Due diligence more than anything > elseone of the shaded netty JARs (never used by our code) has a CVE in > older versions > > On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka wrote: > > > Hi Steven, > > > > Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I > > opened a PR to backport YARN-8990: > > https://github.com/apache/hadoop/pull/3254 > > > > Thanks, > > Akira > > > > On Thu, Jul 29, 2021 at 10:36 AM Steven Rand > > wrote: > > > > > > I think it would be helpful if we could include YARN-8990 and YARN-8992 > > in the 3.2.3 release. Both are important fixes which were included in > > 3.2.0, but never made their way to branch-3.2, so were omitted from both > > 3.2.1 and 3.2.2. > > > > > > Best, > > > Steve > > > > > > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He > wrote: > > >> > > >> cc @dev mail-list. > > >> > > >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He > > wrote: > > >> > > >> > Hi Brahma, > > >> > > > >> > I just created version 3.2.4, and changed all unresolved issues > > (target > > >> > version/s: 3.2.3) to 3.2.4 after checking both of them are not > blocker > > >> > issues. Dashboard[1] is clean now. > > >> > > > >> > Regards, > > >> > - He Xiaoqiao > > >> > > > >> > [1] > > >> > > > > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167 > > >> > > > >> > On Sun, Jul 25, 2021 at 7:45 PM Brahma Reddy Battula < > > bra...@apache.org> > > >> > wrote: > > >> > > > >> >> Hi Xiaoqiao, > > >> >> > > >> >> Thanks for creating the Dashboard, we need to change the filters > and > > >> >> target versions in the jira. > > >> >> > > >> >> On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He > > wrote: > > >> >> > > >> >>> Thanks Brahma for volunteering and driving this release plan. I > just > > >> >>> created a dashboard for 3.2.3 release[1]. > > >> >>> I would like to support for this release line if need. (cc Brahma) > > >> >>> > > >> >>> Thanks. Regards, > > >> >>> - He Xiaoqiao > > >> >>> > > >> >>> [1] > > >> >>> > > >> >>> > > > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167 > > >> >>> > > >> >>> > > >> >>> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka < > aajis...@apache.org> > > >> >>> wrote: > > >> >>> > > >> >>> > Hi Brahma, > > >> >>> > > > >> >>> > Thank you for volunteering! > > >> >>> > > > >> >>> > -Akira > > >> >>> > > > >> >>> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula < > > >> >>> bra...@apache.org> > > >> >>> > wrote: > > >> >>> > > > > >> >>> > > Hi Akira, > > >> >>> > > > > >> >>> > > Thanks for bringing this.. > > >> >>> > > > > >> >>> > > I want to drive this if nobody already plan to do this.. > > >> >>> > > > > >> >>> > > > > >> >>> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka < > > aajis...@apache.org> > > >> >>> > wrote: > > >> >>> > > > > >> >>> > > > Hi all, > > >> >>> > > > > > >> >>> > > > Hadoop 3.2.2 was released half a year ago, and now, we have > > >> >>> > > > accumulated more than 230 commits [1]. Therefore I want to > > start > > >> >>> the > > >> >>> > > > release work for 3.2.3. > >
Re: [DISCUSS] Hadoop 3.2.3 release
Update on this thread. Will cut off the branch next week. And planning to release it in the last week of next month. Please try to mark jira's which are required for this release if anything is missed or reply to this mail. On Sun, Jul 25, 2021 at 5:14 PM Brahma Reddy Battula wrote: > Hi Xiaoqiao, > > Thanks for creating the Dashboard, we need to change the filters and > target versions in the jira. > > On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He wrote: > >> Thanks Brahma for volunteering and driving this release plan. I just >> created a dashboard for 3.2.3 release[1]. >> I would like to support for this release line if need. (cc Brahma) >> >> Thanks. Regards, >> - He Xiaoqiao >> >> [1] >> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167 >> >> >> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka >> wrote: >> >> > Hi Brahma, >> > >> > Thank you for volunteering! >> > >> > -Akira >> > >> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula > > >> > wrote: >> > > >> > > Hi Akira, >> > > >> > > Thanks for bringing this.. >> > > >> > > I want to drive this if nobody already plan to do this.. >> > > >> > > >> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka >> > wrote: >> > > >> > > > Hi all, >> > > > >> > > > Hadoop 3.2.2 was released half a year ago, and now, we have >> > > > accumulated more than 230 commits [1]. Therefore I want to start the >> > > > release work for 3.2.3. >> > > > >> > > > There is one blocker for 3.2.3 [2]. >> > > > - https://issues.apache.org/jira/browse/HDFS-12920 >> > > > >> > > > Is there anyone who would volunteer to be the 3.2.3 release manager? >> > > > Are there any other blockers? If any, please file an issue, raise >> the >> > > > blocker, and add the target version. >> > > > >> > > > [1] >> > > > >> > >> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3 >> > > > [2] >> > > > >> > >> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3 >> > > > >> > > > Regards, >> > > > Akira >> > > > >> > > > >> - >> > > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org >> > > > For additional commands, e-mail: >> mapreduce-dev-h...@hadoop.apache.org >> > > > >> > > > -- >> > > >> > > >> > > >> > > --Brahma Reddy Battula >> > >> > - >> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> > >> > >> > > > -- > > > > --Brahma Reddy Battula > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 3.2.3 release
Hi Xiaoqiao, Thanks for creating the Dashboard, we need to change the filters and target versions in the jira. On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He wrote: > Thanks Brahma for volunteering and driving this release plan. I just > created a dashboard for 3.2.3 release[1]. > I would like to support for this release line if need. (cc Brahma) > > Thanks. Regards, > - He Xiaoqiao > > [1] > https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167 > > > On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka wrote: > > > Hi Brahma, > > > > Thank you for volunteering! > > > > -Akira > > > > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula > > wrote: > > > > > > Hi Akira, > > > > > > Thanks for bringing this.. > > > > > > I want to drive this if nobody already plan to do this.. > > > > > > > > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka > > wrote: > > > > > > > Hi all, > > > > > > > > Hadoop 3.2.2 was released half a year ago, and now, we have > > > > accumulated more than 230 commits [1]. Therefore I want to start the > > > > release work for 3.2.3. > > > > > > > > There is one blocker for 3.2.3 [2]. > > > > - https://issues.apache.org/jira/browse/HDFS-12920 > > > > > > > > Is there anyone who would volunteer to be the 3.2.3 release manager? > > > > Are there any other blockers? If any, please file an issue, raise the > > > > blocker, and add the target version. > > > > > > > > [1] > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3 > > > > [2] > > > > > > > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3 > > > > > > > > Regards, > > > > Akira > > > > > > > > ----- > > > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > > > > For additional commands, e-mail: > mapreduce-dev-h...@hadoop.apache.org > > > > > > > > -- > > > > > > > > > > > > --Brahma Reddy Battula > > > > - > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > > > > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 3.2.3 release
Hi Akira, Thanks for bringing this.. I want to drive this if nobody already plan to do this.. On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka wrote: > Hi all, > > Hadoop 3.2.2 was released half a year ago, and now, we have > accumulated more than 230 commits [1]. Therefore I want to start the > release work for 3.2.3. > > There is one blocker for 3.2.3 [2]. > - https://issues.apache.org/jira/browse/HDFS-12920 > > Is there anyone who would volunteer to be the 3.2.3 release manager? > Are there any other blockers? If any, please file an issue, raise the > blocker, and add the target version. > > [1] > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3 > [2] > https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3 > > Regards, > Akira > > - > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org > > -- --Brahma Reddy Battula
Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?
gt; that > >>>> can perform static analysis, catching potential issues. For example, > >>> Ozone > >>>> adds SonarQube to post-commit, and exports the report to SonarCloud. > >>> Other > >>>> add-ons are available to scan for docker images, vulnerabilities > scans. > >>>> > >>>> 3. JIRA security > >>>> It is possible to set up security level (public/private) in JIRA. This > >>> can > >>>> be used to track vulnerability issues and be made only visible to > >>>> committers. Example: INFRA-15258 > >>>> <https://issues.apache.org/jira/browse/INFRA-15258> > >>>> > >>>> 4. New JIRA fields > >>>> It's possible to add new fields. For example, we can add a "Reviewer" > >>>> field, which could help improve the attention to issues. > >>>> > >>>> 5. Doc update > >>>> It is possible to set up automation such that the doc on the Hadoop > >>> website > >>>> is refreshed for every commit, providing the latest doc to the public. > >>>> > >>>> 6. Webhook > >>>> It's possible to set up webhook such that every commit in GitHub sends > >> a > >>>> notification to the ASF slack. It can be used for other kinds of > >>>> automation. Sky's the limit. > >>>> > >>>> Thoughts? What else can do we? > >>> > >>> - > >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org > >>> > >>> > >> > > > > > > - > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org > > -- --Brahma Reddy Battula
[jira] [Created] (HADOOP-17800) CLONE - Uber-JIRA: Hadoop should support IPv6
Brahma Reddy Battula created HADOOP-17800: - Summary: CLONE - Uber-JIRA: Hadoop should support IPv6 Key: HADOOP-17800 URL: https://issues.apache.org/jira/browse/HADOOP-17800 Project: Hadoop Common Issue Type: Improvement Components: net Reporter: Brahma Reddy Battula Assignee: Nate Edel Hadoop currently treats IPv6 as unsupported. Track related smaller issues to support IPv6. (Current case here is mainly HBase on HDFS, so any suggestions about other test cases/workload are really appreciated.) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [ANNOUNCE] Apache Hadoop 3.3.1 release
Hi Wei-Chiu, Thanks for driving this.. Looks index.html is not updated and Changelog is broken. On Wed, Jun 16, 2021 at 7:59 AM Wei-Chiu Chuang wrote: > Hi All, > > It gives me great pleasure to announce that the Apache Hadoop community has > voted to release Apache Hadoop 3.3.1. > > This is the first stable release of Apache Hadoop 3.3.x line. It contains > 697 bug fixes, improvements and enhancements since 3.3.0. > > Users are encouraged to read the overview of major changes > <https://hadoop.apache.org/docs/r3.3.1/index.html> since 3.3.0. For > details > of 697 bug fixes, improvements, and other enhancements since the previous > 3.3.0 release, please check release notes > < > http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/RELEASENOTES.3.3.1.html > > > and changelog > < > http://hadoop.apache.org/docs/r3.3.1/hadoop-project-dist/hadoop-common/release/3.3.1/CHANGES.3.3.1.html > > > detail > the changes since 3.3.0. > > Many thanks to everyone who contributed to the release, and everyone in the > Apache Hadoop community! This release is a direct result of your great > contributions. > > Many thanks to everyone who helped in this release process! > > Many thanks to Sean Busbey, Chao Sun, Steve Loughran, Masatake Iwasaki, > Michael Stack, Viraj Jasani, Eric Payne, Ayush Saxena, Vinayakumar B, > Takanobu Asanuma, Xiaoqiao He and other folks who continued helps for this > release process. > > Best Regards, > Wei-Chiu Chuang > -- --Brahma Reddy Battula
Re: [E] Re: Do I need a +1 to merge a backport PR?
Currently how the addendum will be added to PR?? Won’t be another PR?? Where build run again ?? On Thu, 3 Jun 2021 at 1:33 AM, Eric Badger wrote: > I'm of a similar opinion to most here. If the backport is clean, I think > it's ok to do it with just the +1 on the original patch. However, please > please please build the code on the target branch before backporting > > Eric > > On Wed, Jun 2, 2021 at 2:46 PM Ayush Saxena wrote: > > > For trivial changes, like changes in import or conflicts due to line > > number or other trivial stuff, I don’t think that is required. Unless the > > general logic isn’t changing, we can go ahead, may be we can do a test > run > > before merging, to be on the safer side as and when required. :-) > > > > -Ayush > > > > > On 02-Jun-2021, at 10:13 AM, Wei-Chiu Chuang > wrote: > > > > > > I'm curious about the GitHub PR conventions we use today... say I want > > to > > > backport a commit from trunk to branch-3.3, and there's a small code > > > conflict so I push a PR against branch-3.3 using GitHub to go through > the > > > precommit check. > > > > > > Do I need explicit approval from another committer to merge the > backport > > > PR? (As.a committer, I know I can merge at any time) or can I merge > when > > > the precommit comes back okay? > > > > --------- > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 3.3.1 release
Hi Bilwa, I have commented on the jira's you mentioned. Based on the stability we can plan this.But needs to be merged ASAP. On Fri, Feb 19, 2021 at 5:20 PM bilwa st wrote: > Hi Brahma, > > Can we have below features in 3.3.1 release? We have been using these > features for a long time. They are stable and tested in bigger clusters. > > 1. Container reuse - https://issues.apache.org/jira/browse/MAPREDUCE-6749 > 2. Speculative attempts should not run on the same node - > https://issues.apache.org/jira/browse/MAPREDUCE-7169 > > Thanks, > Bilwa > > On Thu, Feb 18, 2021, 1:49 PM Brahma Reddy Battula > wrote: > >> Sorry for the late reply.. >> >> I will come up with a plan.. Please let me know if anybody has some >> features/improvements/bugs that need to be included. >> >> On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan wrote: >> >> > Hi Wei-Chiu, >> > >> > What will be the next steps here for 3.3.1 planning? >> > >> > Thanks >> > Sunil >> > >> > On Mon, Feb 8, 2021 at 11:56 PM Stack wrote: >> > >> > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran >> > > > > > >> > > wrote: >> > > >> > > > >> > > > Regarding blockers &c: how about we have a little hackathon where we >> > try >> > > > and get things in. This means a promise of review time from the >> people >> > > with >> > > > commit rights and other people who understand the code (Stack?) >> > > > >> > > > >> > > >> > > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing). >> > > Thanks, >> > > S >> > > >> > > >> > > >> > > >> > > > -steve >> > > > >> > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena >> wrote: >> > > > >> > > > > +1 >> > > > > Just to mention we would need to release hadoop-thirdparty too >> > before. >> > > > > Presently we are using the snapshot version of it. >> > > > > >> > > > > -Ayush >> > > > > >> > > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang > > >> > > > wrote: >> > > > > > >> > > > > > Hi all, >> > > > > > >> > > > > > Hadoop 3.3.0 was released half a year ago, and as of now we've >> > > > > accumulated >> > > > > > more than 400 changes in the branch-3.3. A number of >> downstreamers >> > > are >> > > > > > eagerly waiting for 3.3.1 which addresses the guava version >> > conflict >> > > > > issue. >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20 >> > > > > > >> > > > > > We should start the release work for 3.3.1 before the diff >> becomes >> > > even >> > > > > > larger. >> > > > > > >> > > > > > I believe there are currently only two real blockers for a >> 3.3.1 >> > > > (using >> > > > > > this filter >> > > > > > >> > > > > >> > > > >> > > >> > >> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC >> > > > > > ) >> > > > > > >> > > > > > >> > > > > > 1. HDFS-15566 < >> https://issues.apache.org/jira/browse/HDFS-15566> >> > > > > > 2. >> > > > > > 1. HADOOP-17112 < >> > > > https://issues.apache.org/jira/browse/HADOOP-17112 >> > > > > > >> > > > > > 2. >> > > > > > >> > > > > > >> > > > > > >> > > > > > Is there anyone who would volunteer to be the 3.3.1 RM? >> > > > > > >> > > > > > Also, the HowToRelease wiki does not describe the ARM build >> > process. >> > > > > That's >> > > > > > going to be important for future releases. >> > > > > >> > > > > >> - >> > > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> > > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> > > > > >> > > > > >> > > > >> > > >> > >> >> >> -- >> >> >> >> --Brahma Reddy Battula >> > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 3.3.1 release
Sorry for the late reply.. I will come up with a plan.. Please let me know if anybody has some features/improvements/bugs that need to be included. On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan wrote: > Hi Wei-Chiu, > > What will be the next steps here for 3.3.1 planning? > > Thanks > Sunil > > On Mon, Feb 8, 2021 at 11:56 PM Stack wrote: > > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran > > > > > wrote: > > > > > > > > Regarding blockers &c: how about we have a little hackathon where we > try > > > and get things in. This means a promise of review time from the people > > with > > > commit rights and other people who understand the code (Stack?) > > > > > > > > > > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing). > > Thanks, > > S > > > > > > > > > > > -steve > > > > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena wrote: > > > > > > > +1 > > > > Just to mention we would need to release hadoop-thirdparty too > before. > > > > Presently we are using the snapshot version of it. > > > > > > > > -Ayush > > > > > > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang > > > wrote: > > > > > > > > > > Hi all, > > > > > > > > > > Hadoop 3.3.0 was released half a year ago, and as of now we've > > > > accumulated > > > > > more than 400 changes in the branch-3.3. A number of downstreamers > > are > > > > > eagerly waiting for 3.3.1 which addresses the guava version > conflict > > > > issue. > > > > > > > > > > > > > > > > > > > > https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20 > > > > > > > > > > We should start the release work for 3.3.1 before the diff becomes > > even > > > > > larger. > > > > > > > > > > I believe there are currently only two real blockers for a 3.3.1 > > > (using > > > > > this filter > > > > > > > > > > > > > > > https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC > > > > > ) > > > > > > > > > > > > > > > 1. HDFS-15566 <https://issues.apache.org/jira/browse/HDFS-15566> > > > > > 2. > > > > > 1. HADOOP-17112 < > > > https://issues.apache.org/jira/browse/HADOOP-17112 > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > > > Is there anyone who would volunteer to be the 3.3.1 RM? > > > > > > > > > > Also, the HowToRelease wiki does not describe the ARM build > process. > > > > That's > > > > > going to be important for future releases. > > > > > > > > - > > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > > > > > > > -- --Brahma Reddy Battula
[jira] [Created] (HADOOP-17236) Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640
Brahma Reddy Battula created HADOOP-17236: - Summary: Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640 Key: HADOOP-17236 URL: https://issues.apache.org/jira/browse/HADOOP-17236 Project: Hadoop Common Issue Type: Bug Reporter: Brahma Reddy Battula Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640 -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: [Virtual MEETUP]: Migration to Hadoop 3
One more update from me. We didn't face any issues with YARN, for HDFS you can have a look at the following jira's. https://issues.apache.org/jira/browse/HDFS-13596 https://issues.apache.org/jira/browse/HDFS-14396 https://issues.apache.org/jira/browse/HDFS-14509 Following jira is incompatible for ACL commands.Only hadoop-3 clients will work against hadoop-3 server during the upgrade. https://issues.apache.org/jira/browse/HDFS-6984 On Wed, Aug 26, 2020 at 11:06 PM Brahma Reddy Battula wrote: > > Hi Eric, > > check the following references for the same. > > 01/02/2020 Didi talked about their large scale HDFS cluster upgrade > experience. > > Slides: > https://drive.google.com/open?id=1iwJ1asalYfgnOCBuE-RfeG-NpSocjIcy > > Recording: > https://cloudera.zoom.us/rec/share/7MF_dLX0339OY5391xvkZP8NLrXieaa8gyZK-fYJnUkGOUUXvaUh5cl_6AVYetQl > > Didi studied two upgrade approaches from the community documentation: > express upgrade and rolling upgrade. Rolling upgrade was selected. > > Yahoo Japan was trying out from hadoop-2.6 to hadop-3.2.1 > > https://techblog.yahoo.co.jp/entry/20191206786320/ > > On Wed, Aug 26, 2020 at 6:56 PM epa...@apache.org > wrote: > >> Hello. Just a reminder that today I would like to invite you all to >> discuss your >> experiences migrating from Hadoop 2 to Hadoop 3. >> >> -Eric >> >> On Monday, August 24, 2020, 1:58:37 PM CDT, epa...@apache.org < >> epa...@apache.org> wrote: >> >> Hello everyone! >> >> We are considering migrating to Hadoop 3, and we would be very interested >> to >> hear about your experiences. If you have migrated from Hadoop 2 to Hadoop >> 3 >> and can provide insights, please kindly consider attending the following: >> >> Date: Wednesday, Aug 26, 2020 >> Time: 10:00 A.M. PDT / 12:00 P.M. CDT / 01:00 P.M. EDT / 05:00 P.M. GMT >> Location: Zoom: https://cloudera.zoom.us/j/880548968 >> >> Hope to see you there! >> >> Thank you! >> Eric Payne >> @ Verizon Media >> >> --------- >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org >> >> > > -- > > > > --Brahma Reddy Battula > -- --Brahma Reddy Battula
Re: [Virtual MEETUP]: Migration to Hadoop 3
Hi Eric, check the following references for the same. 01/02/2020 Didi talked about their large scale HDFS cluster upgrade experience. Slides: https://drive.google.com/open?id=1iwJ1asalYfgnOCBuE-RfeG-NpSocjIcy Recording: https://cloudera.zoom.us/rec/share/7MF_dLX0339OY5391xvkZP8NLrXieaa8gyZK-fYJnUkGOUUXvaUh5cl_6AVYetQl Didi studied two upgrade approaches from the community documentation: express upgrade and rolling upgrade. Rolling upgrade was selected. Yahoo Japan was trying out from hadoop-2.6 to hadop-3.2.1 https://techblog.yahoo.co.jp/entry/20191206786320/ On Wed, Aug 26, 2020 at 6:56 PM epa...@apache.org wrote: > Hello. Just a reminder that today I would like to invite you all to > discuss your > experiences migrating from Hadoop 2 to Hadoop 3. > > -Eric > > On Monday, August 24, 2020, 1:58:37 PM CDT, epa...@apache.org < > epa...@apache.org> wrote: > > Hello everyone! > > We are considering migrating to Hadoop 3, and we would be very interested > to > hear about your experiences. If you have migrated from Hadoop 2 to Hadoop 3 > and can provide insights, please kindly consider attending the following: > > Date: Wednesday, Aug 26, 2020 > Time: 10:00 A.M. PDT / 12:00 P.M. CDT / 01:00 P.M. EDT / 05:00 P.M. GMT > Location: Zoom: https://cloudera.zoom.us/j/880548968 > > Hope to see you there! > > Thank you! > Eric Payne > @ Verizon Media > > - > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: common-dev-h...@hadoop.apache.org > > -- --Brahma Reddy Battula
Re: Mandarin Hadoop online sync this week
HI, what you are planning for this week? On Tue, Aug 25, 2020 at 10:18 PM Wei-Chiu Chuang wrote: > Hello, > > There hasn't been a Mandarin online sync for quite some time. I'd like to > call for one this week: > > Date/time: > > 8/27 Thursday Beijing Time 1PM > 8/26 Wednesday US Pacific Time 10PM > > Link: > https://cloudera.zoom.us/j/880548968 > > Past sync summary: > > https://docs.google.com/document/d/1jXM5Ujvf-zhcyw_5kiQVx6g-HeKe-YGnFS_1-qFXomI/edit > -- --Brahma Reddy Battula
[jira] [Created] (HADOOP-17225) Update jackson-mapper-asl-1.9.13 to atlassian version to mitigate: CVE-2019-10172
Brahma Reddy Battula created HADOOP-17225: - Summary: Update jackson-mapper-asl-1.9.13 to atlassian version to mitigate: CVE-2019-10172 Key: HADOOP-17225 URL: https://issues.apache.org/jira/browse/HADOOP-17225 Project: Hadoop Common Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula Currently jersey depends on the jackson, and upgradation of jersey from 1.X to 2.x looks complicated(see HADOOP-15984 and HADOOP-16485). Update jackson-mapper-asl-1.9.13 to atlassian version to mitigate: CVE-2019-10172. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17221) Upgrade log4j-1.2.17 to atlassian ( To Adress: CVE-2019-17571)
Brahma Reddy Battula created HADOOP-17221: - Summary: Upgrade log4j-1.2.17 to atlassian ( To Adress: CVE-2019-17571) Key: HADOOP-17221 URL: https://issues.apache.org/jira/browse/HADOOP-17221 Project: Hadoop Common Issue Type: Bug Reporter: Brahma Reddy Battula Currentlly there are no active release under 1.X in log4j and log4j2 is incompatiable to upgrade (see HADOOP-16206 ) for more details. But following CVE is reported on log4j 1.2.17..I think,we should consider to update to Atlassian([https://mvnrepository.com/artifact/log4j/log4j/1.2.17-atlassian-0.4]) or redhat versions [https://nvd.nist.gov/vuln/detail/CVE-2019-17571] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
[jira] [Created] (HADOOP-17220) Upgrade slf4j to 1.7.30 ( To Adress: CVE-2018-8088)
Brahma Reddy Battula created HADOOP-17220: - Summary: Upgrade slf4j to 1.7.30 ( To Adress: CVE-2018-8088) Key: HADOOP-17220 URL: https://issues.apache.org/jira/browse/HADOOP-17220 Project: Hadoop Common Issue Type: Bug Reporter: Brahma Reddy Battula Assignee: Brahma Reddy Battula To address the following CVE, upgrade the slf4j to latest stable release 1.7.30. [https://nvd.nist.gov/vuln/detail/CVE-2018-8088] Note: We don't use EventData but should consider upgrading. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
Re: Progress
Looks still the problem with yetus isn't solved even after docker is running and has the permissions. As of now, I coped the old script from the build( https://ci-hadoop.apache.org/job/hadoop-qbt-linux-ARM-trunk/8/console). Currently it's running. https://ci-hadoop.apache.org/job/hadoop-qbt-linux-ARM-trunk/8/console On Sun, Jul 26, 2020 at 10:05 PM Akira Ajisaka wrote: > Hi Gavin, > > Thank you for moving the arm nodes. Would you install and start the Docker > daemon on arm nodes? > > ``` > Got permission denied while trying to connect to the Docker daemon socket > at unix:///var/run/docker.sock: Get > http://%2Fvar%2Frun%2Fdocker.sock/v1.40/version: > dial unix /var/run/docker.sock: connect: permission denied > > ``` > > https://ci-hadoop.apache.org/job/hadoop-qbt-linux-ARM-trunk/3/console > > Thanks, > Akira > > On Thu, Jul 23, 2020 at 11:47 AM Zhenyu Zheng > wrote: > > > Thanks for doing this, I was going to post in the JIRA about this > yesterday > > but was disturbed by something, > > please help us migrate the ARM nodes. > > > > BR, > > > > On Thu, Jul 23, 2020 at 1:40 AM Gavin McDonald > > wrote: > > > > > Hi , > > > > > > On Wed, Jul 22, 2020 at 7:00 PM Akira Ajisaka > > wrote: > > > > > > > +CC: common-dev@hadoop > > > > > > > > Hi Gavin, > > > > > > > > Thank you for the reminder. > > > > I have one question. When ARM servers will be moved to ci-hadoop? > > Hadoop > > > > and HBase have daily jobs that run on ARM servers. > > > > > > > > I'll migrate the other Hadoop Jenkins jobs to ci-hadoop this weekend. > > > > > > > > > > Thanks. > > > > > > I'll move 2 of the ARM nodes tomorrow > > > > > > HTH > > > > > > > > > > > > > > Regards, > > > > Akira > > > > > > > > On Wed, Jul 22, 2020 at 8:39 PM Gavin McDonald > > > > > wrote: > > > > > > > > > Hi All, > > > > > > > > > > Seems there is still not much happening in the way of migrating to > > > > > ci-hadoop > > > > > > > > > > Do you need any more information from me, and help needed? > > > > > > > > > > The cut off date is 15th August and builds.a.o will be turned off > on > > > that > > > > > date. > > > > > > > > > > -- > > > > > > > > > > *Gavin McDonald* > > > > > Systems Administrator > > > > > ASF Infrastructure Team > > > > > > > > > > > > > > > > > > -- > > > > > > *Gavin McDonald* > > > Systems Administrator > > > ASF Infrastructure Team > > > > > > -- --Brahma Reddy Battula
[ANNOUNCE] Apache Hadoop 3.3.0 release
Greetings all, It gives me great pleasure to announce that the Apache Hadoop community has voted to release Apache Hadoop 3.3.0. Apache Hadoop 3.3.0 is the first release of Apache Hadoop 3.3 line for the year 2020, which includes 2148 fixes since the previous Hadoop 3.2.0 release. Of these fixes: - 525 in Hadoop Common - 804 in HDFS - 763 in YARN - 56 in MapReduce Apache Hadoop 3.3.0 contains a number of significant features and enhancements.A few of them are noted as below. - *Support ARM:* This is the first release to support ARM architectures - *Upgrade protobuf from 2.5.0 : *Protobuf upgraded to 3.7.1 as protobuf-2.5.0 reached EOL. - *S3A Enhancements : *Lots of enhancements to the S3A code including Delegation Token support, better handling of 404 caching, S3guard performance, resilience improvements - *ABFS Enhancements :* Address issues which surface in the field and tune things which need tuning, add more tests where appropriate.Improve docs, especially troubleshooting. - *Tencent Cloud COS File System Implementation: *Tencent cloud is top 2 cloud vendors in China market and the object store COS is widely used among China’s cloud users. COSN filesytem to support Tencent cloud COS natively in Hadoop. - *Java11 Runtime Support* : Java 11 runtime support is completed. - *HDFS RBF stabilizatio**n*: HDFS Router now supports security. Also contains many bug fixes and improvements. - *DNS Resolution to support Nameservices to IP Support:*DFS clients can use a single domain name to discover servers (namenodes/routers/observers) instead of explicitly listing out all hosts in the config *- Scheduling of opportunistic containers : S*cheduling of opportunistic container through the central RM (YARN-5220), through distributed scheduling (YARN-2877), as well as the scheduling of containers based on actual node utilization (YARN-1011) and the container promotion/demotion (YARN-5085). *- Application Catalog for YARN applications: **Application catalog system which provides an editorial and search interface for YARN applications.*This improves the usability of YARN for managing the life cycle of applications. * For major changes included in Hadoop 3.3 line, please refer to Hadoop3.3.0 main page [1]. * For more details about fixes in 3.3.0 release, please read the CHANGELOG [2] and RELEASENOTES [3]. The release news is posted on the Hadoop website too, you can go to the downloads section directly [4]. Many thanks to everyone who contributed to the release, and everyone in the Apache Hadoop community! This release is a direct result of your great contributions. Many thanks to Vinayakumar B who helped in this release process. [1] https://hadoop.apache.org/docs <https://hadoop.apache.org/docs/r3.2.0/> [2] https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/CHANGELOG.3.3.0.html [3] https://hadoop.apache.org/docs/r3.3.0/hadoop-project-dist/hadoop-common/release/3.3.0/RELEASENOTES.3.3.0.html [4] https://hadoop.apache.org/rele <https://hadoop.apache.org/releases.html> ases.html Regards, Brahma Reddy Battula
Re: [RESULT][VOTE] Rlease Apache Hadoop-3.3.0
Hi Stephen, thanks for bringing this to my attention. Looks it's late..I pushed the release tag ( which can't be reverted) and updated the release date in the jira. Can we plan this next release near future..? On Wed, Jul 15, 2020 at 5:25 PM Stephen O'Donnell wrote: > Hi All, > > Sorry for being a bit late to this, but I wonder if we have a potential > blocker to this release. > > In Cloudera we have recently encountered a serious dataloss issue in HDFS > surrounding snapshots. To hit the dataloss issue, you must have HDFS-13101 > and HDFS-15012 on the build (which branch-3.3.0 does). To prevent it, you > must also have HDFS-15313 and unfortunately, this was only committed to > trunk, so we need to cherry-pick it down the active branches. > > With data loss being a serious issue, should we pull this Jira into > branch-3.3.0 and cut a new release candidate? > > Thanks, > > Stephen. > > On Tue, Jul 14, 2020 at 1:22 PM Brahma Reddy Battula > wrote: > > > Hi All, > > > > With 8 binding and 11 non-binding +1s and no -1s the vote for Apache > > hadoop-3.3.0 Release > > passes. > > > > Thank you everybody for contributing to the release, testing, and voting. > > > > Special thanks whoever verified the ARM Binary as this is the first > release > > to support the ARM in hadoop. > > > > > > Binding +1s > > > > = > > Akira Ajisaka > > Vinayakumar B > > Inigo Goiri > > Surendra Singh Lilhore > > Masatake Iwasaki > > Rakesh Radhakrishnan > > Eric Badger > > Brahma Reddy Battula > > > > Non-binding +1s > > > > = > > Zhenyu Zheng > > Sheng Liu > > Yikun Jiang > > Tianhua huang > > Ayush Saxena > > Hemanth Boyina > > Bilwa S T > > Takanobu Asanuma > > Xiaoqiao He > > CR Hota > > Gergely Pollak > > > > I'm going to work on staging the release. > > > > > > The voting thread is: > > > > https://s.apache.org/hadoop-3.3.0-Release-vote-thread > > > > > > > > --Brahma Reddy Battula > > > -- --Brahma Reddy Battula
[RESULT][VOTE] Rlease Apache Hadoop-3.3.0
Hi All, With 8 binding and 11 non-binding +1s and no -1s the vote for Apache hadoop-3.3.0 Release passes. Thank you everybody for contributing to the release, testing, and voting. Special thanks whoever verified the ARM Binary as this is the first release to support the ARM in hadoop. Binding +1s = Akira Ajisaka Vinayakumar B Inigo Goiri Surendra Singh Lilhore Masatake Iwasaki Rakesh Radhakrishnan Eric Badger Brahma Reddy Battula Non-binding +1s = Zhenyu Zheng Sheng Liu Yikun Jiang Tianhua huang Ayush Saxena Hemanth Boyina Bilwa S T Takanobu Asanuma Xiaoqiao He CR Hota Gergely Pollak I'm going to work on staging the release. The voting thread is: https://s.apache.org/hadoop-3.3.0-Release-vote-thread --Brahma Reddy Battula
Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0
Hi Xiaoqiao, Thanks for bringing this to my attention. It's too rare to occur this scenario and a couple of releases went after HDFS-4882(broken jira). So,You think it's fine to include in the next release? Or if you still insist we can go for another RC... On Mon, Jul 13, 2020 at 12:29 PM Xiaoqiao He wrote: > Thanks Brahma Reddy Battula for your great work here. > Stephen fixed lease leak in namenode, and it is ready now: > https://issues.apache.org/jira/browse/HDFS-14498. > I think this affects 3.3.0-RC0. Would you check this? > Sorry for reporting it so late. > > On Mon, Jul 13, 2020 at 2:53 PM Takanobu Asanuma > wrote: > >> +1(non-binding) >>- verified checksums >>- succeeded in building the package with OpenJDK 8 >>- started HDFS cluster with kerberos with OpenJDK 11 >>- verified Web UIs (NN, DN, Router) >>- Ran some operations of Router-based Federation with Security >>- Ran some operations of Erasure Coding >> >> Thanks, >> Takanobu >> >> >> From: Bilwa S T >> Sent: Monday, July 13, 2020 2:23 >> To: Surendra Singh Lilhore; hemanth boyina >> Cc: Brahma Reddy Battula; mapreduce-dev; Hdfs-dev; Hadoop Common; yarn-dev >> Subject: RE: [VOTE] Release Apache Hadoop 3.3.0 - RC0 >> >> +1(non-binding) >> >> 1. Deployed 3 node cluster >> 2. Browsed through Web UI (RM, NM) >> 3. Executed Jobs (pi, wordcount, TeraGen, TeraSort) >> 4. Verified basic yarn commands >> >> Thanks, >> Bilwa >> >> -Original Message- >> From: Surendra Singh Lilhore [mailto:surendralilh...@gmail.com] >> Sent: 12 July 2020 18:32 >> To: hemanth boyina >> Cc: Iñigo Goiri ; Vinayakumar B < >> vinayakum...@apache.org>; Brahma Reddy Battula ; >> mapreduce-dev ; Hdfs-dev < >> hdfs-...@hadoop.apache.org>; Hadoop Common ; >> yarn-dev >> Subject: Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0 >> >> +1(binding) >> >> Deployed HDFS and Yarn Cluster >> > Verified basic shell commands >> > Ran some jobs >> > Verified UI >> >> -Surendra >> >> On Sat, Jul 11, 2020 at 9:41 PM hemanth boyina < >> hemanthboyina...@gmail.com> >> wrote: >> >> > +1(non-binding) >> > Deployed Cluster with Namenodes and Router *)verified shell commands >> > *)Executed various jobs *)Browsed UI's >> > >> > >> > Thanks, >> > HemanthBoyina >> > >> > >> > On Sat, 11 Jul 2020, 00:05 Iñigo Goiri, wrote: >> > >> > > +1 (Binding) >> > > >> > > Deployed a cluster on Azure VMs with: >> > > * 3 VMs with HDFS Namenodes and Routers >> > > * 2 VMs with YARN Resource Managers >> > > * 5 VMs with HDFS Datanodes and Node Managers >> > > >> > > Tests: >> > > * Executed Tergagen+Terasort+Teravalidate. >> > > * Executed wordcount. >> > > * Browsed through the Web UI. >> > > >> > > >> > > >> > > On Fri, Jul 10, 2020 at 1:06 AM Vinayakumar B >> > > >> > > wrote: >> > > >> > > > +1 (Binding) >> > > > >> > > > -Verified all checksums and Signatures. >> > > > -Verified site, Release notes and Change logs >> > > > + May be changelog and release notes could be grouped based on >> > > > the project at second level for better look (this needs to be >> > > > supported >> > from >> > > > yetus) >> > > > -Tested in x86 local 3-node docker cluster. >> > > > + Built from source with OpenJdk 8 and Ubuntu 18.04 >> > > > + Deployed 3 node docker cluster >> > > > + Ran various Jobs (wordcount, Terasort, Pi, etc) >> > > > >> > > > No Issues reported. >> > > > >> > > > -Vinay >> > > > >> > > > On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu >> > > wrote: >> > > > >> > > > > +1 (non-binding) >> > > > > >> > > > > - checkout the "3.3.0-aarch64-RC0" binaries packages >> > > > > >> > > > > - started a clusters with 3 nodes VMs of Ubuntu 18.04 >> > > > > ARM/aarch64, openjdk-11-jdk >> > > > > >> > > > > - checked some web UIs (NN, DN, RM, NM) >> > > > > >> > > > >
[VOTE] Release Apache Hadoop 3.3.0 - RC0
Hi folks, This is the first release candidate for the first release of Apache Hadoop 3.3.0 line. It contains *1644[1]* fixed jira issues since 3.2.1 which include a lot of features and improvements(read the full set of release notes). Below feature additions are the highlights of the release. - ARM Support - Enhancements and new features on S3a,S3Guard,ABFS - Java 11 Runtime support and TLS 1.3. - Support Tencent Cloud COS File System. - Added security to HDFS Router. - Support non-volatile storage class memory(SCM) in HDFS cache directives - Support Interactive Docker Shell for running Containers. - Scheduling of opportunistic containers - A pluggable device plugin framework to ease vendor plugin development *The RC0 artifacts are at*: http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/ *First release to include ARM binary, Have a check.* *RC tag is *release-3.3.0-RC0. *The maven artifacts are hosted here:* https://repository.apache.org/content/repositories/orgapachehadoop-1271/ *My public key is available here:* https://dist.apache.org/repos/dist/release/hadoop/common/KEYS The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM IST. I have done a few testing with my pseudo cluster. My +1 to start. Regards, Brahma Reddy Battula 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.3.0) AND fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER BY fixVersion ASC
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
Yes, All the blockers are closed.will cut RC soon.. On Thu, Jun 18, 2020 at 7:49 PM Adam Antal wrote: > YARN-10314 is also merged. I don't see any blockers at this point. > (Actually I couldn't see any jiras > < > https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%2Fs%22%20%3D%203.3.0%20ORDER%20BY%20priority%20DESC > > > targeted for 3.3.0). > > In the community sync yesterday we wanted to discuss the 3.3.0 release, but > nobody had information about it in the call. Could you share the latest on > the upcoming 3.3.0 release? > > Thanks, > Adam > > On Mon, Jun 15, 2020 at 9:17 AM Ayush Saxena wrote: > > > YARN-10314 also seems to be a blocker. > > > > https://issues.apache.org/jira/browse/YARN-10314 > > > > We should wait for that as well, should get concluded in a day or two. > > > > -Ayush > > > > > On 15-Jun-2020, at 7:21 AM, Sheng Liu wrote: > > > > > > The HADOOP-17046 <https://issues.apache.org/jira/browse/HADOOP-17046 > > > > has > > > been merged :) > > > > > > Brahma Reddy Battula 于2020年6月4日周四 下午10:43写道： > > > > > >> Following blocker is pending for 3.3.0 release which is ready for > > review. > > >> Mostly we'll have RC soon. > > >> https://issues.apache.org/jira/browse/HADOOP-17046 > > >> > > >> Protobuf dependency was unexpected . > > >> > > >>> On Mon, Jun 1, 2020 at 7:11 AM Sheng Liu > > wrote: > > >>> > > >>> Hi folks, > > >>> > > >>> It looks like the 3.3.0 branch has been created for quite a while. > Not > > >> sure > > >>> if there is remain block issue that need to be addressed before > Hadoop > > >>> 3.3.0 release publishing, maybe we can bring up to here and move the > > >>> release forward ? > > >>> > > >>> Thank. > > >>> > > >>> Brahma Reddy Battula 于2020年3月25日周三 上午1:55写道： > > >>> > > >>>> thanks to all. > > >>>> > > >>>> will make this as optional..will update the wiki accordingly. > > >>>> > > >>>> On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B < > > >> vinayakum...@apache.org> > > >>>> wrote: > > >>>> > > >>>>> Making ARM artifact optional, makes the release process simpler for > > >> RM > > >>>> and > > >>>>> unblocks release process (if there is unavailability of ARM > > >> resources). > > >>>>> > > >>>>> Still there are possible options to collaborate with RM ( as brahma > > >>>>> mentioned earlier) and provide ARM artifact may be before or after > > >>> vote. > > >>>>> If feasible RM can decide to add ARM artifact by collaborating with > > >>>> @Brahma > > >>>>> Reddy Battula or me to get the ARM artifact. > > >>>>> > > >>>>> -Vinay > > >>>>> > > >>>>> On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal > > >>>>> wrote: > > >>>>> > > >>>>>> Thanks for the clarification Brahma. Can you update the proposal > to > > >>>> state > > >>>>>> that it is optional (it may help to put the proposal on cwiki)? > > >>>>>> > > >>>>>> Also if we go ahead then the RM documentation should be clear this > > >> is > > >>>> an > > >>>>>> optional step. > > >>>>>> > > >>>>>> > > >>>>>>> On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula < > > >>>> bra...@apache.org> > > >>>>>> wrote: > > >>>>>>> > > >>>>>>> Sure, we can't make mandatory while voting and we can upload to > > >>>>> downloads > > >>>>>>> once release vote is passed. > > >>>>>>> > > >>>>>>> On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal > > >>>>>>> wrote: > > >>>>>>> > > >>>>>>>>> Sorry,didn't get you...do
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
Following blocker is pending for 3.3.0 release which is ready for review. Mostly we'll have RC soon. https://issues.apache.org/jira/browse/HADOOP-17046 Protobuf dependency was unexpected . On Mon, Jun 1, 2020 at 7:11 AM Sheng Liu wrote: > Hi folks, > > It looks like the 3.3.0 branch has been created for quite a while. Not sure > if there is remain block issue that need to be addressed before Hadoop > 3.3.0 release publishing, maybe we can bring up to here and move the > release forward ? > > Thank. > > Brahma Reddy Battula 于2020年3月25日周三 上午1:55写道： > > > thanks to all. > > > > will make this as optional..will update the wiki accordingly. > > > > On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B > > wrote: > > > > > Making ARM artifact optional, makes the release process simpler for RM > > and > > > unblocks release process (if there is unavailability of ARM resources). > > > > > > Still there are possible options to collaborate with RM ( as brahma > > > mentioned earlier) and provide ARM artifact may be before or after > vote. > > > If feasible RM can decide to add ARM artifact by collaborating with > > @Brahma > > > Reddy Battula or me to get the ARM artifact. > > > > > > -Vinay > > > > > > On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal > > > wrote: > > > > > > > Thanks for the clarification Brahma. Can you update the proposal to > > state > > > > that it is optional (it may help to put the proposal on cwiki)? > > > > > > > > Also if we go ahead then the RM documentation should be clear this is > > an > > > > optional step. > > > > > > > > > > > > > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula < > > bra...@apache.org> > > > > wrote: > > > > > > > > > > Sure, we can't make mandatory while voting and we can upload to > > > downloads > > > > > once release vote is passed. > > > > > > > > > > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal > > > > > wrote: > > > > > > > > > >>> Sorry,didn't get you...do you mean, once release voting is > > > > >>> processed and upload by RM..? > > > > >> > > > > >> Yes, that is what I meant. I don’t want us to make more mandatory > > work > > > > for > > > > >> the release manager because the job is hard enough already. > > > > >> > > > > >> > > > > >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula < > > > bra...@apache.org> > > > > >> wrote: > > > > >>> > > > > >>> Sorry,didn't get you...do you mean, once release voting is > > processed > > > > and > > > > >>> upload by RM..? > > > > >>> > > > > >>> FYI. There is docker image for ARM also which support all scripts > > > > >>> (createrelease, start-build-env.sh, etc ). > > > > >>> > > > > >>> https://issues.apache.org/jira/browse/HADOOP-16797 > > > > >>> > > > > >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal > > > > >>> wrote: > > > > >>> > > > > >>>> Can ARM binaries be provided after the fact? We cannot increase > > the > > > > RM’s > > > > >>>> burden by asking them to generate an extra set of binaries. > > > > >>>> > > > > >>>> > > > > >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula < > > > > bra...@apache.org> > > > > >>>> wrote: > > > > >>>>> > > > > >>>>> + Dev mailing list. > > > > >>>>> > > > > >>>>> -- Forwarded message - > > > > >>>>> From: Brahma Reddy Battula > > > > >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM > > > > >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary > > > > >>>>> To: junping_du > > > > >>>>> > > > > >>>>> > > > > >>>>> thanks junping for your reply. > > > > >>>>> > > > > >>>>>
Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature
StorageContainerLocationProtocolClientSideTranslatorPB.java:[116,47] > > > > > incompatible types: com.google.protobuf.ServiceException cannot be > > > > > converted to org.apache.hadoop.thirdparty.protobuf.ServiceException > > > > > 17:01:19 2020/04/16 00:01:19 INFO: [INFO] 2 errors > > > > > > > > > > > > > > > There's another error where Ozone uses the Hadoop RPC framework > which > > > > uses > > > > > the hadoop.thirdparty protobuf. > > > > > > > > > > [ERROR] Failed to execute goal > > > > > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile > > > > > (default-testCompile) on project hadoop-hdds-container-service: > > > > Compilation > > > > > failure > > > > > [ERROR] > > > > > > > > > > > > > > > > > > > > /Users/weichiu/sandbox/ozone/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java:[103,41] > > > > > incompatible types: com.google.protobuf.BlockingService cannot be > > > > converted > > > > > to org.apache.hadoop.thirdparty.protobuf.BlockingService > > > > > > > > > > BlockingService scmDatanodeService = > > > > > StorageContainerDatanodeProtocolService. > > > > > newReflectiveBlockingService( > > > > > new > > StorageContainerDatanodeProtocolServerSideTranslatorPB( > > > > > server, > Mockito.mock(ProtocolMessageMetrics.class))); > > > > > > > > > > > > > > > > > > > > Ratis probably breaks as well since it depends on the Hadoop RPC > > > > framework > > > > > too. > > > > > > > > > > On Tue, Apr 28, 2020 at 10:58 PM Vinayakumar B < > > > vinayakum...@apache.org> > > > > > wrote: > > > > > > > > > > > hi Wei-Chiu, > > > > > > > > > > > > Can you elaborate on what failures you are facing related to > > > relocated > > > > > > protobuf classes.. ? > > > > > > > > > > > > IFAIK, if the issue with location of protobuf classes, still old > > jar > > > > > > protobuf-2.5.0.jar will be available in classpath. So downstream > > > > > depending > > > > > > on 2.5.0 version of protobuf still be able to access them. > > > > > > > > > > > > -vinay > > > > > > > > > > > > On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang, < > > weic...@cloudera.com > > > > > > > > > > wrote: > > > > > > > > > > > >> I'm sorry for coming to this late. I missed this message. It > > should > > > > have > > > > > >> been a DISCUSS thread rather than NOTICE. > > > > > >> > > > > > >> Looks like this is inevitable. But we should make the downstream > > > > > >> developers aware & make the update easier. As long as it is > stated > > > > > clearly > > > > > >> how to update the code to support Hadoop 3.3, I am okay with > that. > > > > > >> > > > > > >> Here's what I suggest: > > > > > >> (1) label the jira incompatible (just updated the jira) and > > updated > > > > the > > > > > >> release note to tell app developer how to update. > > > > > >> (2) declare ProtobufHelper a public API HADOOP-17019 > > > > > >> <https://issues.apache.org/jira/browse/HADOOP-17019> > > > > > >> > > > > > >> Tez doesn't use the removed Token API, but there's code that > > breaks > > > > with > > > > > >> the relocated protobuf class. The ProtobufHelper API will make > > this > > > > > >> transition much easier. > > > > > >> > > > > > >> Other downstreamers that break with the relocated protobuf > > include: > > > > > Ozone > > > > > >> and HBase. but neither of them use the removed Token API. > > > > > >> > > > > > >> > > > > > >> On Wed, Jan 8, 2020 at 4:40 AM Vinayakumar B < > > > vinayakum...@apache.org > > > > > > > > > > >> wrote: > > > > > >> > > > > > >>> Hi All, > > > > > >>> > > > > > >>>This mail is to notify about the Removal of following public > > > APIs > > > > > from > > > > > >>> Hadoop Common. > > > > > >>> > > > > > >>> ClassName: org.apache.hadoop.security.token.Token > > > > > >>> APIs: > > > > > >>> public Token(TokenProto tokenPB); > > > > > >>> public TokenProto toTokenProto(); > > > > > >>> > > > > > >>>Reason: These APIs are having Generated protobuf classes in > > the > > > > > >>> signature. Right now due to protobuf upgrade in trunk (soon to > be > > > > 3.3.0 > > > > > >>> release) these APIs are breaking the downstream builds, even > > though > > > > > >>> downstreams dont use these APIs (just Loading Token class). > > > > Downstreams > > > > > >>> are > > > > > >>> still referencing having older version (2.5.0) of protobuf, > hence > > > > build > > > > > >>> is > > > > > >>> being broken. > > > > > >>> > > > > > >>> These APIs were added for the internal > purpose(HADOOP-12563), > > > to > > > > > >>> support serializing tokens using protobuf in UGI Credentials. > > > > > >>> Same purpose can be achieved using the Helper classes without > > > > > introducing > > > > > >>> protobuf classes in API signatures. > > > > > >>> > > > > > >>> Token.java is marked as Evolving, so I believe APIs can be > > changed > > > > > >>> whenever > > > > > >>> absolute necessary. > > > > > >>> > > > > > >>> Jira https://issues.apache.org/jira/browse/HADOOP-16621 > has > > > been > > > > > >>> reported to solve downstream build failure. > > > > > >>> > > > > > >>> So since this API was added for internal purpose easy approach > to > > > > solve > > > > > >>> this is to remove APIs and use helper classes. Otherwise, as > > > > mentioned > > > > > in > > > > > >>> HADOOP-16621, workaround will add unnecessary codes to be > > > maintained. > > > > > >>> > > > > > >>> If anyone using these APIs outside hadoop project accidentally, > > > > please > > > > > >>> reply to this mail immediately. > > > > > >>> > > > > > >>> If no objection by next week, will go ahead with removal of > above > > > > said > > > > > >>> APIs > > > > > >>> in HADOOP-16621. > > > > > >>> > > > > > >>> -Vinay > > > > > >>> > > > > > >> > > > > > > > > > > > > > > > -- --Brahma Reddy Battula
Re: [DISCUSS] making Ozone a separate Apache project
+1 On Sat, 16 May 2020 at 12:09 AM, Subru Krishnan wrote: > +1. > > Thanks, > Subru > > On Thu, May 14, 2020 at 10:00 PM Akira Ajisaka > wrote: > > > +1 > > > > -Akira > > > > On Wed, May 13, 2020 at 4:53 PM Elek, Marton wrote: > > > > > > > > > > > I would like to start a discussion to make a separate Apache project > for > > > Ozone > > > > > > > > > > > > ### HISTORY [1] > > > > > > * Apache Hadoop Ozone development started on a feature branch of > > > Hadoop repository (HDFS-7240) > > > > > > * In the October of 2017 a discussion has been started to merge it to > > > the Hadoop main branch > > > > > > * After a long discussion it's merged to Hadoop trunk at the March of > > > 2018 > > > > > > * During the discussion of the merge, it was suggested multiple times > > > to create a separated project for the Ozone. But at that time: > > > 1). Ozone was tightly integrated with Hadoop/HDFS > > > 2). There was an active plan to use Block layer of Ozone (HDDS or > > > HDSL at that time) as the block level of HDFS > > > 3). The community of Ozone was a subset of the HDFS community > > > > > > * The first beta release of Ozone was just released. Seems to be a > > > good time before the first GA to make a decision about the future. > > > > > > > > > > > > ### WHAT HAS BEEN CHANGED > > > > > > During the last years Ozone became more and more independent both at > > > the community and code side. The separation has been suggested again > and > > > again (for example by Owen [2] and Vinod [3]) > > > > > > > > > > > > From COMMUNITY point of view: > > > > > > > > >* Fortunately more and more new contributors are helping Ozone. > > > Originally the Ozone community was a subset of HDFS project. But now a > > > bigger and bigger part of the community is related to Ozone only. > > > > > >* It seems to be easier to _build_ the community as a separated > > project. > > > > > >* A new, younger project might have different practices > > > (communication, commiter criteria, development style) compared to old, > > > mature project > > > > > >* It's easier to communicate (and improve) these standards in a > > > separated projects with clean boundaries > > > > > >* Separated project/brand can help to increase the adoption rate and > > > attract more individual contributor (AFAIK it has been seen in > Submarine > > > after a similar move) > > > > > > * Contribution process can be communicated more easily, we can make > > > first time contribution more easy > > > > > > > > > > > > From CODE point of view Ozone became more and more independent: > > > > > > > > > * Ozone has different release cycle > > > > > > * Code is already separated from Hadoop code base > > > (apache/hadoop-ozone.git) > > > > > > * It has separated CI (github actions) > > > > > > * Ozone uses different (more strict) coding style (zero toleration of > > > unit test / checkstyle errors) > > > > > > * The code itself became more and more independent from Hadoop on > > > Maven level. Originally it was compiled together with the in-tree > latest > > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC, > > > Configuration...) > > > > > > * It starts to use multiple version of Hadoop (on client side) > > > > > > * Volume of resolved issues are already very high on Ozone side > (Ozone > > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all > > > together in the last 2-3 months) > > > > > > > > > Summary: Before the first Ozone GA release, It seems to be a good time > > > to discuss the long-term future of Ozone. Managing it as a separated > TLP > > > project seems to have more benefits. > > > > > > > > > Please let me know what your opinion is... > > > > > > Thanks a lot, > > > Marton > > > > > > > > > > > > > > > > > > [1]: For more details, see: > > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md > > > > > > [2]: > > > > > > > > > https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E > > > > > > [3]: > > > > > > > > > https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E > > > > > > - > > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > > > > > > -- --Brahma Reddy Battula
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
Thanks Akira and Surendra for pointing that out. Looks I missed after creating the branch.. On Thu, 30 Apr 2020 at 10:16 AM, Akira Ajisaka wrote: > Hi Surendra, > > Updated the version to 3.3.1-SNAPSHOT in branch-3.3. > > Thanks, > Akira > > On Wed, Apr 29, 2020 at 4:22 PM Surendra Singh Lilhore < > surendralilh...@gmail.com> wrote: > >> Hi Brahma, >> >> Why the branch-3.3 & branch-3.3.0 pom version is same ?. >> >> In branch-3.3 pom version should be 3.3.1. >> >> Please correct me if I am wrong. >> >> -Surendra >> >> >> On Sat, 25 Apr, 2020, 9:33 am Mingliang Liu, wrote: >> >> > Brahma, >> > >> > What about https://issues.apache.org/jira/browse/HADOOP-17007? >> > >> > Thanks, >> > >> > On Fri, Apr 24, 2020 at 11:07 AM Brahma Reddy Battula < >> bra...@apache.org> >> > wrote: >> > >> > > Ok. Done. Branch created. >> > > >> > > Following blockers are pending, will closely track this. >> > > >> > > https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under >> > discussion >> > > ) >> > > https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available) >> > > https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available) >> > > https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available) >> > > >> > > >> > > On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang >> > > wrote: >> > > >> > > > +1 we should have the branch ASAP. >> > > > >> > > > On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka > > >> > > > wrote: >> > > > >> > > > > > Since blockers are not closed, I didn't cut the branch because >> > > > > multiple branches might confuse or sombody might miss to commit. >> > > > > >> > > > > The current situation is already confusing. The 3.3.1 version >> already >> > > > > exists in JIRA, so some committers wrongly commit non-critical >> issues >> > > to >> > > > > branch-3.3 and set the fix version to 3.3.1. >> > > > > I think now we should cut branch-3.3.0 and freeze source code >> except >> > > the >> > > > > blockers. >> > > > > >> > > > > -Akira >> > > > > >> > > > > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula < >> > > bra...@apache.org> >> > > > > wrote: >> > > > > >> > > > >> Sure, I will do that. >> > > > >> >> > > > >> Since blockers are not closed, I didn't cut the branch because >> > > > >> multiple branches might confuse or sombody might miss to >> > commit.Shall >> > > I >> > > > >> wait till this weekend to create..? >> > > > >> >> > > > >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka < >> aajis...@apache.org >> > > >> > > > >> wrote: >> > > > >> >> > > > >>> Hi Brahma, >> > > > >>> >> > > > >>> Thank you for preparing the release. >> > > > >>> Could you cut branch-3.3.0? I would like to backport some fixes >> for >> > > > >>> 3.3.1 and not for 3.3.0. >> > > > >>> >> > > > >>> Thanks and regards, >> > > > >>> Akira >> > > > >>> >> > > > >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula < >> > > > bra...@apache.org> >> > > > >>> wrote: >> > > > >>> >> > > > >>>> Hi All, >> > > > >>>> >> > > > >>>> we are down to two blockers issues now (YARN-10194 and >> YARN-9848) >> > > > which >> > > > >>>> are in patch available state.Hopefully we can out the RC soon. >> > > > >>>> >> > > > >>>> thanks to @Prabhu Joseph >> > > > ,@masakate,@akira >> > > > >>>> and @Wei-Chiu Chuang and others for >> > helping >> > > > >>>> resloving the blockers. >> > > > >>>> >
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
Ok. Done. Branch created. Following blockers are pending, will closely track this. https://issues.apache.org/jira/browse/HDFS-15287 ( Open: Under discussion ) https://issues.apache.org/jira/browse/YARN-10194 ( Patch Available) https://issues.apache.org/jira/browse/HDFS-15286 ( Patch Available) https://issues.apache.org/jira/browse/YARN-9898 ( Patch Available) On Fri, Apr 24, 2020 at 7:42 PM Wei-Chiu Chuang wrote: > +1 we should have the branch ASAP. > > On Wed, Apr 22, 2020 at 11:07 PM Akira Ajisaka > wrote: > > > > Since blockers are not closed, I didn't cut the branch because > > multiple branches might confuse or sombody might miss to commit. > > > > The current situation is already confusing. The 3.3.1 version already > > exists in JIRA, so some committers wrongly commit non-critical issues to > > branch-3.3 and set the fix version to 3.3.1. > > I think now we should cut branch-3.3.0 and freeze source code except the > > blockers. > > > > -Akira > > > > On Tue, Apr 21, 2020 at 3:05 PM Brahma Reddy Battula > > wrote: > > > >> Sure, I will do that. > >> > >> Since blockers are not closed, I didn't cut the branch because > >> multiple branches might confuse or sombody might miss to commit.Shall I > >> wait till this weekend to create..? > >> > >> On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka > >> wrote: > >> > >>> Hi Brahma, > >>> > >>> Thank you for preparing the release. > >>> Could you cut branch-3.3.0? I would like to backport some fixes for > >>> 3.3.1 and not for 3.3.0. > >>> > >>> Thanks and regards, > >>> Akira > >>> > >>> On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula < > bra...@apache.org> > >>> wrote: > >>> > >>>> Hi All, > >>>> > >>>> we are down to two blockers issues now (YARN-10194 and YARN-9848) > which > >>>> are in patch available state.Hopefully we can out the RC soon. > >>>> > >>>> thanks to @Prabhu Joseph > ,@masakate,@akira > >>>> and @Wei-Chiu Chuang and others for helping > >>>> resloving the blockers. > >>>> > >>>> > >>>> > >>>> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula < > >>>> bra...@apache.org> wrote: > >>>> > >>>>> > >>>>> @Prabhu Joseph > >>>>> >>> Have committed the YARN blocker YARN-10219 to trunk and > >>>>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras - > >>>>> YARN-10233 and HADOOP-16982 > >>>>> which i will help to review and commit. Thanks. > >>>>> > >>>>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as > a > >>>>> blockers. (without YARN-10233 we have given so many releases,it's > not newly > >>>>> introduced.).. Thanks > >>>>> > >>>>> @Vinod Kumar Vavilapalli ,@adam Antal, > >>>>> > >>>>> I noted YARN-9848 as a blocker as you mentioned above. > >>>>> > >>>>> @All, > >>>>> > >>>>> Currently following four blockers are pending for 3.3.0 RC. > >>>>> > >>>>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848. > >>>>> > >>>>> > >>>>> > >>>>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli < > >>>>> vino...@apache.org> wrote: > >>>>> > >>>>>> Looks like a really bad bug to me. > >>>>>> > >>>>>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should > >>>>>> also revert it in a 3.2 maintenance release too. > >>>>>> > >>>>>> Thanks > >>>>>> +Vinod > >>>>>> > >>>>>> > On Apr 14, 2020, at 5:03 PM, Adam Antal .INVALID> > >>>>>> wrote: > >>>>>> > > >>>>>> > Hi everyone, > >>>>>> > > >>>>>> > Sorry for coming a bit late with this, but there's also one jira > >>>>>> that can > >>>>>> > have potential impact on clusters and we should talk about it. > >>>>>> >
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
Sure, I will do that. Since blockers are not closed, I didn't cut the branch because multiple branches might confuse or sombody might miss to commit.Shall I wait till this weekend to create..? On Mon, Apr 20, 2020 at 11:57 AM Akira Ajisaka wrote: > Hi Brahma, > > Thank you for preparing the release. > Could you cut branch-3.3.0? I would like to backport some fixes for 3.3.1 > and not for 3.3.0. > > Thanks and regards, > Akira > > On Fri, Apr 17, 2020 at 11:11 AM Brahma Reddy Battula > wrote: > >> Hi All, >> >> we are down to two blockers issues now (YARN-10194 and YARN-9848) which >> are in patch available state.Hopefully we can out the RC soon. >> >> thanks to @Prabhu Joseph ,@masakate,@akira >> and @Wei-Chiu Chuang and others for helping >> resloving the blockers. >> >> >> >> On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula >> wrote: >> >>> >>> @Prabhu Joseph >>> >>> Have committed the YARN blocker YARN-10219 to trunk and >>> cherry-picked to branch-3.3. Right now, there are two blocker Jiras - >>> YARN-10233 and HADOOP-16982 >>> which i will help to review and commit. Thanks. >>> >>> Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a >>> blockers. (without YARN-10233 we have given so many releases,it's not newly >>> introduced.).. Thanks >>> >>> @Vinod Kumar Vavilapalli ,@adam Antal, >>> >>> I noted YARN-9848 as a blocker as you mentioned above. >>> >>> @All, >>> >>> Currently following four blockers are pending for 3.3.0 RC. >>> >>> HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848. >>> >>> >>> >>> On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli < >>> vino...@apache.org> wrote: >>> >>>> Looks like a really bad bug to me. >>>> >>>> +1 for revert and +1 for making that a 3.3.0 blocker. I think should >>>> also revert it in a 3.2 maintenance release too. >>>> >>>> Thanks >>>> +Vinod >>>> >>>> > On Apr 14, 2020, at 5:03 PM, Adam Antal >>>> wrote: >>>> > >>>> > Hi everyone, >>>> > >>>> > Sorry for coming a bit late with this, but there's also one jira that >>>> can >>>> > have potential impact on clusters and we should talk about it. >>>> > >>>> > Steven Rand found this problem earlier and commented to >>>> > https://issues.apache.org/jira/browse/YARN-4946. >>>> > The bug has impact on the RM state store: the RM does not delete apps >>>> - see >>>> > more details in his comment here: >>>> > >>>> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599 >>>> > . >>>> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848 >>>> with >>>> > the revert task). >>>> > >>>> > It might not be an actual blocker, but since there wasn't any >>>> consensus >>>> > about a follow up action, I thought we should decide how to proceed >>>> before >>>> > release 3.3.0. >>>> > >>>> > Regards, >>>> > Adam >>>> > >>>> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph < >>>> prabhujose.ga...@gmail.com> >>>> > wrote: >>>> > >>>> >> Thanks Brahma for the update. >>>> >> >>>> >> Have committed the YARN blocker YARN-10219 to trunk and >>>> cherry-picked to >>>> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and >>>> >> HADOOP-16982 >>>> >> which i will help to review and commit. Thanks. >>>> >> >>>> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png] >>>> >> >>>> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, >>>> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 >>>> ORDER >>>> >> BY priority DESC >>>> >> >>>> >> >>>> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula < >>>> bra...@apache.org> >>>> >>
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
Hi All, we are down to two blockers issues now (YARN-10194 and YARN-9848) which are in patch available state.Hopefully we can out the RC soon. thanks to @Prabhu Joseph ,@masakate,@akira and @Wei-Chiu Chuang and others for helping resloving the blockers. On Tue, Apr 14, 2020 at 10:49 PM Brahma Reddy Battula wrote: > > @Prabhu Joseph > >>> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked > to branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and > HADOOP-16982 > which i will help to review and commit. Thanks. > > Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a > blockers. (without YARN-10233 we have given so many releases,it's not newly > introduced.).. Thanks > > @Vinod Kumar Vavilapalli ,@adam Antal, > > I noted YARN-9848 as a blocker as you mentioned above. > > @All, > > Currently following four blockers are pending for 3.3.0 RC. > > HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848. > > > > On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli < > vino...@apache.org> wrote: > >> Looks like a really bad bug to me. >> >> +1 for revert and +1 for making that a 3.3.0 blocker. I think should also >> revert it in a 3.2 maintenance release too. >> >> Thanks >> +Vinod >> >> > On Apr 14, 2020, at 5:03 PM, Adam Antal >> wrote: >> > >> > Hi everyone, >> > >> > Sorry for coming a bit late with this, but there's also one jira that >> can >> > have potential impact on clusters and we should talk about it. >> > >> > Steven Rand found this problem earlier and commented to >> > https://issues.apache.org/jira/browse/YARN-4946. >> > The bug has impact on the RM state store: the RM does not delete apps - >> see >> > more details in his comment here: >> > >> https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599 >> > . >> > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848 >> with >> > the revert task). >> > >> > It might not be an actual blocker, but since there wasn't any consensus >> > about a follow up action, I thought we should decide how to proceed >> before >> > release 3.3.0. >> > >> > Regards, >> > Adam >> > >> > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph < >> prabhujose.ga...@gmail.com> >> > wrote: >> > >> >> Thanks Brahma for the update. >> >> >> >> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked >> to >> >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and >> >> HADOOP-16982 >> >> which i will help to review and commit. Thanks. >> >> >> >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png] >> >> >> >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, >> >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 >> ORDER >> >> BY priority DESC >> >> >> >> >> >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula < >> bra...@apache.org> >> >> wrote: >> >> >> >>> *Pending for 3.3.0 Release:* >> >>> >> >>> One Blocker(HADOOP-16963) confirmation and following jira's are open >> as >> >>> these needs to merged to other branches(I am tracking the same, >> Ideally >> >>> this can be closed and can raise seperate jira's to track). >> >>> >> >>> >> >>> 1–4 of 4Refresh results >> >>> < >> >>> >> https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC# >> >>>> >> >>> Columns >> >>> Patch InfoKeyTSummaryAssigneeReporterP >> StatusResolutionUpdatedDueCreated >> >>> <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353 >> >>> <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task] >> >>> <https://issues.apache.org/jira/browse/HDFS-14353> >> >>> >> >>> HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure >> >>> Coding: >>
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
@Prabhu Joseph >>> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked to branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and HADOOP-16982 which i will help to review and commit. Thanks. Looks you committed YARN-10219. Noted YARN-10233 and HADOOP-16982 as a blockers. (without YARN-10233 we have given so many releases,it's not newly introduced.).. Thanks @Vinod Kumar Vavilapalli ,@adam Antal, I noted YARN-9848 as a blocker as you mentioned above. @All, Currently following four blockers are pending for 3.3.0 RC. HADOOP-16963,YARN-10233,HADOOP-16982 and YARN-9848. On Tue, Apr 14, 2020 at 8:11 PM Vinod Kumar Vavilapalli wrote: > Looks like a really bad bug to me. > > +1 for revert and +1 for making that a 3.3.0 blocker. I think should also > revert it in a 3.2 maintenance release too. > > Thanks > +Vinod > > > On Apr 14, 2020, at 5:03 PM, Adam Antal > wrote: > > > > Hi everyone, > > > > Sorry for coming a bit late with this, but there's also one jira that can > > have potential impact on clusters and we should talk about it. > > > > Steven Rand found this problem earlier and commented to > > https://issues.apache.org/jira/browse/YARN-4946. > > The bug has impact on the RM state store: the RM does not delete apps - > see > > more details in his comment here: > > > https://issues.apache.org/jira/browse/YARN-4946?focusedCommentId=16898599&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16898599 > > . > > (FYI He also created https://issues.apache.org/jira/browse/YARN-9848 > with > > the revert task). > > > > It might not be an actual blocker, but since there wasn't any consensus > > about a follow up action, I thought we should decide how to proceed > before > > release 3.3.0. > > > > Regards, > > Adam > > > > On Tue, Apr 14, 2020 at 9:35 AM Prabhu Joseph < > prabhujose.ga...@gmail.com> > > wrote: > > > >> Thanks Brahma for the update. > >> > >> Have committed the YARN blocker YARN-10219 to trunk and cherry-picked to > >> branch-3.3. Right now, there are two blocker Jiras - YARN-10233 and > >> HADOOP-16982 > >> which i will help to review and commit. Thanks. > >> > >> [image: Screen Shot 2020-04-14 at 1.01.51 PM.png] > >> > >> project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, > >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 > ORDER > >> BY priority DESC > >> > >> > >> On Sun, Apr 12, 2020 at 12:19 AM Brahma Reddy Battula < > bra...@apache.org> > >> wrote: > >> > >>> *Pending for 3.3.0 Release:* > >>> > >>> One Blocker(HADOOP-16963) confirmation and following jira's are open as > >>> these needs to merged to other branches(I am tracking the same, Ideally > >>> this can be closed and can raise seperate jira's to track). > >>> > >>> > >>> 1–4 of 4Refresh results > >>> < > >>> > https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC# > >>>> > >>> Columns > >>> Patch InfoKeyTSummaryAssigneeReporterP > StatusResolutionUpdatedDueCreated > >>> <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353 > >>> <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task] > >>> <https://issues.apache.org/jira/browse/HDFS-14353> > >>> > >>> HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure > >>> Coding: > >>> metrics xmitsInProgress become to negative. > >>> <https://issues.apache.org/jira/browse/HDFS-14353> > >>> maobaolong > >>> < > https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong> > >>> maobaolong > >>> < > https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong> > >>> [image: > >>> Major] REOPENED *Unresolved* 11/Apr/20 11/Mar/19 Actions > >>> < > >>> > https://issues.apache.org/jira/rest/api/1.0/issues/13220750/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin > >>>> > >>> <https://issues.apache.org/jira/browse/HDFS-14788> HDFS-14788 > >>> <https://issues.a
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
Thanks Hexiaoqiao for taking care of this jira.Hope other's can do the needful. On Sun, Apr 12, 2020 at 11:25 AM Xiaoqiao He wrote: > Hi Brahma, > > Thanks for your great works here. > About HDFS-12862 <https://issues.apache.org/jira/browse/HDFS-12862>, I try > to raise another JIRA (HDFS-15272) to track backport branch-3.1, close this > one to decouple release progress. > > Thanks, > Hexiaoqiao > > On Sun, Apr 12, 2020 at 2:49 AM Brahma Reddy Battula > wrote: > > > *Pending for 3.3.0 Release:* > > > > One Blocker(HADOOP-16963) confirmation and following jira's are open as > > these needs to merged to other branches(I am tracking the same, Ideally > > this can be closed and can raise seperate jira's to track). > > > > > > 1–4 of 4Refresh results > > < > > > https://issues.apache.org/jira/issues/?jql=project%20in%20(%22Hadoop%20HDFS%22)%20AND%20resolution%20%3D%20Unresolved%20AND%20(cf%5B12310320%5D%20%3D%203.3.0%20OR%20fixVersion%20%3D%203.3.0)%20ORDER%20BY%20priority%20DESC# > > > > > Columns > > Patch InfoKeyTSummaryAssigneeReporterP StatusResolutionUpdatedDueCreated > > <https://issues.apache.org/jira/browse/HDFS-14353> HDFS-14353 > > <https://issues.apache.org/jira/browse/HDFS-14353> [image: Sub-task] > > <https://issues.apache.org/jira/browse/HDFS-14353> > > > > HDFS-8031 <https://issues.apache.org/jira/browse/HDFS-8031> Erasure > > Coding: > > metrics xmitsInProgress become to negative. > > <https://issues.apache.org/jira/browse/HDFS-14353> > > maobaolong > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong> > > maobaolong > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=maobaolong> > > [image: > > Major] REOPENED *Unresolved* 11/Apr/20 11/Mar/19 Actions > > < > > > https://issues.apache.org/jira/rest/api/1.0/issues/13220750/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin > > > > > <https://issues.apache.org/jira/browse/HDFS-14788> HDFS-14788 > > <https://issues.apache.org/jira/browse/HDFS-14788> [image: Improvement] > > <https://issues.apache.org/jira/browse/HDFS-14788> > > > > Use dynamic regex filter to ignore copy of source files in Distcp > > <https://issues.apache.org/jira/browse/HDFS-14788> > > Mukund Thakur > > < > https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mukund-thakur > > > > > Mukund > > Thakur > > < > https://issues.apache.org/jira/secure/ViewProfile.jspa?name=mukund-thakur > > > > > [image: > > Major] OPEN *Unresolved* 11/Apr/20 28/Aug/19 Actions > > < > > > https://issues.apache.org/jira/rest/api/1.0/issues/13253437/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin > > > > > <https://issues.apache.org/jira/browse/HDFS-12862> HDFS-12862 > > <https://issues.apache.org/jira/browse/HDFS-12862> [image: Bug] > > <https://issues.apache.org/jira/browse/HDFS-12862> > > > > CacheDirective becomes invalid when NN restart or failover > > <https://issues.apache.org/jira/browse/HDFS-12862> > > Wang XL > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Wang%20XL> > > Wang > > XL < > https://issues.apache.org/jira/secure/ViewProfile.jspa?name=Wang%20XL> > > [image: > > Major] PATCH AVAILABLE *Unresolved* 11/Apr/20 28/Nov/17 Actions > > < > > > https://issues.apache.org/jira/rest/api/1.0/issues/13121225/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin > > > > > <https://issues.apache.org/jira/browse/HDFS-14476> HDFS-14476 > > <https://issues.apache.org/jira/browse/HDFS-14476> [image: Improvement] > > <https://issues.apache.org/jira/browse/HDFS-14476> > > > > lock too long when fix inconsistent blocks between disk and in-memory > > <https://issues.apache.org/jira/browse/HDFS-14476> > > Sean Chow > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=seanlook> > > Sean > > Chow < > https://issues.apache.org/jira/secure/ViewProfile.jspa?name=seanlook > > > > > [image: > > Major] PATCH AVAILABLE *Unresolved* 12/Apr/20 07/May/19 Actions > > < > > > https://issues.apache.org/jira/rest/api/1.0/issues/13231926/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin > > > > > > > > > 1–5 of 5Re
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
/issues.apache.org/jira/secure/ViewProfile.jspa?name=gandras> Szilard Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth> [image: Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20 28/Nov/19 Actions <https://issues.apache.org/jira/rest/api/1.0/issues/13271201/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin> <https://issues.apache.org/jira/browse/YARN-10002> YARN-10002 <https://issues.apache.org/jira/browse/YARN-10002> [image: Improvement] <https://issues.apache.org/jira/browse/YARN-10002> Code cleanup and improvements in ConfigurationStoreBaseTest <https://issues.apache.org/jira/browse/YARN-10002> Benjamin Teke <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke> Szilard Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth> [image: Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20 28/Nov/19 Actions <https://issues.apache.org/jira/rest/api/1.0/issues/13271215/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin> <https://issues.apache.org/jira/browse/YARN-9998> YARN-9998 <https://issues.apache.org/jira/browse/YARN-9998> [image: Improvement] <https://issues.apache.org/jira/browse/YARN-9998> Code cleanup in LeveldbConfigurationStore <https://issues.apache.org/jira/browse/YARN-9998> Benjamin Teke <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=bteke> Szilard Nemeth <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=snemeth> [image: Minor] PATCH AVAILABLE *Unresolved* 11/Apr/20 28/Nov/19 Actions <https://issues.apache.org/jira/rest/api/1.0/issues/13271202/ActionsAndOperations?atl_token=A5KQ-2QAV-T4JA-FDED_194e108bac53dceebb1b88ae92ef65a9eba913b0_lin> <https://issues.apache.org/jira/browse/YARN-9354> YARN-9354 <https://issues.apache.org/jira/browse/YARN-9354> [image: Improvement] <https://issues.apache.org/jira/browse/YARN-9354> Resources sh <https://issues.apache.org/jira/browse/YARN-9354> On Thu, Apr 9, 2020 at 11:41 PM Brahma Reddy Battula wrote: > thanks a lot Akira and Prabhu for ping. > > All the blockers/critical issues are fixed. Recently two are reported > again like HADOOP-16955 and HADOOP-16963 waiting for confirmation. > > @prabhu, Those 156 jira's are non-blocker issues which needs to moved to > 3.4.0. and filter for blocker/critical is following. > > project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER > BY priority DESC > > > On Thu, Apr 9, 2020 at 12:09 PM Prabhu Joseph > wrote: > >> Hi Brahma, >> >> Please let me know if i can be of any help in review and commit / >> clear the pending issues in YARN and MapReduce targeted for 3.3.0. >> I see there are 156 YARN + MapReduce Unresolved Jiras which are either >> with >> target or fix version set to 3.3.0. >> >> project in (YARN, MAPREDUCE) AND resolution = Unresolved AND (cf[12310320] >> = 3.3.0 OR fixVersion = 3.3.0) ORDER BY priority DESC >> >> Thanks, >> Prabhu Joseph >> >> On Tue, Apr 7, 2020 at 1:27 PM Akira Ajisaka wrote: >> >> > Hi Brahma, >> > >> > How is this issue going? >> > If there are some blockers, I can help you. >> > >> > -Akira >> > >> > On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula > > >> > wrote: >> > >> > > Hi All, >> > > >> > > branch-3.3 created[1] and trunk set to 3.4.0-SNAPSHOT. >> > > >> > > while committing jira's please set the proper fix version and if >> there is >> > > any blocker/critical please commit to branch-3.3 or let me know. >> > > >> > > Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous >> > > communication in another thread[2]). >> > > >> > > >> > > 1.https://github.com/apache/hadoop/commits/branch-3.3 >> > > 2. >> https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2020-3 >> > > >> > > >> > > >> > > >> > > --Brahma Reddy Battula >> > > >> > >> > > > -- > > > > --Brahma Reddy Battula > -- --Brahma Reddy Battula
Re: [Hadoop-3.3 Release update]- branch-3.3 has created
thanks a lot Akira and Prabhu for ping. All the blockers/critical issues are fixed. Recently two are reported again like HADOOP-16955 and HADOOP-16963 waiting for confirmation. @prabhu, Those 156 jira's are non-blocker issues which needs to moved to 3.4.0. and filter for blocker/critical is following. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER BY priority DESC On Thu, Apr 9, 2020 at 12:09 PM Prabhu Joseph wrote: > Hi Brahma, > > Please let me know if i can be of any help in review and commit / > clear the pending issues in YARN and MapReduce targeted for 3.3.0. > I see there are 156 YARN + MapReduce Unresolved Jiras which are either with > target or fix version set to 3.3.0. > > project in (YARN, MAPREDUCE) AND resolution = Unresolved AND (cf[12310320] > = 3.3.0 OR fixVersion = 3.3.0) ORDER BY priority DESC > > Thanks, > Prabhu Joseph > > On Tue, Apr 7, 2020 at 1:27 PM Akira Ajisaka wrote: > > > Hi Brahma, > > > > How is this issue going? > > If there are some blockers, I can help you. > > > > -Akira > > > > On Mon, Mar 30, 2020 at 3:50 AM Brahma Reddy Battula > > wrote: > > > > > Hi All, > > > > > > branch-3.3 created[1] and trunk set to 3.4.0-SNAPSHOT. > > > > > > while committing jira's please set the proper fix version and if there > is > > > any blocker/critical please commit to branch-3.3 or let me know. > > > > > > Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous > > > communication in another thread[2]). > > > > > > > > > 1.https://github.com/apache/hadoop/commits/branch-3.3 > > > 2. > https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2020-3 > > > > > > > > > > > > > > > --Brahma Reddy Battula > > > > > > -- --Brahma Reddy Battula
Re: JENKINS is not posting build results in jiras
Thanks Akira!! On Wed, Mar 25, 2020 at 8:22 AM Sunil Govindan wrote: > Thanks Akira for helping. > > Best Regards, > Sunil > > On Wed, Mar 25, 2020 at 6:41 AM Akira Ajisaka wrote: > > > Sorry it was the root cause. Reverted. > > Now the precommit job can add comments to GitHub. > > https://github.com/apache/hadoop/pull/1820#issuecomment-603545202 > > > > Akira > > > > On Tue, Mar 24, 2020 at 8:53 PM Akira Ajisaka > wrote: > > > > > A few days ago I upgraded the build image and maybe it is the root > cause. > > > Created a pull request to revert the change to see if it is really the > > > cause: https://github.com/apache/hadoop/pull/1911 > > > > > > Thanks, > > > Akira > > > > > > On Tue, Mar 24, 2020 at 8:21 PM Akira Ajisaka > > wrote: > > > > > >> Thanks Sunil for the report. > > >> > > >> The precommit jobs are failing to add comments to GitHub also. > > >> I'm looking into the cause, and any help is appreciated. > > >> > > >> Thanks, > > >> Akira > > >> > > >> > > >> On Mon, Mar 23, 2020 at 7:39 PM Sunil Govindan > > wrote: > > >> > > >>> Hello > > >>> > > >>> From last few days, jenkins results are not getting published in > jiras. > > >>> > > >>> For eg, > > https://builds.apache.org/job/PreCommit-YARN-Build/25735/console > > >>> has > > >>> results. but I cannot see the same in YARN-9879. Similarly in the > cases > > >>> of YARN-10198 etc. > > >>> > > >>> I could see a message in console o/p as "Adding comment to JIRA". But > > its > > >>> not happening. > > >>> > > >>> Any ideas? > > >>> > > >>> Thanks > > >>> Sunil > > >>> > > >> > > > -- --Brahma Reddy Battula
[Hadoop-3.3 Release update]- branch-3.3 has created
Hi All, branch-3.3 created[1] and trunk set to 3.4.0-SNAPSHOT. while committing jira's please set the proper fix version and if there is any blocker/critical please commit to branch-3.3 or let me know. Planning start 3.3.0 RC0 voting on April-4th,2020(as per previous communication in another thread[2]). 1.https://github.com/apache/hadoop/commits/branch-3.3 2.https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2020-3 --Brahma Reddy Battula
Re: Hadoop 3.3 Release Plan Proposal
*Important update: *branch-3.3 has created. and now trunk is to target to 3.4.0 ( As per above plan). ( will send in another mail with subject and post in slack group also.) Please merge any critical or blocker issues to branch-3.3 also. Thanks. On Tue, Mar 24, 2020 at 11:34 PM Brahma Reddy Battula wrote: > > Critical and blockers issues came down[1]. > > Planning the cut branch-3.3 this weekend and start release vote on next > weekend. > > > > 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER > BY priority DESC > > > > On Thu, Feb 27, 2020 at 12:42 AM Brahma Reddy Battula > wrote: > >> Hi All, >> >> Inline with the original 3.3.0 communication proposal dated 8th Jan 2020, >> I would like to provide more updates[1]. >> >> We are approaching previously proposed code freeze date (March 10,2020). >> So I would like to cut 3.3 branch on *10th March* and point existing *trunk >> to 3.4* if there are no issues. >> >> *Current Release Plan:* >> >> *Feature freeze Date *: all features to merge by Feb 28, 2020.( it's >> almost done) >> *Code freeze Date *: blockers/critical only, no improvements and >> blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues >> are there which are very old and I am tracking same.) >> *Release Date*: March 15, 2020. >> >> *Please let me know if I missed anything.* >> >> 1. >> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0 >> 2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, >> Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER >> BY priority DESC >> >> >> On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula < >> brbapa...@gmail.com> wrote: >> >>> >>> Wiki was updated for 3.3 >>> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0 >>> . >>> >>> >>> >I'll move out anything that isn't needed. >>> >>> thanks steve. >>> >>> > We need to fix the shaded protobuf in >>> > Token issue to even get spark to compile. >>> >>> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621 >>> >>> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran >>> wrote: >>> >>>> > >>>> > 2. Features close to finish: >>>> > >>>> > >>>> > *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( >>>> owner >>>> > : Steve Loughran) >>>> > *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features & >>>> > fixes. ( owner : Steve Loughran) >>>> > *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. >>>> ( >>>> > owner : Steve Loughran) >>>> > >>>> > I'll move out anything that isn't needed. >>>> >>>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing >>>> and >>>> I've got some provisional tuning to do. That is -if things didn't work >>>> in >>>> the test/production deployments, I'd know about the regressions (e.g. >>>> HADOOP-16751). >>>> >>>> This is S3A and ABFS code -no idea about the rest, and inevitably the >>>> big >>>> JAR changes will have surprises. We need to fix the shaded protobuf in >>>> Token issue to even get spark to compile. >>>> >>>> -Steve >>>> >>>> > >>>> > >>>> >>> >>> >>> -- >>> --Brahma Reddy Battula >>> >> >> >> -- >> >> >> >> --Brahma Reddy Battula >> > > > -- > > > > --Brahma Reddy Battula > -- --Brahma Reddy Battula
Re: Hadoop 3.3 Release Plan Proposal
Critical and blockers issues came down[1]. Planning the cut branch-3.3 this weekend and start release vote on next weekend. 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER BY priority DESC On Thu, Feb 27, 2020 at 12:42 AM Brahma Reddy Battula wrote: > Hi All, > > Inline with the original 3.3.0 communication proposal dated 8th Jan 2020, > I would like to provide more updates[1]. > > We are approaching previously proposed code freeze date (March 10,2020). > So I would like to cut 3.3 branch on *10th March* and point existing *trunk > to 3.4* if there are no issues. > > *Current Release Plan:* > > *Feature freeze Date *: all features to merge by Feb 28, 2020.( it's > almost done) > *Code freeze Date *: blockers/critical only, no improvements and > blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues > are there which are very old and I am tracking same.) > *Release Date*: March 15, 2020. > > *Please let me know if I missed anything.* > > 1. > https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0 > 2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, > Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER > BY priority DESC > > > On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula > wrote: > >> >> Wiki was updated for 3.3 >> https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0. >> >> >> >I'll move out anything that isn't needed. >> >> thanks steve. >> >> > We need to fix the shaded protobuf in >> > Token issue to even get spark to compile. >> >> Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621 >> >> On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran >> wrote: >> >>> > >>> > 2. Features close to finish: >>> > >>> > >>> > *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( >>> owner >>> > : Steve Loughran) >>> > *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features & >>> > fixes. ( owner : Steve Loughran) >>> > *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. ( >>> > owner : Steve Loughran) >>> > >>> > I'll move out anything that isn't needed. >>> >>> FWIW, most of these are in CDP 1.x, so there's been reasonable testing >>> and >>> I've got some provisional tuning to do. That is -if things didn't work in >>> the test/production deployments, I'd know about the regressions (e.g. >>> HADOOP-16751). >>> >>> This is S3A and ABFS code -no idea about the rest, and inevitably the big >>> JAR changes will have surprises. We need to fix the shaded protobuf in >>> Token issue to even get spark to compile. >>> >>> -Steve >>> >>> > >>> > >>> >> >> >> -- >> --Brahma Reddy Battula >> > > > -- > > > > --Brahma Reddy Battula > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
thanks to all. will make this as optional..will update the wiki accordingly. On Wed, Mar 18, 2020 at 12:05 AM Vinayakumar B wrote: > Making ARM artifact optional, makes the release process simpler for RM and > unblocks release process (if there is unavailability of ARM resources). > > Still there are possible options to collaborate with RM ( as brahma > mentioned earlier) and provide ARM artifact may be before or after vote. > If feasible RM can decide to add ARM artifact by collaborating with @Brahma > Reddy Battula or me to get the ARM artifact. > > -Vinay > > On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal > wrote: > > > Thanks for the clarification Brahma. Can you update the proposal to state > > that it is optional (it may help to put the proposal on cwiki)? > > > > Also if we go ahead then the RM documentation should be clear this is an > > optional step. > > > > > > > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula > > wrote: > > > > > > Sure, we can't make mandatory while voting and we can upload to > downloads > > > once release vote is passed. > > > > > > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal > > > wrote: > > > > > >>> Sorry,didn't get you...do you mean, once release voting is > > >>> processed and upload by RM..? > > >> > > >> Yes, that is what I meant. I don’t want us to make more mandatory work > > for > > >> the release manager because the job is hard enough already. > > >> > > >> > > >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula < > bra...@apache.org> > > >> wrote: > > >>> > > >>> Sorry,didn't get you...do you mean, once release voting is processed > > and > > >>> upload by RM..? > > >>> > > >>> FYI. There is docker image for ARM also which support all scripts > > >>> (createrelease, start-build-env.sh, etc ). > > >>> > > >>> https://issues.apache.org/jira/browse/HADOOP-16797 > > >>> > > >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal > > >>> wrote: > > >>> > > >>>> Can ARM binaries be provided after the fact? We cannot increase the > > RM’s > > >>>> burden by asking them to generate an extra set of binaries. > > >>>> > > >>>> > > >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula < > > bra...@apache.org> > > >>>> wrote: > > >>>>> > > >>>>> + Dev mailing list. > > >>>>> > > >>>>> -- Forwarded message - > > >>>>> From: Brahma Reddy Battula > > >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM > > >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary > > >>>>> To: junping_du > > >>>>> > > >>>>> > > >>>>> thanks junping for your reply. > > >>>>> > > >>>>> bq. I think most of us in Hadoop community doesn't want to > have > > >>>> biased > > >>>>> on ARM or any other platforms. > > >>>>> > > >>>>> Yes, release voting will be based on the source code.AFAIK,Binary > we > > >> are > > >>>>> providing for user to easy to download and verify. > > >>>>> > > >>>>> bq. The only thing I try to understand is how much complexity > get > > >>>>> involved for our RM work. Does that potentially become a blocker > for > > >>>> future > > >>>>> releases? And how we can get rid of this risk. > > >>>>> > > >>>>> As I mentioned earlier, RM need to access the ARM machine(it will > be > > >>>>> donated and current qbt also using one ARM machine) and build tar > > using > > >>>> the > > >>>>> keys. As it can be common machine, RM can delete his keys once > > release > > >>>>> approved. > > >>>>> Can be sorted out as I mentioned earlier.(For accessing the ARM > > >> machine) > > >>>>> > > >>>>> bq. If you can list the concrete work that RM need to do > extra > > >> for > > >>>>> ARM rel
Re: [RESULT] [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1
My late +1.. I missed this.. On Thu, 19 Mar 2020 at 12:43 AM, Vinayakumar B wrote: > Vote passes with 4 +1s (3 binding, 1 non-binding) and no -1s > > Thanks everyone for verification. > I will push out bits soon. > > -Vinay > > > On Sun, Mar 15, 2020 at 5:25 PM Surendra Singh Lilhore < > surendralilh...@gmail.com> wrote: > > > +1 > > > > -Built trunk with -Dhadoop-thirdparty-protobuf.version=1.0.0 > > -Verified artifacts available in repo. > > > > > > Thanks > > Surendra > > > > > > On Fri, Mar 13, 2020 at 9:51 AM Akira Ajisaka > wrote: > > > >> +1 > >> > >> - Verified signatures and checksums > >> - Built jars and docs from source > >> - Built hadoop trunk with hadoop-thirdparty 1.0.0 > >> - Checked rat files and documents > >> - Checked LICENSE and NOTICE files > >> > >> Thanks, > >> Akira > >> > >> On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B > >> wrote: > >> > >> > Hi folks, > >> > > >> > Thanks to everyone's help on this release. > >> > > >> > I have re-created a release candidate (RC1) for Apache Hadoop > Thirdparty > >> > 1.0.0. > >> > > >> > RC Release artifacts are available at : > >> > > >> > > >> > http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/ > >> > > >> > Maven artifacts are available in staging repo: > >> > > >> > > >> > https://repository.apache.org/content/repositories/orgapachehadoop-1261/ > >> > > >> > The RC tag in git is here: > >> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1 > >> > > >> > And my public key is at: > >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > >> > > >> > *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm > >> IST.* > >> > > >> > For the testing, I have verified Hadoop trunk compilation with > >> >"-DdistMgmtSnapshotsUrl= > >> > > >> > https://repository.apache.org/content/repositories/orgapachehadoop-1261/ > >> > -Dhadoop-thirdparty-protobuf.version=1.0.0" > >> > > >> > My +1 to start. > >> > > >> > -Vinay > >> > > >> > > > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
Sure, I will update in cwiki,Once it's concluded here..Thanks a lot arpit... On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal wrote: > Thanks for the clarification Brahma. Can you update the proposal to state > that it is optional (it may help to put the proposal on cwiki)? > > Also if we go ahead then the RM documentation should be clear this is an > optional step. > > > > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula > wrote: > > > > Sure, we can't make mandatory while voting and we can upload to downloads > > once release vote is passed. > > > > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal > > wrote: > > > >>> Sorry,didn't get you...do you mean, once release voting is > >>> processed and upload by RM..? > >> > >> Yes, that is what I meant. I don’t want us to make more mandatory work > for > >> the release manager because the job is hard enough already. > >> > >> > >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula > >> wrote: > >>> > >>> Sorry,didn't get you...do you mean, once release voting is processed > and > >>> upload by RM..? > >>> > >>> FYI. There is docker image for ARM also which support all scripts > >>> (createrelease, start-build-env.sh, etc ). > >>> > >>> https://issues.apache.org/jira/browse/HADOOP-16797 > >>> > >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal > >>> wrote: > >>> > >>>> Can ARM binaries be provided after the fact? We cannot increase the > RM’s > >>>> burden by asking them to generate an extra set of binaries. > >>>> > >>>> > >>>>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula < > bra...@apache.org> > >>>> wrote: > >>>>> > >>>>> + Dev mailing list. > >>>>> > >>>>> -- Forwarded message - > >>>>> From: Brahma Reddy Battula > >>>>> Date: Tue, Mar 17, 2020 at 10:31 PM > >>>>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary > >>>>> To: junping_du > >>>>> > >>>>> > >>>>> thanks junping for your reply. > >>>>> > >>>>> bq. I think most of us in Hadoop community doesn't want to have > >>>> biased > >>>>> on ARM or any other platforms. > >>>>> > >>>>> Yes, release voting will be based on the source code.AFAIK,Binary we > >> are > >>>>> providing for user to easy to download and verify. > >>>>> > >>>>> bq. The only thing I try to understand is how much complexity get > >>>>> involved for our RM work. Does that potentially become a blocker for > >>>> future > >>>>> releases? And how we can get rid of this risk. > >>>>> > >>>>> As I mentioned earlier, RM need to access the ARM machine(it will be > >>>>> donated and current qbt also using one ARM machine) and build tar > using > >>>> the > >>>>> keys. As it can be common machine, RM can delete his keys once > release > >>>>> approved. > >>>>> Can be sorted out as I mentioned earlier.(For accessing the ARM > >> machine) > >>>>> > >>>>> bq. If you can list the concrete work that RM need to do extra > >> for > >>>>> ARM release, that would help us to better understand. > >>>>> > >>>>> I can write and update for future reference. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵 wrote: > >>>>> > >>>>>> Hi Brahma, > >>>>>> I think most of us in Hadoop community doesn't want to have biased > >>>> on > >>>>>> ARM or any other platforms. > >>>>>> The only thing I try to understand is how much complexity get > >>>>>> involved for our RM work. Does that potentially become a blocker for > >>>> future > >>>>>> releases? And how we can get rid of this risk. > >>>
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
Sure, we can't make mandatory while voting and we can upload to downloads once release vote is passed. On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal wrote: > > Sorry,didn't get you...do you mean, once release voting is > > processed and upload by RM..? > > Yes, that is what I meant. I don’t want us to make more mandatory work for > the release manager because the job is hard enough already. > > > > On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula > wrote: > > > > Sorry,didn't get you...do you mean, once release voting is processed and > > upload by RM..? > > > > FYI. There is docker image for ARM also which support all scripts > > (createrelease, start-build-env.sh, etc ). > > > > https://issues.apache.org/jira/browse/HADOOP-16797 > > > > On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal > > wrote: > > > >> Can ARM binaries be provided after the fact? We cannot increase the RM’s > >> burden by asking them to generate an extra set of binaries. > >> > >> > >>> On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula > >> wrote: > >>> > >>> + Dev mailing list. > >>> > >>> -- Forwarded message - > >>> From: Brahma Reddy Battula > >>> Date: Tue, Mar 17, 2020 at 10:31 PM > >>> Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary > >>> To: junping_du > >>> > >>> > >>> thanks junping for your reply. > >>> > >>> bq. I think most of us in Hadoop community doesn't want to have > >> biased > >>> on ARM or any other platforms. > >>> > >>> Yes, release voting will be based on the source code.AFAIK,Binary we > are > >>> providing for user to easy to download and verify. > >>> > >>> bq. The only thing I try to understand is how much complexity get > >>> involved for our RM work. Does that potentially become a blocker for > >> future > >>> releases? And how we can get rid of this risk. > >>> > >>> As I mentioned earlier, RM need to access the ARM machine(it will be > >>> donated and current qbt also using one ARM machine) and build tar using > >> the > >>> keys. As it can be common machine, RM can delete his keys once release > >>> approved. > >>> Can be sorted out as I mentioned earlier.(For accessing the ARM > machine) > >>> > >>> bq. If you can list the concrete work that RM need to do extra > for > >>> ARM release, that would help us to better understand. > >>> > >>> I can write and update for future reference. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Tue, Mar 17, 2020 at 10:41 AM 俊平堵 wrote: > >>> > >>>> Hi Brahma, > >>>>I think most of us in Hadoop community doesn't want to have biased > >> on > >>>> ARM or any other platforms. > >>>>The only thing I try to understand is how much complexity get > >>>> involved for our RM work. Does that potentially become a blocker for > >> future > >>>> releases? And how we can get rid of this risk. > >>>> If you can list the concrete work that RM need to do extra for ARM > >>>> release, that would help us to better understand. > >>>> > >>>> Thanks, > >>>> > >>>> Junping > >>>> > >>>> Akira Ajisaka 于2020年3月13日周五 上午12:34写道： > >>>> > >>>>> If you can provide ARM release for future releases, I'm fine with > that. > >>>>> > >>>>> Thanks, > >>>>> Akira > >>>>> > >>>>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula < > >> bra...@apache.org> > >>>>> wrote: > >>>>> > >>>>>> thanks Akira. > >>>>>> > >>>>>> Currently only problem is dedicated ARM for future RM.This i want to > >>>>> sort > >>>>>> out like below,if you've some other,please let me know. > >>>>>> > >>>>>> i) Single machine and share cred to future RM ( as we can delete > keys > >>>>> once > >>>>>> release is over). > >>
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
Sorry,didn't get you...do you mean, once release voting is processed and upload by RM..? FYI. There is docker image for ARM also which support all scripts (createrelease, start-build-env.sh, etc ). https://issues.apache.org/jira/browse/HADOOP-16797 On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal wrote: > Can ARM binaries be provided after the fact? We cannot increase the RM’s > burden by asking them to generate an extra set of binaries. > > > > On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula > wrote: > > > > + Dev mailing list. > > > > -- Forwarded message - > > From: Brahma Reddy Battula > > Date: Tue, Mar 17, 2020 at 10:31 PM > > Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary > > To: junping_du > > > > > > thanks junping for your reply. > > > > bq. I think most of us in Hadoop community doesn't want to have > biased > > on ARM or any other platforms. > > > > Yes, release voting will be based on the source code.AFAIK,Binary we are > > providing for user to easy to download and verify. > > > > bq. The only thing I try to understand is how much complexity get > > involved for our RM work. Does that potentially become a blocker for > future > > releases? And how we can get rid of this risk. > > > > As I mentioned earlier, RM need to access the ARM machine(it will be > > donated and current qbt also using one ARM machine) and build tar using > the > > keys. As it can be common machine, RM can delete his keys once release > > approved. > > Can be sorted out as I mentioned earlier.(For accessing the ARM machine) > > > > bq. If you can list the concrete work that RM need to do extra for > > ARM release, that would help us to better understand. > > > > I can write and update for future reference. > > > > > > > > > > > > > > > > > > > > On Tue, Mar 17, 2020 at 10:41 AM 俊平堵 wrote: > > > >> Hi Brahma, > >> I think most of us in Hadoop community doesn't want to have biased > on > >> ARM or any other platforms. > >> The only thing I try to understand is how much complexity get > >> involved for our RM work. Does that potentially become a blocker for > future > >> releases? And how we can get rid of this risk. > >> If you can list the concrete work that RM need to do extra for ARM > >> release, that would help us to better understand. > >> > >> Thanks, > >> > >> Junping > >> > >> Akira Ajisaka 于2020年3月13日周五 上午12:34写道： > >> > >>> If you can provide ARM release for future releases, I'm fine with that. > >>> > >>> Thanks, > >>> Akira > >>> > >>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula < > bra...@apache.org> > >>> wrote: > >>> > >>>> thanks Akira. > >>>> > >>>> Currently only problem is dedicated ARM for future RM.This i want to > >>> sort > >>>> out like below,if you've some other,please let me know. > >>>> > >>>> i) Single machine and share cred to future RM ( as we can delete keys > >>> once > >>>> release is over). > >>>> ii) Creating the jenkins project ( may be we need to discuss in the > >>>> board..) > >>>> iii) I can provide ARM release for future releases. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka > >>> wrote: > >>>> > >>>>> Hi Brahma, > >>>>> > >>>>> I think we cannot do any of your proposed actions. > >>>>> > >>>>> > >>>> > >>> > http://www.apache.org/legal/release-policy.html#owned-controlled-hardware > >>>>>> Strictly speaking, releases must be verified on hardware owned and > >>>>> controlled by the committer. That means hardware the committer has > >>>> physical > >>>>> possession and control of and exclusively full > >>> administrative/superuser > >>>>> access to. That's because only such hardware is qualified to hold a > >>> PGP > >>>>> private key, and the release should be verified on the machine the > >>>> private
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
Thanks Masatake!! I was aware of this thread which you given for reference as I am the source to discuss this(as I verified binary and given some comments). Please check following for same. https://lists.apache.org/list.html?common-dev@hadoop.apache.org:2017-7 AFAIK, that discussion whether we should vote ton he binary or not.Even Andrew discussed with legal team [1] and finally it was concluded that vote should only on source I think. 1. https://issues.apache.org/jira/browse/LEGAL-323 On Tue, Mar 17, 2020 at 11:23 AM Masatake Iwasaki < iwasak...@oss.nttdata.co.jp> wrote: > This thread seems to be relevant. > > https://lists.apache.org/thread.html/0d2a1b39f7e890c4f40be5fd92f107fbf048b936005901b7b53dd0f1%40%3Ccommon-dev.hadoop.apache.org%3E > > > Convenience binary artifacts are not official release artifacts and thus > > are not voted on. However, since they are distributed by Apache, they > are > > still subject to the same distribution requirements as official release > > artifacts. This means they need to have a LICENSE and NOTICE file, > follow > > ASF licensing rules, etc. The PMC needs to ensure that binary artifacts > > meet these requirements. > > > > However, being a "convenience" artifact doesn't mean it isn't important. > > The appropriate level of quality for binary artifacts is left up to the > > project. An OpenOffice person mentioned the quality of their binary > > artifacts is super important since very few of their users will compile > > their own office suite. > > > > I don't know if we've discussed the topic of binary artifact quality in > > Hadoop. My stance is that if we're going to publish something, it > should be > > good, or we shouldn't publish it at all. I think we do want to publish > > binary tarballs (it's the easiest way for new users to get started with > > Hadoop), so it's fair to consider them when evaluating a release. > > Just providing build machine to RM would not be enough if > PMC need to ensure that binary artifiacts meet these requirements. > > Thanks, > Masatake Iwasaki > > On 3/17/20 14:11, 俊平堵 wrote: > > Hi Brahma, > > I think most of us in Hadoop community doesn't want to have biased > on > > ARM or any other platforms. > > The only thing I try to understand is how much complexity get > involved > > for our RM work. Does that potentially become a blocker for future > > releases? And how we can get rid of this risk. > >If you can list the concrete work that RM need to do extra for ARM > > release, that would help us to better understand. > > > > Thanks, > > > > Junping > > > > Akira Ajisaka 于2020年3月13日周五 上午12:34写道： > > > >> If you can provide ARM release for future releases, I'm fine with that. > >> > >> Thanks, > >> Akira > >> > >> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula > > >> wrote: > >> > >>> thanks Akira. > >>> > >>> Currently only problem is dedicated ARM for future RM.This i want to > sort > >>> out like below,if you've some other,please let me know. > >>> > >>> i) Single machine and share cred to future RM ( as we can delete keys > >> once > >>> release is over). > >>> ii) Creating the jenkins project ( may be we need to discuss in the > >>> board..) > >>> iii) I can provide ARM release for future releases. > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka > >> wrote: > >>>> Hi Brahma, > >>>> > >>>> I think we cannot do any of your proposed actions. > >>>> > >>>> > >> > http://www.apache.org/legal/release-policy.html#owned-controlled-hardware > >>>>> Strictly speaking, releases must be verified on hardware owned and > >>>> controlled by the committer. That means hardware the committer has > >>> physical > >>>> possession and control of and exclusively full > administrative/superuser > >>>> access to. That's because only such hardware is qualified to hold a > PGP > >>>> private key, and the release should be verified on the machine the > >>> private > >>>> key lives on or on a machine as trusted as that. > >>>> > >>>> https://www.apache.org/dev/release-distribution.html#sigs-and-sums >
Fwd: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
+ Dev mailing list. -- Forwarded message - From: Brahma Reddy Battula Date: Tue, Mar 17, 2020 at 10:31 PM Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary To: junping_du thanks junping for your reply. bq. I think most of us in Hadoop community doesn't want to have biased on ARM or any other platforms. Yes, release voting will be based on the source code.AFAIK,Binary we are providing for user to easy to download and verify. bq. The only thing I try to understand is how much complexity get involved for our RM work. Does that potentially become a blocker for future releases? And how we can get rid of this risk. As I mentioned earlier, RM need to access the ARM machine(it will be donated and current qbt also using one ARM machine) and build tar using the keys. As it can be common machine, RM can delete his keys once release approved. Can be sorted out as I mentioned earlier.(For accessing the ARM machine) bq. If you can list the concrete work that RM need to do extra for ARM release, that would help us to better understand. I can write and update for future reference. On Tue, Mar 17, 2020 at 10:41 AM 俊平堵 wrote: > Hi Brahma, > I think most of us in Hadoop community doesn't want to have biased on > ARM or any other platforms. > The only thing I try to understand is how much complexity get > involved for our RM work. Does that potentially become a blocker for future > releases? And how we can get rid of this risk. > If you can list the concrete work that RM need to do extra for ARM > release, that would help us to better understand. > > Thanks, > > Junping > > Akira Ajisaka 于2020年3月13日周五 上午12:34写道： > >> If you can provide ARM release for future releases, I'm fine with that. >> >> Thanks, >> Akira >> >> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula >> wrote: >> >> > thanks Akira. >> > >> > Currently only problem is dedicated ARM for future RM.This i want to >> sort >> > out like below,if you've some other,please let me know. >> > >> > i) Single machine and share cred to future RM ( as we can delete keys >> once >> > release is over). >> > ii) Creating the jenkins project ( may be we need to discuss in the >> > board..) >> > iii) I can provide ARM release for future releases. >> > >> > >> > >> > >> > >> > >> > >> > On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka >> wrote: >> > >> > > Hi Brahma, >> > > >> > > I think we cannot do any of your proposed actions. >> > > >> > > >> > >> http://www.apache.org/legal/release-policy.html#owned-controlled-hardware >> > > > Strictly speaking, releases must be verified on hardware owned and >> > > controlled by the committer. That means hardware the committer has >> > physical >> > > possession and control of and exclusively full >> administrative/superuser >> > > access to. That's because only such hardware is qualified to hold a >> PGP >> > > private key, and the release should be verified on the machine the >> > private >> > > key lives on or on a machine as trusted as that. >> > > >> > > https://www.apache.org/dev/release-distribution.html#sigs-and-sums >> > > > Private keys MUST NOT be stored on any ASF machine. Likewise, >> > signatures >> > > for releases MUST NOT be created on ASF machines. >> > > >> > > We need to have dedicated physical ARM machines for each release >> manager, >> > > and now it is not feasible. >> > > If you provide an unofficial ARM binary release in some repository, >> > that's >> > > okay. >> > > >> > > -Akira >> > > >> > > On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula < >> bra...@apache.org> >> > > wrote: >> > > >> > >> Hello folks, >> > >> >> > >> As currently trunk will support ARM based compilation and qbt(1) is >> > >> running >> > >> from several months with quite stable, hence planning to propose ARM >> > >> binary >> > >> this time. >> > >> >> > >> ( Note : As we'll know voting will be based on the source,so this >> will >> > not >> > >> issue.) >> > >> >> > >> *Proposed Change:* >> > >> Currently in downloads we are keeping only x86 binary(2),Can we keep >> ARM >> > >> binary also.? >> > >> >> > >> *Actions:* >> > >> a) *Dedicated* *Machine*: >> > >>i) Dedicated ARM machine will be donated which I confirmed >> > >>ii) Or can use jenkins ARM machine itself which is currently >> used >> > >> for ARM >> > >> b) *Automate Release:* How about having one release project in >> > jenkins..? >> > >> So that future RM's just trigger the jenkin project. >> > >> >> > >> Please let me know your thoughts on this. >> > >> >> > >> >> > >> 1. >> > >> >> > >> >> > >> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ >> > >> 2.https://hadoop.apache.org/releases.html >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> --Brahma Reddy Battula >> > >> >> > > >> > >> > -- >> > >> > >> > >> > --Brahma Reddy Battula >> > >> > -- --Brahma Reddy Battula -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
thanks Akira. Currently only problem is dedicated ARM for future RM.This i want to sort out like below,if you've some other,please let me know. i) Single machine and share cred to future RM ( as we can delete keys once release is over). ii) Creating the jenkins project ( may be we need to discuss in the board..) iii) I can provide ARM release for future releases. On Thu, Mar 12, 2020 at 5:14 PM Akira Ajisaka wrote: > Hi Brahma, > > I think we cannot do any of your proposed actions. > > http://www.apache.org/legal/release-policy.html#owned-controlled-hardware > > Strictly speaking, releases must be verified on hardware owned and > controlled by the committer. That means hardware the committer has physical > possession and control of and exclusively full administrative/superuser > access to. That's because only such hardware is qualified to hold a PGP > private key, and the release should be verified on the machine the private > key lives on or on a machine as trusted as that. > > https://www.apache.org/dev/release-distribution.html#sigs-and-sums > > Private keys MUST NOT be stored on any ASF machine. Likewise, signatures > for releases MUST NOT be created on ASF machines. > > We need to have dedicated physical ARM machines for each release manager, > and now it is not feasible. > If you provide an unofficial ARM binary release in some repository, that's > okay. > > -Akira > > On Thu, Mar 12, 2020 at 7:57 PM Brahma Reddy Battula > wrote: > >> Hello folks, >> >> As currently trunk will support ARM based compilation and qbt(1) is >> running >> from several months with quite stable, hence planning to propose ARM >> binary >> this time. >> >> ( Note : As we'll know voting will be based on the source,so this will not >> issue.) >> >> *Proposed Change:* >> Currently in downloads we are keeping only x86 binary(2),Can we keep ARM >> binary also.? >> >> *Actions:* >> a) *Dedicated* *Machine*: >>i) Dedicated ARM machine will be donated which I confirmed >>ii) Or can use jenkins ARM machine itself which is currently used >> for ARM >> b) *Automate Release:* How about having one release project in jenkins..? >> So that future RM's just trigger the jenkin project. >> >> Please let me know your thoughts on this. >> >> >> 1. >> >> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ >> 2.https://hadoop.apache.org/releases.html >> >> >> >> >> >> >> --Brahma Reddy Battula >> > -- --Brahma Reddy Battula
[DISCUSS] Hadoop 3.3.0 Release include ARM binary
Hello folks, As currently trunk will support ARM based compilation and qbt(1) is running from several months with quite stable, hence planning to propose ARM binary this time. ( Note : As we'll know voting will be based on the source,so this will not issue.) *Proposed Change:* Currently in downloads we are keeping only x86 binary(2),Can we keep ARM binary also.? *Actions:* a) *Dedicated* *Machine*: i) Dedicated ARM machine will be donated which I confirmed ii) Or can use jenkins ARM machine itself which is currently used for ARM b) *Automate Release:* How about having one release project in jenkins..? So that future RM's just trigger the jenkin project. Please let me know your thoughts on this. 1. https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/ 2.https://hadoop.apache.org/releases.html --Brahma Reddy Battula
Re: Hadoop 3.3 Release Plan Proposal
Hi All, Inline with the original 3.3.0 communication proposal dated 8th Jan 2020, I would like to provide more updates[1]. We are approaching previously proposed code freeze date (March 10,2020). So I would like to cut 3.3 branch on *10th March* and point existing *trunk to 3.4* if there are no issues. *Current Release Plan:* *Feature freeze Date *: all features to merge by Feb 28, 2020.( it's almost done) *Code freeze Date *: blockers/critical only, no improvements and blocker/critical bug-fixes March 10, 2020 [2] ( as of now only 12 issues are there which are very old and I am tracking same.) *Release Date*: March 15, 2020. *Please let me know if I missed anything.* 1. https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0 2.project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, Critical) AND resolution = Unresolved AND "Target Version/s" = 3.3.0 ORDER BY priority DESC On Wed, Jan 22, 2020 at 11:22 PM Brahma Reddy Battula wrote: > > Wiki was updated for 3.3 > https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0. > > > >I'll move out anything that isn't needed. > > thanks steve. > > > We need to fix the shaded protobuf in > > Token issue to even get spark to compile. > > Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621 > > On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran > wrote: > >> > >> > 2. Features close to finish: >> > >> > >> > *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( >> owner >> > : Steve Loughran) >> > *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features & >> > fixes. ( owner : Steve Loughran) >> > *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. ( >> > owner : Steve Loughran) >> > >> > I'll move out anything that isn't needed. >> >> FWIW, most of these are in CDP 1.x, so there's been reasonable testing and >> I've got some provisional tuning to do. That is -if things didn't work in >> the test/production deployments, I'd know about the regressions (e.g. >> HADOOP-16751). >> >> This is S3A and ABFS code -no idea about the rest, and inevitably the big >> JAR changes will have surprises. We need to fix the shaded protobuf in >> Token issue to even get spark to compile. >> >> -Steve >> >> > >> > >> > > > -- > --Brahma Reddy Battula > -- --Brahma Reddy Battula
Re: Hadoop 3.3 Release Plan Proposal
Wiki was updated for 3.3 https://cwiki.apache.org/confluence/display/HADOOP/Roadmap#Roadmap-3.3.0. >I'll move out anything that isn't needed. thanks steve. > We need to fix the shaded protobuf in > Token issue to even get spark to compile. Looks this is done. https://issues.apache.org/jira/browse/HADOOP-16621 On Wed, Jan 8, 2020 at 7:41 PM Steve Loughran wrote: > > > > 2. Features close to finish: > > > > > > *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( > owner > > : Steve Loughran) > > *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features & > > fixes. ( owner : Steve Loughran) > > *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. ( > > owner : Steve Loughran) > > > > I'll move out anything that isn't needed. > > FWIW, most of these are in CDP 1.x, so there's been reasonable testing and > I've got some provisional tuning to do. That is -if things didn't work in > the test/production deployments, I'd know about the regressions (e.g. > HADOOP-16751). > > This is S3A and ABFS code -no idea about the rest, and inevitably the big > JAR changes will have surprises. We need to fix the shaded protobuf in > Token issue to even get spark to compile. > > -Steve > > > > > > -- --Brahma Reddy Battula
Hadoop 3.3 Release Plan Proposal
Hi All, To continue a faster cadence of releases to accommodate more features,we could plan a Hadoop 3.3 release around March Mid. To start the process sooner, and to establish a timeline, I propose to target Hadoop 3.3.0 release by March Mid 2020. (About 2 months from now). I would also would like to take this opportunity to come up with a detailed plan. Feature Freeze Date : All features should be merged by Feb 28, 2020. Code Freeze Date : blockers/critical only, no improvements and non blocker/critical bug-fixes March 10, 2020. Release Date: March 15, 2020 I have tried to come up with a list of features on my radar which could be candidates 1. Merged & Completed features: * HDFS-13891 HDFS RBF stabilization phase 1. (Owner: Brahma) * HDFS-12345: Scale testing HDFS NameNode with real metadata and workloads (Dynamometer) (owner: Erik Krogen) * HDFS-13762: Support non-volatile storage class memory(SCM) in HDFS cache directives ( owner: Feilong He) * HADOOP-16095 : Support impersonation for AuthenticationFilter (owner: Eric Yang) * YARN-7129: Application Catalog for YARN applications (Owner: Eric Yang) * YARN-5542: Scheduling of opportunistic containers (owner: Konstantinos Karanasos) * YARN-9473: Support Vector Engine ( a new accelerator hardware) based on pluggable device framework. (owner :Peter Bacsko) * YARN-9264: [Umbrella] Follow-up on IntelOpenCL FPGA plugin. (owner: Peter Bacsko) * YARN-9145: [Umbrella] Dynamically add or remove auxiliary services 2. Features close to finish: * HADOOP-13363 Upgrade protobuf from 2.5.0 to something newer (Owner: Vinay) *YARN-1011: Schedule containers based on utilization of currently allocated containers * (Owner: Haibo Chen) *YARN-9698: [Umbrella] Tools to help migration from Fair Scheduler to Capacity Scheduler. (Owner: Weiwei Yang) *YARN-9050: [Umbrella] Usability improvements for scheduler activities. (Owner: Tao Yang) *YARN-8851:[Umbrella] A pluggable device plugin framework to ease vendor plugin development (owner: Zhankun Tang) * YARN-9014: runC container runtime (owner : Eric Badger) *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( owner : Steve Loughran) *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features & fixes. ( owner : Steve Loughran) *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. ( owner : Steve Loughran) *HADOOP-15338: Support Java 11 LTS in Hadoop (owner: Akira Ajisaka) 3. Summary of Issues Status There are 1781 issues are fixed in 3.3.0(1) which very big number. 13 Blocker and critical issues are open(2),I will followup owners to get status on each of them to get in by code Freeze date. Please let me know if I missed any features targeted to 3.3 per this timeline. I would like to volunteer myself as release manager of 3.3.0 release. Please let me know if you have any suggestions. Reference: 1) project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = Fixed AND fixVersion = 3.3.0 2) project in (YARN, HADOOP, MAPREDUCE, HDFS) AND priority in (Blocker, Critical) AND resolution = Unresolved AND "TargetVersion/s" = 3.3.0 ORDER BY priority DESC Note: i) added the owners based on the jira assignee and reporter.. Please correct me ii) will update cwiki Regards, Brahma Reddy Battula
Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts
> can add some points here. > > > >> > > > > > >> > > Thoughts? > > > >> > > > > > > >> > > > .. Owen > > > >> > > > > > > >> > > > On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B < > > > >> vinayakum...@apache.org > > > >> > > > > > >> > > > wrote: > > > >> > > > > > > >> > > >> Hi All, > > > >> > > >> > > > >> > > >>I wanted to discuss about the separate repo for thirdparty > > > >> > > dependencies > > > >> > > >> which we need to shaded and include in Hadoop component's > jars. > > > >> > > >> > > > >> > > >>Apologies for the big text ahead, but this needs clear > > > >> > explanation!! > > > >> > > >> > > > >> > > >>Right now most needed such dependency is protobuf. > Protobuf > > > >> > > dependency > > > >> > > >> was not upgraded from 2.5.0 onwards with the fear that > > downstream > > > >> > > builds, > > > >> > > >> which depends on transitive dependency protobuf coming from > > > >> hadoop's > > > >> > > jars, > > > >> > > >> may fail with the upgrade. Apparently protobuf does not > > guarantee > > > >> > source > > > >> > > >> compatibility, though it guarantees wire compatibility > between > > > >> > versions. > > > >> > > >> Because of this behavior, version upgrade may cause breakage > in > > > >> known > > > >> > > and > > > >> > > >> unknown (private?) downstreams. > > > >> > > >> > > > >> > > >>So to tackle this, we came up the following proposal in > > > >> > HADOOP-13363. > > > >> > > >> > > > >> > > >>Luckily, As far as I know, no APIs, either public to user > or > > > >> > between > > > >> > > >> Hadoop processes, is not directly using protobuf classes in > > > >> > signatures. > > > >> > > >> (If > > > >> > > >> any exist, please let us know). > > > >> > > >> > > > >> > > >>Proposal: > > > >> > > >> > > > >> > > >> > > > >> > > >>1. Create a artifact(s) which contains shaded > dependencies. > > > All > > > >> > such > > > >> > > >> shading/relocation will be with known prefix > > > >> > > >> **org.apache.hadoop.thirdparty.**. > > > >> > > >>2. Right now protobuf jar (ex: > > > >> > > o.a.h.thirdparty:hadoop-shaded-protobuf) > > > >> > > >> to start with, all **com.google.protobuf** classes will be > > > >> relocated > > > >> > as > > > >> > > >> **org.apache.hadoop.thirdparty.com.google.protobuf**. > > > >> > > >>3. Hadoop modules, which needs protobuf as dependency, > will > > > add > > > >> > this > > > >> > > >> shaded artifact as dependency (ex: > > > >> > > >> o.a.h.thirdparty:hadoop-shaded-protobuf). > > > >> > > >>4. All previous usages of "com.google.protobuf" will be > > > >> relocated > > > >> > to > > > >> > > >> "org.apache.hadoop.thirdparty.com.google.protobuf" in the > code > > > and > > > >> > will > > > >> > > be > > > >> > > >> committed. Please note, this replacement is One-Time directly > > in > > > >> > source > > > >> > > >> code, NOT during compile and package. > > > >> > > >>5. Once all usages of "com.google.protobuf" is relocated, > > then > > > >> > hadoop > > > >> > > >> dont care about which version of original "protobuf-java" is > > in > > > >> > > >> dependency. > > > >> > > >>6. Just keep "protobuf-java:2.5.0" in dependency tree not > to > > > >> break > > > >> > > the > > > >> > > >> downstreams. But hadoop will be originally using the latest > > > >> protobuf > > > >> > > >> present in "o.a.h.thirdparty:hadoop-shaded-protobuf". > > > >> > > >> > > > >> > > >>7. Coming back to separate repo, Following are most > > > appropriate > > > >> > > reasons > > > >> > > >> of keeping shaded dependency artifact in separate repo > instead > > of > > > >> > > >> submodule. > > > >> > > >> > > > >> > > >> 7a. These artifacts need not be built all the time. It > > needs > > > >> to > > > >> > be > > > >> > > >> built only when there is a change in the dependency version > or > > > the > > > >> > build > > > >> > > >> process. > > > >> > > >> 7b. If added as "submodule in Hadoop repo", > > > >> > > maven-shade-plugin:shade > > > >> > > >> will execute only in package phase. That means, "mvn compile" > > or > > > >> "mvn > > > >> > > >> test-compile" will not be failed as this artifact will not > have > > > >> > > relocated > > > >> > > >> classes, instead it will have original classes, resulting in > > > >> > compilation > > > >> > > >> failure. Workaround, build thirdparty submodule first and > > exclude > > > >> > > >> "thirdparty" submodule in other executions. This will be a > > > complex > > > >> > > process > > > >> > > >> compared to keeping in a separate repo. > > > >> > > >> > > > >> > > >> 7c. Separate repo, will be a subproject of Hadoop, using > > the > > > >> > same > > > >> > > >> HADOOP jira project, with different versioning prefixed with > > > >> > > "thirdparty-" > > > >> > > >> (ex: thirdparty-1.0.0). > > > >> > > >> 7d. Separate will have same release process as Hadoop. > > > >> > > >> > > > >> > > >>HADOOP-13363 ( > > > >> https://issues.apache.org/jira/browse/HADOOP-13363) > > > >> > > is > > > >> > > >> an > > > >> > > >> umbrella jira tracking the changes to protobuf upgrade. > > > >> > > >> > > > >> > > >>PR (https://github.com/apache/hadoop-thirdparty/pull/1) > has > > > >> been > > > >> > > >> raised > > > >> > > >> for separate repo creation in (HADOOP-16595 ( > > > >> > > >> https://issues.apache.org/jira/browse/HADOOP-16595) > > > >> > > >> > > > >> > > >>Please provide your inputs for the proposal and review the > > PR > > > >> to > > > >> > > >> proceed with the proposal. > > > >> > > >> > > > >> > > >> > > > >> > > >-Thanks, > > > >> > > >>Vinay > > > >> > > >> > > > >> > > >> On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli < > > > >> > > >> vino...@apache.org> > > > >> > > >> wrote: > > > >> > > >> > > > >> > > >> > Moving the thread to the dev lists. > > > >> > > >> > > > > >> > > >> > Thanks > > > >> > > >> > +Vinod > > > >> > > >> > > > > >> > > >> > > On Sep 23, 2019, at 11:43 PM, Vinayakumar B < > > > >> > > vinayakum...@apache.org> > > > >> > > >> > wrote: > > > >> > > >> > > > > > >> > > >> > > Thanks Marton, > > > >> > > >> > > > > > >> > > >> > > Current created 'hadoop-thirdparty' repo is empty right > > now. > > > >> > > >> > > Whether to use that repo for shaded artifact or not will > > be > > > >> > > >> monitored in > > > >> > > >> > > HADOOP-13363 umbrella jira. Please feel free to join the > > > >> > discussion. > > > >> > > >> > > > > > >> > > >> > > There is no existing codebase is being moved out of > hadoop > > > >> repo. > > > >> > So > > > >> > > I > > > >> > > >> > think > > > >> > > >> > > right now we are good to go. > > > >> > > >> > > > > > >> > > >> > > -Vinay > > > >> > > >> > > > > > >> > > >> > > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek < > > > e...@apache.org> > > > >> > > wrote: > > > >> > > >> > > > > > >> > > >> > >> > > > >> > > >> > >> I am not sure if it's defined when is a vote required. > > > >> > > >> > >> > > > >> > > >> > >> https://www.apache.org/foundation/voting.html > > > >> > > >> > >> > > > >> > > >> > >> Personally I think it's a big enough change to send a > > > >> > notification > > > >> > > to > > > >> > > >> > the > > > >> > > >> > >> dev lists with a 'lazy consensus' closure > > > >> > > >> > >> > > > >> > > >> > >> Marton > > > >> > > >> > >> > > > >> > > >> > >> On 2019/09/23 17:46:37, Vinayakumar B < > > > >> vinayakum...@apache.org> > > > >> > > >> wrote: > > > >> > > >> > >>> Hi, > > > >> > > >> > >>> > > > >> > > >> > >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may > > be > > > >> more > > > >> > in > > > >> > > >> > >> future) > > > >> > > >> > >>> will be kept as a shaded artifact in a separate repo, > > which > > > >> will > > > >> > > be > > > >> > > >> > >>> referred as dependency in hadoop modules. This > approach > > > >> avoids > > > >> > > >> shading > > > >> > > >> > >> of > > > >> > > >> > >>> every submodule during build. > > > >> > > >> > >>> > > > >> > > >> > >>> So question is does any VOTE required before asking to > > > >> create a > > > >> > > git > > > >> > > >> > repo? > > > >> > > >> > >>> > > > >> > > >> > >>> On selfserve platform > > > >> > > https://gitbox.apache.org/setup/newrepo.html > > > >> > > >> > >>> I can access see that, requester should be PMC. > > > >> > > >> > >>> > > > >> > > >> > >>> Wanted to confirm here first. > > > >> > > >> > >>> > > > >> > > >> > >>> -Vinay > > > >> > > >> > >>> > > > >> > > >> > >> > > > >> > > >> > >> > > > >> > > > > > - > > > >> > > >> > >> To unsubscribe, e-mail: > > > private-unsubscr...@hadoop.apache.org > > > >> > > >> > >> For additional commands, e-mail: > > > >> private-h...@hadoop.apache.org > > > >> > > >> > >> > > > >> > > >> > >> > > > >> > > >> > > > > >> > > >> > > > > >> > > >> > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > -- --Brahma Reddy Battula
Re: [DISCUSS] drop mapreduce-pipes
+1 on deleting this . On Mon, 6 Jan 2020 at 7:39 PM, Steve Loughran wrote: > It has been many years since anyone did any work on mapreduce-pipes and > given the large number of hadoop-cluster streaming frameworks in the ASF > alone, probably the same number of years since anyone actually launched a > pipe job other than during testing. > > As such, it's a background noise maintenance problem and extra delays in > test runs. > > What do people think about deleting it? > > -Steve > -- --Brahma Reddy Battula
Re: [DISCUSS] Hadoop 2019 Release Planning
Thanks Akira for resuming this.. I am interested on 3.3 release ..will act as RM .will update the wiki as well.. On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota wrote: > I'm interested in doing a release of hadoop. > The version we need an RM is 3.1.3 right? What's the target date for that? > > Thanks, > Gabor > > On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka wrote: > > > Thank you Wangda. > > > > Now it's 2020. Let's release Hadoop 3.3.0. > > I created a wiki page for tracking blocker/critical issues for 3.3.0 and > > I'll check the issues in the list. > > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release > > If you find blocker/critical issues in trunk, please set the target > version > > to 3.3.0 for tracking. > > > > > We still need RM for 3.3.0 and 3.1.3. > > I can work as a release manager for 3.3.0. Is there anyone who wants to > be > > a RM? > > > > Thanks and regards, > > Akira > > > > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang > > wrote: > > > > > Thanks Wangda for bring this up! > > > > > > I ran the submarine 0.2.0 release before with a lot of help from folks > > > especially Sunil. :D > > > And this time I would like to help to release the 3.1.4. Thanks! > > > > > > BR, > > > Zhankun > > > > > > Hui Fei 于2019年8月16日 周五下午7:19写道： > > > > > > > Hi Wangda, > > > > Thanks for bringing this up! > > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is > a > > > > problem. > > > > Hope commiters watch and review these issues, Thanks > > > > https://issues.apache.org/jira/browse/HDFS-13596 > > > > https://issues.apache.org/jira/browse/HDFS-14396 > > > > > > > > Wangda Tan 于2019年8月10日周六 上午10:59写道： > > > > > > > > > Hi all, > > > > > > > > > > Hope this email finds you well > > > > > > > > > > I want to hear your thoughts about what should be the release plan > > for > > > > > 2019. > > > > > > > > > > In 2018, we released: > > > > > - 1 maintenance release of 2.6 > > > > > - 3 maintenance releases of 2.7 > > > > > - 3 maintenance releases of 2.8 > > > > > - 3 releases of 2.9 > > > > > - 4 releases of 3.0 > > > > > - 2 releases of 3.1 > > > > > > > > > > Total 16 releases in 2018. > > > > > > > > > > In 2019, by far we only have two releases: > > > > > - 1 maintenance release of 3.1 > > > > > - 1 minor release of 3.2. > > > > > > > > > > However, the community put a lot of efforts to stabilize features > of > > > > > various release branches. > > > > > There're: > > > > > - 217 fixed patches in 3.1.3 [1] > > > > > - 388 fixed patches in 3.2.1 [2] > > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!) > > > > > > > > > > I think it is the time to do maintenance releases of 3.1/3.2 and > do a > > > > minor > > > > > release for 3.3.0. > > > > > > > > > > In addition, I saw community discussion to do a 2.8.6 release for > > > > security > > > > > fixes. > > > > > > > > > > Any other releases? I think there're release plans for Ozone as > well. > > > And > > > > > please add your thoughts. > > > > > > > > > > Volunteers welcome! If you have interests to run a release as > Release > > > > > Manager (or co-Resource Manager), please respond to this email > thread > > > so > > > > we > > > > > can coordinate. > > > > > > > > > > Thanks, > > > > > Wangda Tan > > > > > > > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = > Fixed > > > AND > > > > > fixVersion = 3.1.3 > > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = > Fixed > > > AND > > > > > fixVersion = 3.2.1 > > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution = > Fixed > > > AND > > > > > fixVersion = 3.3.0 > > > > > > > > > > > > > > > -- --Brahma Reddy Battula
Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts
l not have > >> > > relocated > >> > > >> classes, instead it will have original classes, resulting in > >> > compilation > >> > > >> failure. Workaround, build thirdparty submodule first and exclude > >> > > >> "thirdparty" submodule in other executions. This will be a > complex > >> > > process > >> > > >> compared to keeping in a separate repo. > >> > > >> > >> > > >> 7c. Separate repo, will be a subproject of Hadoop, using the > >> > same > >> > > >> HADOOP jira project, with different versioning prefixed with > >> > > "thirdparty-" > >> > > >> (ex: thirdparty-1.0.0). > >> > > >> 7d. Separate will have same release process as Hadoop. > >> > > >> > >> > > >>HADOOP-13363 ( > >> https://issues.apache.org/jira/browse/HADOOP-13363) > >> > > is > >> > > >> an > >> > > >> umbrella jira tracking the changes to protobuf upgrade. > >> > > >> > >> > > >>PR (https://github.com/apache/hadoop-thirdparty/pull/1) has > >> been > >> > > >> raised > >> > > >> for separate repo creation in (HADOOP-16595 ( > >> > > >> https://issues.apache.org/jira/browse/HADOOP-16595) > >> > > >> > >> > > >>Please provide your inputs for the proposal and review the PR > >> to > >> > > >> proceed with the proposal. > >> > > >> > >> > > >> > >> > > >-Thanks, > >> > > >>Vinay > >> > > >> > >> > > >> On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli < > >> > > >> vino...@apache.org> > >> > > >> wrote: > >> > > >> > >> > > >> > Moving the thread to the dev lists. > >> > > >> > > >> > > >> > Thanks > >> > > >> > +Vinod > >> > > >> > > >> > > >> > > On Sep 23, 2019, at 11:43 PM, Vinayakumar B < > >> > > vinayakum...@apache.org> > >> > > >> > wrote: > >> > > >> > > > >> > > >> > > Thanks Marton, > >> > > >> > > > >> > > >> > > Current created 'hadoop-thirdparty' repo is empty right now. > >> > > >> > > Whether to use that repo for shaded artifact or not will be > >> > > >> monitored in > >> > > >> > > HADOOP-13363 umbrella jira. Please feel free to join the > >> > discussion. > >> > > >> > > > >> > > >> > > There is no existing codebase is being moved out of hadoop > >> repo. > >> > So > >> > > I > >> > > >> > think > >> > > >> > > right now we are good to go. > >> > > >> > > > >> > > >> > > -Vinay > >> > > >> > > > >> > > >> > > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek < > e...@apache.org> > >> > > wrote: > >> > > >> > > > >> > > >> > >> > >> > > >> > >> I am not sure if it's defined when is a vote required. > >> > > >> > >> > >> > > >> > >> https://www.apache.org/foundation/voting.html > >> > > >> > >> > >> > > >> > >> Personally I think it's a big enough change to send a > >> > notification > >> > > to > >> > > >> > the > >> > > >> > >> dev lists with a 'lazy consensus' closure > >> > > >> > >> > >> > > >> > >> Marton > >> > > >> > >> > >> > > >> > >> On 2019/09/23 17:46:37, Vinayakumar B < > >> vinayakum...@apache.org> > >> > > >> wrote: > >> > > >> > >>> Hi, > >> > > >> > >>> > >> > > >> > >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be > >> more > >> > in > >> > > >> > >> future) > >> > > >> > >>> will be kept as a shaded artifact in a separate repo, which > >> will > >> > > be > >> > > >> > >>> referred as dependency in hadoop modules. This approach > >> avoids > >> > > >> shading > >> > > >> > >> of > >> > > >> > >>> every submodule during build. > >> > > >> > >>> > >> > > >> > >>> So question is does any VOTE required before asking to > >> create a > >> > > git > >> > > >> > repo? > >> > > >> > >>> > >> > > >> > >>> On selfserve platform > >> > > https://gitbox.apache.org/setup/newrepo.html > >> > > >> > >>> I can access see that, requester should be PMC. > >> > > >> > >>> > >> > > >> > >>> Wanted to confirm here first. > >> > > >> > >>> > >> > > >> > >>> -Vinay > >> > > >> > >>> > >> > > >> > >> > >> > > >> > >> > >> > > > - > >> > > >> > >> To unsubscribe, e-mail: > private-unsubscr...@hadoop.apache.org > >> > > >> > >> For additional commands, e-mail: > >> private-h...@hadoop.apache.org > >> > > >> > >> > >> > > >> > >> > >> > > >> > > >> > > >> > > >> > > >> > >> > > > > >> > > > >> > > >> > > -- --Brahma Reddy Battula
[jira] [Resolved] (HADOOP-16310) Log of a slow RPC request should contain the parameter of the request
[
https://issues.apache.org/jira/browse/HADOOP-16310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula resolved HADOOP-16310.
---
Resolution: Duplicate
> Log of a slow RPC request should contain the parameter of the request
> -
>
> Key: HADOOP-16310
> URL: https://issues.apache.org/jira/browse/HADOOP-16310
> Project: Hadoop Common
> Issue Type: Improvement
> Components: rpc-server
>Affects Versions: 3.1.1, 2.7.7, 3.1.2
>Reporter: lindongdong
>Priority: Minor
>
> Now, the log of a slow RPC request just contains the
> *methodName*,*processingTime* and *client*. Code is here:
> {code:java}
> if ((rpcMetrics.getProcessingSampleCount() > minSampleSize) &&
> (processingTime > threeSigma)) {
> if(LOG.isWarnEnabled()) {
> String client = CurCall.get().toString();
> LOG.warn(
> "Slow RPC : " + methodName + " took " + processingTime +
> " milliseconds to process from client " + client);
> }
> rpcMetrics.incrSlowRpc();
> }{code}
>
> It is not enough to analyze why the RPC request is slow.
> The parameter of the request is a very important thing, and need to be logged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org
RE: [VOTE] Moving Submarine to a separate Apache project proposal
+1, Thanks for Wangda's proposal. I am interested to participate in this project. Please include me -Original Message- From: Wanqiang Ji [mailto:wanqiang...@gmail.com] Sent: Wednesday, September 04, 2019 6:53 PM To: Wangda Tan Cc: submarine-dev ; yarn-dev ; Hdfs-dev ; mapreduce-dev ; Hadoop Common ; private Subject: Re: [VOTE] Moving Submarine to a separate Apache project proposal +1 Thanks for Wangda's proposal. It is indeed amazing to see the growth and development of submarine. As the TLP will attract more developers to join. I will put more energy into it and contribute more feature. Look forward to the next change in submarine. Thanks, Wanqiang Ji On Wed, Sep 4, 2019 at 3:09 PM Bibin Chundatt wrote: > +1 > Thank you for the proposal. > I am interested in project. Please include me as well in the project. > > Thanks, > Bibin > > On Tue, Sep 3, 2019 at 6:41 PM Ayush Saxena > wrote: > > > +1 > > Thanx for the proposal. > > > > I would even like to participate in the project. > > Please add me as well. > > > > -Ayush > > > > > > > On 03-Sep-2019, at 6:00 PM, Vinayakumar B > > > > > wrote: > > > > > > +1 > > > > > > Thanks for the proposal. > > > Its very interesting project and looks very promising one by > > > looking at > > the > > > participations from various companies and the speed of development. > > > > > > I would also like to participate in the project. > > > Please add me as well. > > > > > > Thanks, > > > -Vinay > > > > > > On Tue, 3 Sep 2019, 12:38 pm Rakesh Radhakrishnan, > > > > > > > wrote: > > > > > >> +1, Thanks for the proposal. > > >> > > >> I am interested to participate in this project. Please include me > > >> as > > well > > >> in the project. > > >> > > >> Thanks, > > >> Rakesh > > >> > > >> On Tue, Sep 3, 2019 at 11:59 AM zhankun tang > > >> > > >> wrote: > > >> > > >>> +1 > > >>> > > >>> Thanks for Wangda's proposal. > > >>> > > >>> The submarine project is born within Hadoop, but not limited to > Hadoop. > > >> It > > >>> began with a trainer on YARN but it quickly realized that only a > > trainer > > >> is > > >>> not enough to meet the AI platform requirements. But now there's > > >>> no user-friendly open-source solution covers the whole AI > > >>> pipeline like > > data > > >>> engineering, training, and serving. And the underlying data > > >> infrastructure > > >>> itself is also evolving, for instance, many people love k8s. Not > > >> mentioning > > >>> there're many AI domain problems in this area to be solved. > > >>> It's almost for sure that building such an ML platform would > > >>> utilize various other open-source components taking ML into > > >>> consideration initially. > > >>> > > >>> I see submarine grows rapidly towards an enterprise-grade ML > > >>> platform > > >> which > > >>> could potentially enable AI ability for data engineer and scientist. > > This > > >>> is an exciting thing for both the community and the industry. > > >>> > > >>> BR, > > >>> Zhankun > > >>> > > >>> > > On Tue, 3 Sep 2019 at 13:34, Xun Liu wrote: > > > > +1 > > > > Hello everyone, I am a member of the submarine development team. > > I have been contributing to submarine for more than a year. > > I have seen the progress of submarine development very fast. > > In more than a year, there are 9 long-term developers of > > different companies. Contributing, submarine cumulative code > > has more than 200,000 lines of code, is > > >> growing > > very fast, > > and is used in the production environment of multiple companies. > > > > In the submarine development group, there are 5 PMCs and > > 7committer > > >>> members > > from Hadoop, spark, zeppelin projects. > > They are very familiar with the development process and > specifications > > >> of > > the apache community, > > and can well grasp the project development progress and project > > >> quality. > > So I recommend submarine to be a TLP project directly. > > > > We will continue to contribute to the submarine project. :-) > > > > Xun Liu > > Regards > > > > > On Tue, 3 Sep 2019 at 12:01, Devaraj K wrote: > > > > > > +1 > > > > > > Thanks Wangda for the proposal. > > > I would like to participate in this project, Please add me > > > also to > > >> the > > > project. > > > > > > Regards > > > Devaraj K > > > > > > On Mon, Sep 2, 2019 at 8:50 PM zac yuan > > > > > >> wrote: > > > > > >> +1 > > >> > > >> Submarine will be a complete solution for AI service development. > > >> It > > can > > >> take advantage of two best cluster systems: yarn and k8s, > > >> which > > >> will > > help > > >> more and more people get AI ability. To be a separate Apache > > >> project, > > > will > > >> accelerate the procedure of development apparently. > > >> > > >> Loo
Re: [Notice] Merge HDFS-13891(RBF) to trunk
Merged. thanks to Arpit for quoting the merge process. From: Brahma Reddy Battula Sent: Tuesday, June 25, 2019 12:39 AM To: hdfs-...@hadoop.apache.org; common-dev@hadoop.apache.org Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; Hdfs-dev; Iñigo Goiri; Hadoop Common Subject: [Notice] Merge HDFS-13891(RBF) to trunk Rebased HDFS-13891 against trunk and ran the jenkins (1).Only two four testcases are failing which flaky and there is already jira to track this. (HDFS-14461) Going to merge the branch soon which contains 77 commits,so please bear with us. 1.https://builds.apache.org/job/PreCommit-HDFS-Build/27044/console From: Brahma Reddy Battula Sent: Sunday, June 9, 2019 9:56 PM To: Xiaoqiao He Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; Hdfs-dev; Iñigo Goiri Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk Hi All, Given the positive response to the discussion thread [1], here is the formal vote thread to merge HDFS-13891 in to trunk. Summary of code changes: 1. Code changes for this branch are done in the hadoop-hdfs-rbf subproject, there is no impact to hadoop-hdfs and hadoop-common. 2. Added Security support for RBF 3. Added Missing Client Protocol API's 4. Bug Fixes/ Improvments The vote will run for 7 days, ending Sat June 15th. I will start this vote with my +1. Regards, Brahma Reddy Battula 1). https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > Thanks Brahma for starting the thread. > +1 for merging. > > He Xiaoqiao > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > wrote: > >> Thanks Brahma initiating this. >> +1(non-binding) for merge. >> >> @Uber we have almost all changes specially rbf security in production for >> a >> while now without issues. >> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < >> giovanni.fumar...@gmail.com> wrote: >> >> > +1 on merging. >> > >> > Thanks Brahma for starting the thread. >> > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri wrote: >> > >> > > Thank you Brahma for pushing this. >> > > >> > > As you mentioned, we have already taken most of the changes into >> > > production. >> > > I want to highlight that the main contribution is the addition of >> > security. >> > > We have been able to test this at a smaller scale (~500 servers and 4 >> > > subclusters) and the performance is great with our current ZooKeeper >> > > deployment. >> > > I would also like to highlight that all the changes are constrained to >> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS. >> > > >> > > +1 on merging >> > > >> > > Inigo >> > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka >> > wrote: >> > > >> > > > Thanks Brahma for starting the discussion. >> > > > I'm +1 for merging this. >> > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster >> > > > with 2 routers (not in production) and running several tests. >> > > > >> > > > Regards, >> > > > Akira >> > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < >> > bra...@apache.org> >> > > > wrote: >> > > > > >> > > > > Dear Hadoop Developers >> > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. >> We >> > > have >> > > > > been working on this feature from last several months. >> > > > > This feature work received the contributions from different >> > companies. >> > > > All >> > > > > of the feature development happened smoothly and collaboratively >> in >> > > > JIRAs. >> > > > > >> > > > > Kindly do take a look at the branch and raise issues/concerns that >> > need >> > > > to >> > > > > be addressed before the merge. >> > > > > >> > > > > *Highlights of HDFS-13891 Branch:* >> > > > > = >> > > > > >> > > > > Adding Security to RBF(1) >> > > > &g
[Notice] Merge HDFS-13891(RBF) to trunk
Rebased HDFS-13891 against trunk and ran the jenkins (1).Only two four testcases are failing which flaky and there is already jira to track this. (HDFS-14461) Going to merge the branch soon which contains 77 commits,so please bear with us. 1.https://builds.apache.org/job/PreCommit-HDFS-Build/27044/console From: Brahma Reddy Battula Sent: Sunday, June 9, 2019 9:56 PM To: Xiaoqiao He Cc: Akira Ajisaka; Chittaranjan Hota; Giovanni Matteo Fumarola; Hadoop Common; Hdfs-dev; Iñigo Goiri Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk Hi All, Given the positive response to the discussion thread [1], here is the formal vote thread to merge HDFS-13891 in to trunk. Summary of code changes: 1. Code changes for this branch are done in the hadoop-hdfs-rbf subproject, there is no impact to hadoop-hdfs and hadoop-common. 2. Added Security support for RBF 3. Added Missing Client Protocol API's 4. Bug Fixes/ Improvments The vote will run for 7 days, ending Sat June 15th. I will start this vote with my +1. Regards, Brahma Reddy Battula 1). https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > Thanks Brahma for starting the thread. > +1 for merging. > > He Xiaoqiao > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > wrote: > >> Thanks Brahma initiating this. >> +1(non-binding) for merge. >> >> @Uber we have almost all changes specially rbf security in production for >> a >> while now without issues. >> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < >> giovanni.fumar...@gmail.com> wrote: >> >> > +1 on merging. >> > >> > Thanks Brahma for starting the thread. >> > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri wrote: >> > >> > > Thank you Brahma for pushing this. >> > > >> > > As you mentioned, we have already taken most of the changes into >> > > production. >> > > I want to highlight that the main contribution is the addition of >> > security. >> > > We have been able to test this at a smaller scale (~500 servers and 4 >> > > subclusters) and the performance is great with our current ZooKeeper >> > > deployment. >> > > I would also like to highlight that all the changes are constrained to >> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS. >> > > >> > > +1 on merging >> > > >> > > Inigo >> > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka >> > wrote: >> > > >> > > > Thanks Brahma for starting the discussion. >> > > > I'm +1 for merging this. >> > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster >> > > > with 2 routers (not in production) and running several tests. >> > > > >> > > > Regards, >> > > > Akira >> > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < >> > bra...@apache.org> >> > > > wrote: >> > > > > >> > > > > Dear Hadoop Developers >> > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. >> We >> > > have >> > > > > been working on this feature from last several months. >> > > > > This feature work received the contributions from different >> > companies. >> > > > All >> > > > > of the feature development happened smoothly and collaboratively >> in >> > > > JIRAs. >> > > > > >> > > > > Kindly do take a look at the branch and raise issues/concerns that >> > need >> > > > to >> > > > > be addressed before the merge. >> > > > > >> > > > > *Highlights of HDFS-13891 Branch:* >> > > > > = >> > > > > >> > > > > Adding Security to RBF(1) >> > > > > Adding Missing Client API's(2) >> > > > > Improvements/Bug Fixing >> > > > > Critical - HDFS-13637, HDFS-13834 >> > > > > >> > > > > *Commits:* >> > > > > >> > > > > >> > > > > No of JIRAs Resolved: 72 >> > > > > >> > > >
[RESULT][VOTE] Merge HDFS-13891(RBF) to trunk
Vote is passed with 3 binding +1's and no -1 ( 5 nonbinding votes) . thanks to all for voting. Will try to merge this week.. On Tue, Jun 11, 2019 at 1:13 PM Takanobu Asanuma wrote: > +1(non-binding). > > Regards, > - Takanobu > > > From: Xiaoqiao He > Sent: Monday, June 10, 2019 15:47 > To: Ranith Sardar > Cc: Brahma Reddy Battula; Hadoop Common; Hdfs-dev > Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk > > +1 (non-binding) > > - Try to merge branch HDFS-13891(RBF) to trunk at local and no conflict or > failure. > - Built from merged sources. > - Ran 85 RBF test class at local and result shows: (Tests run: 639, > Failures: 2, Errors: 2, Skipped: 2) - failed tests include > #TestRouterWithSecureStartup and #TestRouterHttpDelegationToken. I don't > think it is blocked issue. > > Thanks Brahma for organizing the merge. > > > On Mon, Jun 10, 2019 at 12:27 PM Ranith Sardar > wrote: > > > +1 (Non-binding) > > > > -Original Message- > > From: Brahma Reddy Battula [mailto:bra...@apache.org] > > Sent: 09 June 2019 19:31 > > To: Hadoop Common ; Hdfs-dev < > > hdfs-...@hadoop.apache.org> > > Subject: [VOTE] Merge HDFS-13891(RBF) to trunk > > > > Updated mail... > > > > -- Forwarded message - > > From: Brahma Reddy Battula > > Date: Sun, Jun 9, 2019 at 7:26 PM > > Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk > > To: Xiaoqiao He > > Cc: Akira Ajisaka , Chittaranjan Hota < > > chitts.h...@gmail.com>, Giovanni Matteo Fumarola < > > giovanni.fumar...@gmail.com>, Hadoop Common < > common-dev@hadoop.apache.org>, > > Hdfs-dev , Iñigo Goiri > > > > > > Hi All, > > > > Given the positive response to the discussion thread [1], here is the > > formal vote thread to merge HDFS-13891 in to trunk. > > > > Summary of code changes: > > 1. Code changes for this branch are done in the hadoop-hdfs-rbf > > subproject, there is no impact to hadoop-hdfs and hadoop-common. > > 2. Added Security support for RBF > > 3. Added Missing Client Protocol API's > > 4. Bug Fixes/ Improvments > > > > > > The vote will run for 7 days, ending Sat June 15th. I will start > this > > vote with my +1. > > > > Regards, > > Brahma Reddy Battula > > > > 1). > > > > > https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E > > > > > > On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > > > > > Thanks Brahma for starting the thread. > > > +1 for merging. > > > > > > He Xiaoqiao > > > > > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > > > > > > wrote: > > > > > >> Thanks Brahma initiating this. > > >> +1(non-binding) for merge. > > >> > > >> @Uber we have almost all changes specially rbf security in production > > >> for a while now without issues. > > >> > > >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < > > >> giovanni.fumar...@gmail.com> wrote: > > >> > > >> > +1 on merging. > > >> > > > >> > Thanks Brahma for starting the thread. > > >> > > > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri > > wrote: > > >> > > > >> > > Thank you Brahma for pushing this. > > >> > > > > >> > > As you mentioned, we have already taken most of the changes into > > >> > > production. > > >> > > I want to highlight that the main contribution is the addition of > > >> > security. > > >> > > We have been able to test this at a smaller scale (~500 servers > > >> > > and 4 > > >> > > subclusters) and the performance is great with our current > > >> > > ZooKeeper deployment. > > >> > > I would also like to highlight that all the changes are > > >> > > constrained to hadoop-hdfs-rbf and there is no differences in > > commons or HDFS. > > >> > > > > >> > > +1 on merging > > >> > > > > >> > > Inigo > > >> > > > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka > > >> > > > > >> > wrote: > > >> > > > &g
Re: [DISCUSS] Merge HDFS-13891(RBF) to trunk
Dear Arpit, Thanks for taking look into it. ECBlockGroupStats.merge() is Utility method which is moved to hadoop-hdfs-client module. Ideally it could have been seperate jira. But this changes will not induce any issues, will take necessary action for this. On Mon, Jun 10, 2019 at 8:40 PM Arpit Agarwal wrote: > I scanned the merge payload for changes to non-RBF code. The changes are > minimal, which is good. > > The only commit that I didn’t understand was: > https://issues.apache.org/jira/browse/HDFS-14268 > > The jira description doesn’t make it clear why ECBlockGroupStats is > modified. > > +0 apart from that. > > > On Jun 1, 2019, at 8:40 PM, Brahma Reddy Battula > wrote: > > Dear Hadoop Developers > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have > been working on this feature from last several months. > This feature work received the contributions from different companies. All > of the feature development happened smoothly and collaboratively in JIRAs. > > Kindly do take a look at the branch and raise issues/concerns that need to > be addressed before the merge. > > *Highlights of HDFS-13891 Branch:* > = > > Adding Security to RBF(1) > Adding Missing Client API's(2) > Improvements/Bug Fixing > Critical - HDFS-13637, HDFS-13834 > > *Commits:* > > > No of JIRAs Resolved: 72 > > All this commits are in RBF Module. No changes in hdfs/common. > > *Tested Cluster:* > = > > Most of these changes verified at Uber,Microsoft,Huawei and some other > compaines. > > *Uber*: Most changes are running in production @Uber including the critical > security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers. > Zookeeper as a state store to hold delegation tokens were also stress > tested to hold more than 2 Million tokens. --CR Hota > > *MicroSoft*: Most of these changes are currently running in production at > Microsoft.The security has also been tested in a 500 server cluster with 4 > subclsuters. --Inigo Goiri > > *Huawei* : Deployed all this changes in 20 node cluster with 3 > routers.Planning deploy 10K production cluster. > > *Contributors:* > === > > Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham > Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang > Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood > Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and > contributing to this. > > *Future Tasks:* > ==== > > will cleanup the jira's under this umbrella and contiue to work. > > Reference: > 1) https://issues.apache.org/jira/browse/HDFS-13532 > 2) https://issues.apache.org/jira/browse/HDFS-13655 > > > > > --Brahma Reddy Battula > > > -- --Brahma Reddy Battula
[VOTE] Merge HDFS-13891(RBF) to trunk
Updated mail... -- Forwarded message - From: Brahma Reddy Battula Date: Sun, Jun 9, 2019 at 7:26 PM Subject: Re: [VOTE] Merge HDFS-13891(RBF) to trunk To: Xiaoqiao He Cc: Akira Ajisaka , Chittaranjan Hota < chitts.h...@gmail.com>, Giovanni Matteo Fumarola < giovanni.fumar...@gmail.com>, Hadoop Common , Hdfs-dev , Iñigo Goiri Hi All, Given the positive response to the discussion thread [1], here is the formal vote thread to merge HDFS-13891 in to trunk. Summary of code changes: 1. Code changes for this branch are done in the hadoop-hdfs-rbf subproject, there is no impact to hadoop-hdfs and hadoop-common. 2. Added Security support for RBF 3. Added Missing Client Protocol API's 4. Bug Fixes/ Improvments The vote will run for 7 days, ending Sat June 15th. I will start this vote with my +1. Regards, Brahma Reddy Battula 1). https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > Thanks Brahma for starting the thread. > +1 for merging. > > He Xiaoqiao > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > wrote: > >> Thanks Brahma initiating this. >> +1(non-binding) for merge. >> >> @Uber we have almost all changes specially rbf security in production for >> a >> while now without issues. >> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < >> giovanni.fumar...@gmail.com> wrote: >> >> > +1 on merging. >> > >> > Thanks Brahma for starting the thread. >> > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri wrote: >> > >> > > Thank you Brahma for pushing this. >> > > >> > > As you mentioned, we have already taken most of the changes into >> > > production. >> > > I want to highlight that the main contribution is the addition of >> > security. >> > > We have been able to test this at a smaller scale (~500 servers and 4 >> > > subclusters) and the performance is great with our current ZooKeeper >> > > deployment. >> > > I would also like to highlight that all the changes are constrained to >> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS. >> > > >> > > +1 on merging >> > > >> > > Inigo >> > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka >> > wrote: >> > > >> > > > Thanks Brahma for starting the discussion. >> > > > I'm +1 for merging this. >> > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster >> > > > with 2 routers (not in production) and running several tests. >> > > > >> > > > Regards, >> > > > Akira >> > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < >> > bra...@apache.org> >> > > > wrote: >> > > > > >> > > > > Dear Hadoop Developers >> > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. >> We >> > > have >> > > > > been working on this feature from last several months. >> > > > > This feature work received the contributions from different >> > companies. >> > > > All >> > > > > of the feature development happened smoothly and collaboratively >> in >> > > > JIRAs. >> > > > > >> > > > > Kindly do take a look at the branch and raise issues/concerns that >> > need >> > > > to >> > > > > be addressed before the merge. >> > > > > >> > > > > *Highlights of HDFS-13891 Branch:* >> > > > > = >> > > > > >> > > > > Adding Security to RBF(1) >> > > > > Adding Missing Client API's(2) >> > > > > Improvements/Bug Fixing >> > > > > Critical - HDFS-13637, HDFS-13834 >> > > > > >> > > > > *Commits:* >> > > > > >> > > > > >> > > > > No of JIRAs Resolved: 72 >> > > > > >> > > > > All this commits are in RBF Module. No changes in hdfs/common. >> > > > > >> > > > > *Tested Cluster:* >> > > > > = >> > > > > >>
Re: [VOTE] Merge HDFS-13891(RBF) to trunk
Hi All, Given the positive response to the discussion thread [1], here is the formal vote thread to merge HDFS-13891 in to trunk. Summary of code changes: 1. Code changes for this branch are done in the hadoop-hdfs-rbf subproject, there is no impact to hadoop-hdfs and hadoop-common. 2. Added Security support for RBF 3. Added Missing Client Protocol API's 4. Bug Fixes/ Improvments The vote will run for 7 days, ending Sat June 15th. I will start this vote with my +1. Regards, Brahma Reddy Battula 1). https://lists.apache.org/thread.html/cdc2e084874b30bf6af2dd827bcbdba4ab8d3d983a8b9796e61608e6@%3Ccommon-dev.hadoop.apache.org%3E On Wed, Jun 5, 2019 at 3:44 PM Xiaoqiao He wrote: > Thanks Brahma for starting the thread. > +1 for merging. > > He Xiaoqiao > > On Tue, Jun 4, 2019 at 1:53 PM Chittaranjan Hota > wrote: > >> Thanks Brahma initiating this. >> +1(non-binding) for merge. >> >> @Uber we have almost all changes specially rbf security in production for >> a >> while now without issues. >> >> On Mon, Jun 3, 2019 at 12:56 PM Giovanni Matteo Fumarola < >> giovanni.fumar...@gmail.com> wrote: >> >> > +1 on merging. >> > >> > Thanks Brahma for starting the thread. >> > >> > On Mon, Jun 3, 2019 at 10:00 AM Iñigo Goiri wrote: >> > >> > > Thank you Brahma for pushing this. >> > > >> > > As you mentioned, we have already taken most of the changes into >> > > production. >> > > I want to highlight that the main contribution is the addition of >> > security. >> > > We have been able to test this at a smaller scale (~500 servers and 4 >> > > subclusters) and the performance is great with our current ZooKeeper >> > > deployment. >> > > I would also like to highlight that all the changes are constrained to >> > > hadoop-hdfs-rbf and there is no differences in commons or HDFS. >> > > >> > > +1 on merging >> > > >> > > Inigo >> > > >> > > On Sun, Jun 2, 2019 at 10:19 PM Akira Ajisaka >> > wrote: >> > > >> > > > Thanks Brahma for starting the discussion. >> > > > I'm +1 for merging this. >> > > > >> > > > FYI: In Yahoo! JAPAN, deployed all these changes in 20 nodes cluster >> > > > with 2 routers (not in production) and running several tests. >> > > > >> > > > Regards, >> > > > Akira >> > > > >> > > > On Sun, Jun 2, 2019 at 12:40 PM Brahma Reddy Battula < >> > bra...@apache.org> >> > > > wrote: >> > > > > >> > > > > Dear Hadoop Developers >> > > > > >> > > > > I would like to propose RBF Branch (HDFS-13891) merge into trunk. >> We >> > > have >> > > > > been working on this feature from last several months. >> > > > > This feature work received the contributions from different >> > companies. >> > > > All >> > > > > of the feature development happened smoothly and collaboratively >> in >> > > > JIRAs. >> > > > > >> > > > > Kindly do take a look at the branch and raise issues/concerns that >> > need >> > > > to >> > > > > be addressed before the merge. >> > > > > >> > > > > *Highlights of HDFS-13891 Branch:* >> > > > > = >> > > > > >> > > > > Adding Security to RBF(1) >> > > > > Adding Missing Client API's(2) >> > > > > Improvements/Bug Fixing >> > > > > Critical - HDFS-13637, HDFS-13834 >> > > > > >> > > > > *Commits:* >> > > > > >> > > > > >> > > > > No of JIRAs Resolved: 72 >> > > > > >> > > > > All this commits are in RBF Module. No changes in hdfs/common. >> > > > > >> > > > > *Tested Cluster:* >> > > > > = >> > > > > >> > > > > Most of these changes verified at Uber,Microsoft,Huawei and some >> > other >> > > > > compaines. >> > > > > >> > > > > *Uber*: Most changes are running in production @Uber including the >> > > > critical >> > > > > security changes, HDFS Clust
[DISCUSS] Merge HDFS-13891(RBF) to trunk
Dear Hadoop Developers I would like to propose RBF Branch (HDFS-13891) merge into trunk. We have been working on this feature from last several months. This feature work received the contributions from different companies. All of the feature development happened smoothly and collaboratively in JIRAs. Kindly do take a look at the branch and raise issues/concerns that need to be addressed before the merge. *Highlights of HDFS-13891 Branch:* = Adding Security to RBF(1) Adding Missing Client API's(2) Improvements/Bug Fixing Critical - HDFS-13637, HDFS-13834 *Commits:* No of JIRAs Resolved: 72 All this commits are in RBF Module. No changes in hdfs/common. *Tested Cluster:* = Most of these changes verified at Uber,Microsoft,Huawei and some other compaines. *Uber*: Most changes are running in production @Uber including the critical security changes, HDFS Clusters are 4000+ nodes with 8 HDFS Routers. Zookeeper as a state store to hold delegation tokens were also stress tested to hold more than 2 Million tokens. --CR Hota *MicroSoft*: Most of these changes are currently running in production at Microsoft.The security has also been tested in a 500 server cluster with 4 subclsuters. --Inigo Goiri *Huawei* : Deployed all this changes in 20 node cluster with 3 routers.Planning deploy 10K production cluster. *Contributors:* === Many thanks to Akira Ajisaka,Mohammad Arshad,Takanobu Asanuma,Shubham Dewan,CR Hota,Fei Hui,Inigo Goiri,Dibyendu Karmakar,Fengna Li,Gang Li,Surendra Singh Lihore,Ranith Sardar,Ayush Saxena,He Xiaoqiao,Sherwood Zheng,Daryn Sharp,VinayaKumar B,Anu Engineer for invloving discussions and contributing to this. *Future Tasks:* will cleanup the jira's under this umbrella and contiue to work. Reference: 1) https://issues.apache.org/jira/browse/HDFS-13532 2) https://issues.apache.org/jira/browse/HDFS-13655 --Brahma Reddy Battula
[RESULT][VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)
Here is my +1 too. Thanks to everyone that participated. The vote to “unprotect the HDFS-13891 branch” is now closed. It PASSED with 4(+1 binding) votes and no 0 or -1 votes: Binding Votes: === Íñigo Goiri Akira Ajisaka Anu Engineer Brahma Reddy Battula Non Binding votes : = Giovanni Matteo Fumarola Takanobu Asanuma Ajay Kumar Dinesh Chitlangia Aaron Fabbri. On Thu, May 16, 2019 at 2:02 AM, Aaron Fabbri wrote: > +1 to unprotect feature branch (in general) for rebasing against trunk. > > On Tue, May 14, 2019 at 7:53 PM Dinesh Chitlangia > wrote: > > > +1(non-binding) for branch. > > > > -Dinesh > > > > > > > > > > On Tue, May 14, 2019 at 10:04 PM Brahma Reddy Battula > > > wrote: > > > > > Yes Arpit,it’s not for trunk.. > > > > > > > > > On Wed, May 15, 2019 at 2:11 AM, Arpit Agarwal > > > wrote: > > > > > > > The request is specific to HDFS-13891, correct? > > > > > > > > We should not allow force push on trunk. > > > > > > > > > > > > > On May 14, 2019, at 8:07 AM, Anu Engineer > > .INVALID> > > > > wrote: > > > > > > > > > > Is it possible to unprotect the branches and not the trunk? > > Generally, > > > a > > > > > force push to trunk indicates a mistake and we have had that in the > > > past. > > > > > This is just a suggestion, even if this request is not met, I am > > still > > > > +1. > > > > > > > > > > Thanks > > > > > Anu > > > > > > > > > > > > > > > > > > > > On Tue, May 14, 2019 at 4:58 AM Takanobu Asanuma < > > > tasan...@yahoo-corp.jp > > > > > > > > > > wrote: > > > > > > > > > >> +1. > > > > >> > > > > >> Thanks! > > > > >> - Takanobu > > > > >> > > > > >> > > > > >> From: Akira Ajisaka > > > > >> Sent: Tuesday, May 14, 2019 4:26:30 PM > > > > >> To: Giovanni Matteo Fumarola > > > > >> Cc: Iñigo Goiri; Brahma Reddy Battula; Hadoop Common; Hdfs-dev > > > > >> Subject: Re: [VOTE] Unprotect HDFS-13891 (HDFS RBF Branch) > > > > >> > > > > >> +1 to unprotect the branch. > > > > >> > > > > >> Thanks, > > > > >> Akira > > > > >> > > > > >> On Tue, May 14, 2019 at 3:11 PM Giovanni Matteo Fumarola > > > > >> wrote: > > > > >>> > > > > >>> +1 to unprotect the branches for rebases. > > > > >>> > > > > >>> On Mon, May 13, 2019 at 11:01 PM Iñigo Goiri > > > > wrote: > > > > >>> > > > > >>>> Syncing the branch to trunk should be a fairly standard task. > > > > >>>> Is there a way to do this without rebasing and forcing the push? > > > > >>>> As far as I know this has been the standard for other branches > > and I > > > > >> don't > > > > >>>> know of any alternative. > > > > >>>> We should clarify the process as having to get PMC consensus to > > > rebase > > > > >> a > > > > >>>> branch seems a little overkill to me. > > > > >>>> > > > > >>>> +1 from my side to un protect the branch to do the rebase. > > > > >>>> > > > > >>>> On Mon, May 13, 2019, 22:46 Brahma Reddy Battula < > > bra...@apache.org > > > > > > > > >>>> wrote: > > > > >>>> > > > > >>>>> Hi Folks, > > > > >>>>> > > > > >>>>> INFRA-18181 made all the Hadoop branches are protected. > > > > >>>>> Badly HDFS-13891 branch needs to rebased as we contribute core > > > > >> patches > > > > >>>>> trunk..So,currently we are stuck with rebase as it’s not > allowed > > to > > > > >> force > > > > >>>>> push.Hence raised INFRA-18361. > > > > >>>>> > > > > >>>>> Can we have a quick vote for INFRA sign-off to proceed as this > is > > > > >>>> blocking > > > > >>>>> all branch commits?? > > > > >>>>> > > > > >>>>> -- > > > > >>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> --Brahma Reddy Battula > > > > >>>>> > > > > >>>> > > > > >> > > > > >> > > - > > > > >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > > > > >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > > > >> > > > > >> > > > > > > > > -- > > > > > > > > > > > > --Brahma Reddy Battula > > > > > > -- --Brahma Reddy Battula
Re: [VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)
Yes Arpit,it’s not for trunk.. On Wed, May 15, 2019 at 2:11 AM, Arpit Agarwal wrote: > The request is specific to HDFS-13891, correct? > > We should not allow force push on trunk. > > > > On May 14, 2019, at 8:07 AM, Anu Engineer > wrote: > > > > Is it possible to unprotect the branches and not the trunk? Generally, a > > force push to trunk indicates a mistake and we have had that in the past. > > This is just a suggestion, even if this request is not met, I am still > +1. > > > > Thanks > > Anu > > > > > > > > On Tue, May 14, 2019 at 4:58 AM Takanobu Asanuma > > > wrote: > > > >> +1. > >> > >> Thanks! > >> - Takanobu > >> > >> ____________ > >> From: Akira Ajisaka > >> Sent: Tuesday, May 14, 2019 4:26:30 PM > >> To: Giovanni Matteo Fumarola > >> Cc: Iñigo Goiri; Brahma Reddy Battula; Hadoop Common; Hdfs-dev > >> Subject: Re: [VOTE] Unprotect HDFS-13891 (HDFS RBF Branch) > >> > >> +1 to unprotect the branch. > >> > >> Thanks, > >> Akira > >> > >> On Tue, May 14, 2019 at 3:11 PM Giovanni Matteo Fumarola > >> wrote: > >>> > >>> +1 to unprotect the branches for rebases. > >>> > >>> On Mon, May 13, 2019 at 11:01 PM Iñigo Goiri > wrote: > >>> > >>>> Syncing the branch to trunk should be a fairly standard task. > >>>> Is there a way to do this without rebasing and forcing the push? > >>>> As far as I know this has been the standard for other branches and I > >> don't > >>>> know of any alternative. > >>>> We should clarify the process as having to get PMC consensus to rebase > >> a > >>>> branch seems a little overkill to me. > >>>> > >>>> +1 from my side to un protect the branch to do the rebase. > >>>> > >>>> On Mon, May 13, 2019, 22:46 Brahma Reddy Battula > >>>> wrote: > >>>> > >>>>> Hi Folks, > >>>>> > >>>>> INFRA-18181 made all the Hadoop branches are protected. > >>>>> Badly HDFS-13891 branch needs to rebased as we contribute core > >> patches > >>>>> trunk..So,currently we are stuck with rebase as it’s not allowed to > >> force > >>>>> push.Hence raised INFRA-18361. > >>>>> > >>>>> Can we have a quick vote for INFRA sign-off to proceed as this is > >>>> blocking > >>>>> all branch commits?? > >>>>> > >>>>> -- > >>>>> > >>>>> > >>>>> > >>>>> --Brahma Reddy Battula > >>>>> > >>>> > >> > >> - > >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >> > >> > > -- --Brahma Reddy Battula
[VOTE] Unprotect HDFS-13891 (HDFS RBF Branch)
Hi Folks, INFRA-18181 made all the Hadoop branches are protected. Badly HDFS-13891 branch needs to rebased as we contribute core patches trunk..So,currently we are stuck with rebase as it’s not allowed to force push.Hence raised INFRA-18361. Can we have a quick vote for INFRA sign-off to proceed as this is blocking all branch commits?? -- --Brahma Reddy Battula
Re: [ANNOUNCE] Eric Badger is now a committer!
Congratulations Eric!!! On Tue, Mar 5, 2019 at 10:50 PM, Eric Payne wrote: > It is my pleasure to announce that Eric Badger has accepted an invitation > to become a Hadoop Core committer. > > Congratulations, Eric! This is well-deserved! > > -Eric Payne > -- --Brahma Reddy Battula
Re: [VOTE] Release Apache Hadoop 3.2.0 - RC1
Sunil thanks for driving this. +1 ,(binding) --Built from Source. --Installed pseudo cluster. --Ran basic shell commands --Ran sample jobs pi,wordcount. On Tue, Jan 8, 2019 at 5:12 PM Sunil G wrote: > Hi folks, > > > Thanks to all of you who helped in this release [1] and for helping to vote > for RC0. I have created second release candidate (RC1) for Apache Hadoop > 3.2.0. > > > Artifacts for this RC are available here: > > http://home.apache.org/~sunilg/hadoop-3.2.0-RC1/ > > > RC tag in git is release-3.2.0-RC1. > > > > The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1178/ > > > This vote will run 7 days (5 weekdays), ending on 14th Jan at 11:59 pm PST. > > > > 3.2.0 contains 1092 [2] fixed JIRA issues since 3.1.0. Below feature > additions > > are the highlights of this release. > > 1. Node Attributes Support in YARN > > 2. Hadoop Submarine project for running Deep Learning workloads on YARN > > 3. Support service upgrade via YARN Service API and CLI > > 4. HDFS Storage Policy Satisfier > > 5. Support Windows Azure Storage - Blob file system in Hadoop > > 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a > > 7. Improvements in Router-based HDFS federation > > > > Thanks to Wangda, Vinod, Marton for helping me in preparing the release. > > I have done few testing with my pseudo cluster. My +1 to start. > > > > Regards, > > Sunil > > > > [1] > > > https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0) > AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved > ORDER BY fixVersion ASC > -- --Brahma Reddy Battula
RE: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby
My late +1. Really it's useful feature.. Great work. -Original Message- From: Konstantin Shvachko [mailto:shv.had...@gmail.com] Sent: Saturday, December 22, 2018 6:48 AM To: Hadoop Common ; hdfs-dev Cc: mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: [Result] [VOTE - 2] Merge HDFS-12943 branch to trunk - Consistent Reads from Standby Obviously +1 from me. With four binding +1s, two non-binding +1s, and no -1s this vote passes. Thank you folks for working on the feature and for voting. Will do the merge in bit. Thanks, --Konst On Fri, Dec 14, 2018 at 6:16 PM Konstantin Shvachko wrote: > Hi Hadoop developers, > > I would like to propose to merge to trunk the feature branch > HDFS-12943 for Consistent Reads from Standby Node. The feature is > intended to scale read RPC workloads. On large clusters reads comprise > 95% of all RPCs to the NameNode. We should be able to accommodate > higher overall RPC workloads (up to 4x by some estimates) by adding multiple > ObserverNodes. > > The main functionality has been implemented see sub-tasks of HDFS-12943. > We followed up with the test plan. Testing was done on two independent > clusters (see HDFS-14058 and HDFS-14059) with security enabled. > We ran standard HDFS commands, MR jobs, admin commands including > manual failover. > We know of one cluster running this feature in production. > > Since the previous vote we addressed Daryn's concern (see HDFS-13873), > added documentation for the new feature, and fixed a few other jiras. > > I attached a unified patch to the umbrella jira for the review. > Please vote on this thread. The vote will run for 7 days until Wed Dec 21. > > Thanks, > --Konstantin >
Re: [DISCUSS] Move to gitbox
+1 On Sat, Dec 8, 2018 at 1:26 PM, Akira Ajisaka wrote: > Hi all, > > Apache Hadoop git repository is in git-wip-us server and it will be > decommissioned. > If there are no objection, I'll file a JIRA ticket with INFRA to > migrate to https://gitbox.apache.org/ and update documentation. > > According to ASF infra team, the timeframe is as follows: > > > - December 9th 2018 -> January 9th 2019: Voluntary (coordinated) > relocation > > - January 9th -> February 6th: Mandated (coordinated) relocation > > - February 7th: All remaining repositories are mass migrated. > > This timeline may change to accommodate various scenarios. > > If we got consensus by January 9th, I can file a ticket with INFRA and > migrate it. > Even if we cannot got consensus, the repository will be migrated by > February 7th. > > Regards, > Akira > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > > -- --Brahma Reddy Battula
RE: [VOTE] Release Apache Hadoop 2.9.2 (RC0)
Akira, Thanks for driving this release. +1 (binding) -- Built from source -- Installed HA cluster --Verified basic operations --Ran Sample Jobs --Browsed the UI -Brahma Reddy Battula -Original Message- From: Akira Ajisaka [mailto:aajis...@apache.org] Sent: Wednesday, November 14, 2018 6:33 AM To: common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org Subject: [VOTE] Release Apache Hadoop 2.9.2 (RC0) Hi folks, I have put together a release candidate (RC0) for Hadoop 2.9.2. It includes 204 bug fixes and improvements since 2.9.1. [1] The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/ Git signed tag is release-2.9.2-RC0 and the checksum is 826afbeae31ca687bc2f8471dc841b66ed2c6704 The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1166/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Please try the release and vote. The vote will run for 5 days. [1] https://s.apache.org/2.9.2-fixed-jiras Thanks, Akira - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: HADOOP-14163 proposal for new hadoop.apache.org
+1 It’s better to new version link in old version. Brahma Reddy Battula On Fri, Aug 31, 2018 at 9:59 PM, Sangjin Lee wrote: > +1. Thanks for the work, Marton! > > On Fri, Aug 31, 2018 at 8:37 AM Vinod Kumar Vavilapalli < > vino...@apache.org> > wrote: > > > Is there no way to host the new site and the old site concurrently? And > > link back & forth? > > > > +Vinod > > > > > > > On Aug 31, 2018, at 1:07 AM, Elek, Marton wrote: > > > > > > Bumping this thread at last time. > > > > > > I have the following proposal: > > > > > > 1. I will request a new git repository hadoop-site.git and import the > > new site to there (which has exactly the same content as the existing > site). > > > > > > 2. I will ask infra to use the new repository as the source of > > hadoop.apache.org > > > > > > 3. I will sync manually all of the changes in the next two months back > > to the svn site from the git (release announcements, new committers) > > > > > > IN CASE OF ANY PROBLEM we can switch back to the svn without any > problem. > > > > > > If no-one objects within three days, I'll assume lazy consensus and > > start with this plan. Please comment if you have objections. > > > > > > Again: it allows immediate fallback at any time as svn repo will be > kept > > as is (+ I will keep it up-to-date in the next 2 months) > > > > > > Thanks, > > > Marton > > > > > > > > > On 06/21/2018 09:00 PM, Elek, Marton wrote: > > >> Thank you very much to bump up this thread. > > >> About [2]: (Just for the clarification) the content of the proposed > > website is exactly the same as the old one. > > >> About [1]. I believe that the "mvn site" is perfect for the > > documentation but for website creation there are more simple and powerful > > tools. > > >> Hugo has more simple compared to jekyll. Just one binary, without > > dependencies, works everywhere (mac, linux, windows) > > >> Hugo has much more powerful compared to "mvn site". Easier to > > create/use more modern layout/theme, and easier to handle the content > (for > > example new release announcements could be generated as part of the > release > > process) > > >> I think it's very low risk to try out a new approach for the site (and > > easy to rollback in case of problems) > > >> Marton > > >> ps: I just updated the patch/preview site with the recent releases: > > >> *** > > >> * http://hadoop.anzix.net * > > >> *** > > >> On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote: > > >>> Got pinged about this offline. > > >>> > > >>> Thanks for keeping at it, Marton! > > >>> > > >>> I think there are two road-blocks here > > >>> (1) Is the mechanism using which the website is built good enough - > > mvn-site / hugo etc? > > >>> (2) Is the new website good enough? > > >>> > > >>> For (1), I just think we need more committer attention and get > > feedback rapidly and get it in. > > >>> > > >>> For (2), how about we do it in a different way in the interest of > > progress? > > >>> - We create a hadoop.apache.org/new-site/ where this new site > goes. > > >>> - We then modify the existing web-site to say that there is a new > > site/experience that folks can click on a link and navigate to > > >>> - As this new website matures and gets feedback & fixes, we finally > > pull the plug at a later point of time when we think we are good to go. > > >>> > > >>> Thoughts? > > >>> > > >>> +Vinod > > >>> > > >>>> On Feb 16, 2018, at 3:10 AM, Elek, Marton wrote: > > >>>> > > >>>> Hi, > > >>>> > > >>>> I would like to bump this thread up. > > >>>> > > >>>> TLDR; There is a proposed version of a new hadoop site which is > > available from here: https://elek.github.io/hadoop-site-proposal/ and > > https://issues.apache.org/jira/browse/HADOOP-14163 > > >>>> > > >>>> Please let me know what you think about it. > > >>>> > > >>>> > > >>>> Longer version: > > >
Re: [VOTE] Release Apache Hadoop 3.1.1 - RC0
+1 ,(binding). Wangda Tan thanks for driving this. --Built with source. --Installed the HA cluster --Ran basic hdfs operations through shell/API --Ran Sample jobs like pi, wordcount --Browsed UI. On Fri, Aug 3, 2018 at 12:14 AM Wangda Tan wrote: > Hi folks, > > I've created RC0 for Apache Hadoop 3.1.1. The artifacts are available here: > > http://people.apache.org/~wangda/hadoop-3.1.1-RC0/ > > The RC tag in git is release-3.1.1-RC0: > https://github.com/apache/hadoop/commits/release-3.1.1-RC0 > > The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1139/ > > You can find my public key at > http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS > > This vote will run 5 days from now. > > 3.1.1 contains 435 [1] fixed JIRA issues since 3.1.0. > > I have done testing with a pseudo cluster and distributed shell job. My +1 > to start. > > Best, > Wangda Tan > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.1) > ORDER BY priority DESC >
Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0)
Thanks yongjun zhang for driving this release. +1 (binding). ---Built from the source ---Installed HA cluster ---Execute the basic shell commands ---Browsed the UI's ---Ran sample jobs like pi,wordcount From: Yongjun Zhang Sent: Friday, June 8, 2018 1:04 PM To: Allen Wittenauer Cc: Hadoop Common; Hdfs-dev; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: Re: [VOTE] Release Apache Hadoop 3.0.3 (RC0) BTW, thanks Allen and Steve for discussing and suggestion about the site build problem I hit earlier, I did the following step mvn install -DskipTests before doing the steps Nanda listed helped to solve the problems. --Yongjun On Thu, Jun 7, 2018 at 6:15 PM, Yongjun Zhang wrote: > Thank you all very much for the testing, feedback and discussion! > > I was able to build outside docker, by following the steps Nanda > described, I saw the same problem; then I tried 3.0.2 released a while > back, it has the same issue. > > As Allen pointed out, it seems the steps to build site are not correct. I > have not figured out the correct steps yet. > > At this point, I think this issue should not block the 3.0.3 issue. While > at the same time we need to figure out the right steps to build the site. > Would you please let me know if you think differently? > > We only have the site build issue reported so far. And we don't have > enough PMC votes yet. So need some more PMCs to help. > > Thanks again, and best regards, > > --Yongjun > > > On Thu, Jun 7, 2018 at 4:15 PM, Allen Wittenauer > wrote: > >> > On Jun 7, 2018, at 11:47 AM, Steve Loughran >> wrote: >> > >> > Actually, Yongjun has been really good at helping me get set up for a >> 2.7.7 release, including "things you need to do to get GPG working in the >> docker image” >> >> *shrugs* I use a different release script after some changes >> broke the in-tree version for building on OS X and I couldn’t get the fixes >> committed upstream. So not sure what the problems are that you are hitting. >> >> > On Jun 7, 2018, at 1:08 PM, Nandakumar Vadivelu < >> nvadiv...@hortonworks.com> wrote: >> > >> > It will be helpful if we can get the correct steps, and also update the >> wiki. >> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+ >> Release+Validation >> >> Yup. Looking forward to seeing it. >> - >> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org >> >> >
RE: [VOTE] Release Apache Hadoop 2.8.4 (RC0)
Thanks Junping for driving this release. +1 (binding) -- Build successfully from the source code -- Start HA cluster -- Verified basic shell operations -- Ran pi,wordcount -- Browsed the NN and RM UI -Brahma Reddy Battula -Original Message- From: 俊平堵 [mailto:junping...@apache.org] Sent: 09 May 2018 01:41 To: Hadoop Common ; Hdfs-dev ; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: [VOTE] Release Apache Hadoop 2.8.4 (RC0) Hi all, I've created the first release candidate (RC0) for Apache Hadoop 2.8.4. This is our next maint release to follow up 2.8.3. It includes 77 important fixes and improvements. The RC artifacts are available at: http://home.apache.org/~junping_du/hadoop-2.8.4-RC0 The RC tag in git is: release-2.8.4-RC0 The maven artifacts are available via repository.apache.org< http://repository.apache.org> at: https://repository.apache.org/content/repositories/orgapachehadoop-1118 Please try the release and vote; the vote will run for the usual 5 working days, ending on 5/14/2018 PST time. Thanks, Junping - To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-dev-h...@hadoop.apache.org
RE: Apache Hadoop 3.1.1 release plan
Thanks Wangda Tan for driving the 3.1.1 release.Yes,This can be better addition to 3.1 line release for improving quality. Looks only following two are pending which are in review state. Hope you are monitoring these two. https://issues.apache.org/jira/browse/YARN-8265 https://issues.apache.org/jira/browse/YARN-8236 Note : https://issues.apache.org/jira/browse/YARN-8247==> committed branch-3.1 -Original Message- From: Wangda Tan [mailto:wheele...@gmail.com] Sent: 19 April 2018 17:49 To: Hadoop Common ; mapreduce-...@hadoop.apache.org; Hdfs-dev ; yarn-...@hadoop.apache.org Subject: Apache Hadoop 3.1.1 release plan Hi, All We have released Apache Hadoop 3.1.0 on Apr 06. To further improve the quality of the release, we plan to release 3.1.1 at May 06. The focus of 3.1.1 will be fixing blockers / critical bugs and other enhancements. So far there are 100 JIRAs [1] have fix version marked to 3.1.1. We plan to cut branch-3.1.1 on May 01 and vote for RC on the same day. Please feel free to share your insights. Thanks, Wangda Tan [1] project in (YARN, "Hadoop HDFS", "Hadoop Common", "Hadoop Map/Reduce") AND fixVersion = 3.1.1
RE: [VOTE] Release Apache Hadoop 2.7.6 (RC0)
Konstantin thanks for driving this. +1 (binding) --Built from the source --Installed HA cluster -Verified the basic shell commands -Ran sample jobs like pi,wordcount -Browsed the UI's -Original Message- From: Konstantin Shvachko [mailto:shv.had...@gmail.com] Sent: 10 April 2018 07:14 To: Hadoop Common ; hdfs-dev ; mapreduce-...@hadoop.apache.org; yarn-...@hadoop.apache.org Subject: [VOTE] Release Apache Hadoop 2.7.6 (RC0) Hi everybody, This is the next dot release of Apache Hadoop 2.7 line. The previous one 2.7.5 was released on December 14, 2017. Release 2.7.6 includes critical bug fixes and optimizations. See more details in Release Note: http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/ Please give it a try and vote on this thread. The vote will run for 5 days ending 04/16/2018. My up to date public key is available from: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Thanks, --Konstantin
Re: [VOTE] Release Apache Hadoop 3.1.0 (RC1)
Wangda thanks for driving this. +1(binding) --Built from source --Installed HA cluster --Verified Basic Shell commands --Ran Sample Jobs --Browsed the UI's. On Fri, Mar 30, 2018 at 9:45 AM, Wangda Tan wrote: > Hi folks, > > Thanks to the many who helped with this release since Dec 2017 [1]. We've > created RC1 for Apache Hadoop 3.1.0. The artifacts are available here: > > http://people.apache.org/~wangda/hadoop-3.1.0-RC1 > > The RC tag in git is release-3.1.0-RC1. Last git commit SHA is > 16b70619a24cdcf5d3b0fcf4b58ca77238ccbe6d > > The maven artifacts are available via repository.apache.org at > https://repository.apache.org/content/repositories/orgapachehadoop-1090/ > This vote will run 5 days, ending on Apr 3 at 11:59 pm Pacific. > > 3.1.0 contains 766 [2] fixed JIRA issues since 3.0.0. Notable additions > include the first class GPU/FPGA support on YARN, Native services, Support > rich placement constraints in YARN, S3-related enhancements, allow HDFS > block replicas to be provided by an external storage system, etc. > > For 3.1.0 RC0 vote discussion, please see [3]. > > We’d like to use this as a starting release for 3.1.x [1], depending on how > it goes, get it stabilized and potentially use a 3.1.1 in several weeks as > the stable release. > > We have done testing with a pseudo cluster: > - Ran distributed job. > - GPU scheduling/isolation. > - Placement constraints (intra-application anti-affinity) by using > distributed shell. > > My +1 to start. > > Best, > Wangda/Vinod > > [1] > https://lists.apache.org/thread.html/b3fb3b6da8b6357a68513a6dfd104b > c9e19e559aedc5ebedb4ca08c8@%3Cyarn-dev.hadoop.apache.org%3E > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.0) > AND fixVersion not in (3.0.0, 3.0.0-beta1) AND status = Resolved ORDER BY > fixVersion ASC > [3] > https://lists.apache.org/thread.html/b3a7dc075b7329fd660f65b48237d7 > 2d4061f26f83547e41d0983ea6@%3Cyarn-dev.hadoop.apache.org%3E >
