Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
go ahead and cut it...I'd forgotten about it. thanks, steve On Fri, 31 Mar 2023 at 06:44, Ayush Saxena wrote: > We have a daily build running for 3.3.5: > https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/ > > We have already released it, so I feel we can disable it. Will do it > tomorrow, if nobody objects. In case the one who configured it wants > to do it early, feel free to do so. > > We already have one for branch-3.3 which runs weekly which most > probably most of us don't follow :) > > -Ayush > > On Wed, 22 Mar 2023 at 00:20, Steve Loughran > wrote: > > > > ok, here's my summary, even though most of the binding voters forgot to > > declare they were on the PMC. > > > > +1 binding > > > > Steve Loughran > > Chris Nauroth > > Masatake Iwasaki > > Ayush Saxena > > Xiaoqiao He > > > > +1 non-binding > > > > Viraj Jasani > > > > > > 0 or -1 votes: none. > > > > > > Accordingly: the release is good! > > > > I will send the formal announcement out tomorrow > > > > A big thank you to everyone who qualified the RC, I know its a lot of > work. > > We can now get this out and *someone else* can plan the followup. > > > > > > steve > > > > On Mon, 20 Mar 2023 at 16:01, Chris Nauroth wrote: > > > > > +1 > > > > > > Thank you for the release candidate, Steve! > > > > > > * Verified all checksums. > > > * Verified all signatures. > > > * Built from source, including native code on Linux. > > > * mvn clean package -Pnative -Psrc -Drequire.openssl > -Drequire.snappy > > > -Drequire.zstd -DskipTests > > > * Tests passed. > > > * mvn --fail-never clean test -Pnative -Dparallel-tests > > > -Drequire.snappy -Drequire.zstd -Drequire.openssl > > > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > > * Checked dependency tree to make sure we have all of the expected > library > > > updates that are mentioned in the release notes. > > > * mvn -o dependency:tree > > > * Confirmed that hadoop-openstack is now just a stub placeholder > artifact > > > with no code. > > > * For ARM verification: > > > * Ran "file " on all native binaries in the ARM tarball to > confirm > > > they actually came out with ARM as the architecture. > > > * Output of hadoop checknative -a on ARM looks good. > > > * Ran a MapReduce job with the native bzip2 codec for compression, > and > > > it worked fine. > > > * Ran a MapReduce job with YARN configured to use > > > LinuxContainerExecutor and verified launching the containers through > > > container-executor worked. > > > > > > Chris Nauroth > > > > > > > > > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena > wrote: > > > > > > > +1(Binding) > > > > > > > > * Built from source (x86 & ARM) > > > > * Successful Native Build (x86 & ARM) > > > > * Verified Checksums (x86 & ARM) > > > > * Verified Signature (x86 & ARM) > > > > * Checked the output of hadoop version (x86 & ARM) > > > > * Verified the output of hadoop checknative (x86 & ARM) > > > > * Ran some basic HDFS shell commands. > > > > * Ran some basic Yarn shell commands. > > > > * Played a bit with HDFS Erasure Coding. > > > > * Ran TeraGen & TeraSort > > > > * Browed through NN, DN, RM & NM UI > > > > * Skimmed over the contents of website. > > > > * Skimmed over the contents of maven repo. > > > > * Selectively ran some HDFS & CloudStore tests > > > > > > > > Thanx Steve for driving the release. Good Luck!!! > > > > > > > > -Ayush > > > > > > > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He > > > wrote: > > > > > > > > > > +1 > > > > > > > > > > * Verified signature and checksum of the source tarball. > > > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean > package > > > > > -DskipTests -Pnative -Pdist -Dtar`. > > > > > * Setup pseudo cluster with HDFS and YARN. > > > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check > the > > > > > result. > > > > > * Run example mr applications and check the result - Pi & > wordcount. > > > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager > > > etc. > > > > > > > > > > Thanks Steve for your work. > > > > > > > > > > Best Regards, > > > > > - He Xiaoqiao > > > > > > > > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki < > > > > iwasak...@oss.nttdata.com> > > > > >> wrote: > > > > >> > > > > >> +1 > > > > >> > > > > >> + verified the signature and checksum of the source tarball. > > > > >> > > > > >> + built from the source tarball on Rocky Linux 8 (x86_64) and > OpenJDK > > > 8 > > > > >> with native profile enabled. > > > > >> + launched pseudo distributed cluster including kms and httpfs > with > > > > >> Kerberos and SSL enabled. > > > > >> + created encryption zone, put and read files via httpfs. > > > > >> + ran example MR wordcount over encryption zone. > > > > >> + checked the binary of container-executor. > > > > >> > > > > >> + built rpm packages by Bigtop (with trivial modifications) on > Rocky > > > > Linux > > > > >> 8 (aarch64). > > > > >> + ran smoke-tests of
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
We have a daily build running for 3.3.5: https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/ We have already released it, so I feel we can disable it. Will do it tomorrow, if nobody objects. In case the one who configured it wants to do it early, feel free to do so. We already have one for branch-3.3 which runs weekly which most probably most of us don't follow :) -Ayush On Wed, 22 Mar 2023 at 00:20, Steve Loughran wrote: > > ok, here's my summary, even though most of the binding voters forgot to > declare they were on the PMC. > > +1 binding > > Steve Loughran > Chris Nauroth > Masatake Iwasaki > Ayush Saxena > Xiaoqiao He > > +1 non-binding > > Viraj Jasani > > > 0 or -1 votes: none. > > > Accordingly: the release is good! > > I will send the formal announcement out tomorrow > > A big thank you to everyone who qualified the RC, I know its a lot of work. > We can now get this out and *someone else* can plan the followup. > > > steve > > On Mon, 20 Mar 2023 at 16:01, Chris Nauroth wrote: > > > +1 > > > > Thank you for the release candidate, Steve! > > > > * Verified all checksums. > > * Verified all signatures. > > * Built from source, including native code on Linux. > > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy > > -Drequire.zstd -DskipTests > > * Tests passed. > > * mvn --fail-never clean test -Pnative -Dparallel-tests > > -Drequire.snappy -Drequire.zstd -Drequire.openssl > > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > > * Checked dependency tree to make sure we have all of the expected library > > updates that are mentioned in the release notes. > > * mvn -o dependency:tree > > * Confirmed that hadoop-openstack is now just a stub placeholder artifact > > with no code. > > * For ARM verification: > > * Ran "file " on all native binaries in the ARM tarball to confirm > > they actually came out with ARM as the architecture. > > * Output of hadoop checknative -a on ARM looks good. > > * Ran a MapReduce job with the native bzip2 codec for compression, and > > it worked fine. > > * Ran a MapReduce job with YARN configured to use > > LinuxContainerExecutor and verified launching the containers through > > container-executor worked. > > > > Chris Nauroth > > > > > > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena wrote: > > > > > +1(Binding) > > > > > > * Built from source (x86 & ARM) > > > * Successful Native Build (x86 & ARM) > > > * Verified Checksums (x86 & ARM) > > > * Verified Signature (x86 & ARM) > > > * Checked the output of hadoop version (x86 & ARM) > > > * Verified the output of hadoop checknative (x86 & ARM) > > > * Ran some basic HDFS shell commands. > > > * Ran some basic Yarn shell commands. > > > * Played a bit with HDFS Erasure Coding. > > > * Ran TeraGen & TeraSort > > > * Browed through NN, DN, RM & NM UI > > > * Skimmed over the contents of website. > > > * Skimmed over the contents of maven repo. > > > * Selectively ran some HDFS & CloudStore tests > > > > > > Thanx Steve for driving the release. Good Luck!!! > > > > > > -Ayush > > > > > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He > > wrote: > > > > > > > > +1 > > > > > > > > * Verified signature and checksum of the source tarball. > > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > > > -DskipTests -Pnative -Pdist -Dtar`. > > > > * Setup pseudo cluster with HDFS and YARN. > > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the > > > > result. > > > > * Run example mr applications and check the result - Pi & wordcount. > > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager > > etc. > > > > > > > > Thanks Steve for your work. > > > > > > > > Best Regards, > > > > - He Xiaoqiao > > > > > > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki < > > > iwasak...@oss.nttdata.com> > > > >> wrote: > > > >> > > > >> +1 > > > >> > > > >> + verified the signature and checksum of the source tarball. > > > >> > > > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK > > 8 > > > >> with native profile enabled. > > > >> + launched pseudo distributed cluster including kms and httpfs with > > > >> Kerberos and SSL enabled. > > > >> + created encryption zone, put and read files via httpfs. > > > >> + ran example MR wordcount over encryption zone. > > > >> + checked the binary of container-executor. > > > >> > > > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky > > > Linux > > > >> 8 (aarch64). > > > >> + ran smoke-tests of hdfs, yarn and mapreduce. > > > >> + built site documentation and skimmed the contents. > > > >> + Javadocs are contained. > > > >> > > > >> Thanks, > > > >> Masatake Iwasaki > > > >> > > > >>> On 2023/03/16 4:47, Steve Loughran wrote: > > > >>> Apache Hadoop 3.3.5 > > > >>> > > > >>> Mukund and I have put together a release candidate (RC3) for Hadoop > > > >> 3.3.5. > > > >>> > > > >>> What we would like is for
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
ok, here's my summary, even though most of the binding voters forgot to declare they were on the PMC. +1 binding Steve Loughran Chris Nauroth Masatake Iwasaki Ayush Saxena Xiaoqiao He +1 non-binding Viraj Jasani 0 or -1 votes: none. Accordingly: the release is good! I will send the formal announcement out tomorrow A big thank you to everyone who qualified the RC, I know its a lot of work. We can now get this out and *someone else* can plan the followup. steve On Mon, 20 Mar 2023 at 16:01, Chris Nauroth wrote: > +1 > > Thank you for the release candidate, Steve! > > * Verified all checksums. > * Verified all signatures. > * Built from source, including native code on Linux. > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy > -Drequire.zstd -DskipTests > * Tests passed. > * mvn --fail-never clean test -Pnative -Dparallel-tests > -Drequire.snappy -Drequire.zstd -Drequire.openssl > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 > * Checked dependency tree to make sure we have all of the expected library > updates that are mentioned in the release notes. > * mvn -o dependency:tree > * Confirmed that hadoop-openstack is now just a stub placeholder artifact > with no code. > * For ARM verification: > * Ran "file " on all native binaries in the ARM tarball to confirm > they actually came out with ARM as the architecture. > * Output of hadoop checknative -a on ARM looks good. > * Ran a MapReduce job with the native bzip2 codec for compression, and > it worked fine. > * Ran a MapReduce job with YARN configured to use > LinuxContainerExecutor and verified launching the containers through > container-executor worked. > > Chris Nauroth > > > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena wrote: > > > +1(Binding) > > > > * Built from source (x86 & ARM) > > * Successful Native Build (x86 & ARM) > > * Verified Checksums (x86 & ARM) > > * Verified Signature (x86 & ARM) > > * Checked the output of hadoop version (x86 & ARM) > > * Verified the output of hadoop checknative (x86 & ARM) > > * Ran some basic HDFS shell commands. > > * Ran some basic Yarn shell commands. > > * Played a bit with HDFS Erasure Coding. > > * Ran TeraGen & TeraSort > > * Browed through NN, DN, RM & NM UI > > * Skimmed over the contents of website. > > * Skimmed over the contents of maven repo. > > * Selectively ran some HDFS & CloudStore tests > > > > Thanx Steve for driving the release. Good Luck!!! > > > > -Ayush > > > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He > wrote: > > > > > > +1 > > > > > > * Verified signature and checksum of the source tarball. > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > > -DskipTests -Pnative -Pdist -Dtar`. > > > * Setup pseudo cluster with HDFS and YARN. > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the > > > result. > > > * Run example mr applications and check the result - Pi & wordcount. > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager > etc. > > > > > > Thanks Steve for your work. > > > > > > Best Regards, > > > - He Xiaoqiao > > > > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki < > > iwasak...@oss.nttdata.com> > > >> wrote: > > >> > > >> +1 > > >> > > >> + verified the signature and checksum of the source tarball. > > >> > > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK > 8 > > >> with native profile enabled. > > >> + launched pseudo distributed cluster including kms and httpfs with > > >> Kerberos and SSL enabled. > > >> + created encryption zone, put and read files via httpfs. > > >> + ran example MR wordcount over encryption zone. > > >> + checked the binary of container-executor. > > >> > > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky > > Linux > > >> 8 (aarch64). > > >> + ran smoke-tests of hdfs, yarn and mapreduce. > > >> + built site documentation and skimmed the contents. > > >> + Javadocs are contained. > > >> > > >> Thanks, > > >> Masatake Iwasaki > > >> > > >>> On 2023/03/16 4:47, Steve Loughran wrote: > > >>> Apache Hadoop 3.3.5 > > >>> > > >>> Mukund and I have put together a release candidate (RC3) for Hadoop > > >> 3.3.5. > > >>> > > >>> What we would like is for anyone who can to verify the tarballs, > > >> especially > > >>> anyone who can try the arm64 binaries as we want to include them too. > > >>> > > >>> The RC is available at: > > >>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > >>> > > >>> The git tag is release-3.3.5-RC3, commit 706d88266ab > > >>> > > >>> The maven artifacts are staged at > > >>> > > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > >>> > > >>> You can find my public key at: > > >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > >>> > > >>> Change log > > >>> > > >> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > >>> > > >>> Release
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
+1 Thank you for the release candidate, Steve! * Verified all checksums. * Verified all signatures. * Built from source, including native code on Linux. * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy -Drequire.zstd -DskipTests * Tests passed. * mvn --fail-never clean test -Pnative -Dparallel-tests -Drequire.snappy -Drequire.zstd -Drequire.openssl -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8 * Checked dependency tree to make sure we have all of the expected library updates that are mentioned in the release notes. * mvn -o dependency:tree * Confirmed that hadoop-openstack is now just a stub placeholder artifact with no code. * For ARM verification: * Ran "file " on all native binaries in the ARM tarball to confirm they actually came out with ARM as the architecture. * Output of hadoop checknative -a on ARM looks good. * Ran a MapReduce job with the native bzip2 codec for compression, and it worked fine. * Ran a MapReduce job with YARN configured to use LinuxContainerExecutor and verified launching the containers through container-executor worked. Chris Nauroth On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena wrote: > +1(Binding) > > * Built from source (x86 & ARM) > * Successful Native Build (x86 & ARM) > * Verified Checksums (x86 & ARM) > * Verified Signature (x86 & ARM) > * Checked the output of hadoop version (x86 & ARM) > * Verified the output of hadoop checknative (x86 & ARM) > * Ran some basic HDFS shell commands. > * Ran some basic Yarn shell commands. > * Played a bit with HDFS Erasure Coding. > * Ran TeraGen & TeraSort > * Browed through NN, DN, RM & NM UI > * Skimmed over the contents of website. > * Skimmed over the contents of maven repo. > * Selectively ran some HDFS & CloudStore tests > > Thanx Steve for driving the release. Good Luck!!! > > -Ayush > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He wrote: > > > > +1 > > > > * Verified signature and checksum of the source tarball. > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > > -DskipTests -Pnative -Pdist -Dtar`. > > * Setup pseudo cluster with HDFS and YARN. > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the > > result. > > * Run example mr applications and check the result - Pi & wordcount. > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc. > > > > Thanks Steve for your work. > > > > Best Regards, > > - He Xiaoqiao > > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki < > iwasak...@oss.nttdata.com> > >> wrote: > >> > >> +1 > >> > >> + verified the signature and checksum of the source tarball. > >> > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 > >> with native profile enabled. > >> + launched pseudo distributed cluster including kms and httpfs with > >> Kerberos and SSL enabled. > >> + created encryption zone, put and read files via httpfs. > >> + ran example MR wordcount over encryption zone. > >> + checked the binary of container-executor. > >> > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky > Linux > >> 8 (aarch64). > >> + ran smoke-tests of hdfs, yarn and mapreduce. > >> + built site documentation and skimmed the contents. > >> + Javadocs are contained. > >> > >> Thanks, > >> Masatake Iwasaki > >> > >>> On 2023/03/16 4:47, Steve Loughran wrote: > >>> Apache Hadoop 3.3.5 > >>> > >>> Mukund and I have put together a release candidate (RC3) for Hadoop > >> 3.3.5. > >>> > >>> What we would like is for anyone who can to verify the tarballs, > >> especially > >>> anyone who can try the arm64 binaries as we want to include them too. > >>> > >>> The RC is available at: > >>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > >>> > >>> The git tag is release-3.3.5-RC3, commit 706d88266ab > >>> > >>> The maven artifacts are staged at > >>> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > >>> > >>> You can find my public key at: > >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > >>> > >>> Change log > >>> > >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > >>> > >>> Release notes > >>> > >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > >>> > >>> This is off branch-3.3 and is the first big release since 3.3.2. > >>> > >>> Key changes include > >>> > >>> * Big update of dependencies to try and keep those reports of > >>> transitive CVEs under control -both genuine and false positives. > >>> * HDFS RBF enhancements > >>> * Critical fix to ABFS input stream prefetching for correct reading. > >>> * Vectored IO API for all FSDataInputStream implementations, with > >>> high-performance versions for file:// and s3a:// filesystems. > >>> file:// through java native io > >>> s3a:// parallel GET requests. > >>> * This release includes Arm64 binaries. Please can anyone with > >>> compatible systems validate
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
+1(Binding) * Built from source (x86 & ARM) * Successful Native Build (x86 & ARM) * Verified Checksums (x86 & ARM) * Verified Signature (x86 & ARM) * Checked the output of hadoop version (x86 & ARM) * Verified the output of hadoop checknative (x86 & ARM) * Ran some basic HDFS shell commands. * Ran some basic Yarn shell commands. * Played a bit with HDFS Erasure Coding. * Ran TeraGen & TeraSort * Browed through NN, DN, RM & NM UI * Skimmed over the contents of website. * Skimmed over the contents of maven repo. * Selectively ran some HDFS & CloudStore tests Thanx Steve for driving the release. Good Luck!!! -Ayush > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He wrote: > > +1 > > * Verified signature and checksum of the source tarball. > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package > -DskipTests -Pnative -Pdist -Dtar`. > * Setup pseudo cluster with HDFS and YARN. > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the > result. > * Run example mr applications and check the result - Pi & wordcount. > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc. > > Thanks Steve for your work. > > Best Regards, > - He Xiaoqiao > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki >> wrote: >> >> +1 >> >> + verified the signature and checksum of the source tarball. >> >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 >> with native profile enabled. >> + launched pseudo distributed cluster including kms and httpfs with >> Kerberos and SSL enabled. >> + created encryption zone, put and read files via httpfs. >> + ran example MR wordcount over encryption zone. >> + checked the binary of container-executor. >> >> + built rpm packages by Bigtop (with trivial modifications) on Rocky Linux >> 8 (aarch64). >> + ran smoke-tests of hdfs, yarn and mapreduce. >> + built site documentation and skimmed the contents. >> + Javadocs are contained. >> >> Thanks, >> Masatake Iwasaki >> >>> On 2023/03/16 4:47, Steve Loughran wrote: >>> Apache Hadoop 3.3.5 >>> >>> Mukund and I have put together a release candidate (RC3) for Hadoop >> 3.3.5. >>> >>> What we would like is for anyone who can to verify the tarballs, >> especially >>> anyone who can try the arm64 binaries as we want to include them too. >>> >>> The RC is available at: >>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ >>> >>> The git tag is release-3.3.5-RC3, commit 706d88266ab >>> >>> The maven artifacts are staged at >>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/ >>> >>> You can find my public key at: >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS >>> >>> Change log >>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md >>> >>> Release notes >>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md >>> >>> This is off branch-3.3 and is the first big release since 3.3.2. >>> >>> Key changes include >>> >>> * Big update of dependencies to try and keep those reports of >>> transitive CVEs under control -both genuine and false positives. >>> * HDFS RBF enhancements >>> * Critical fix to ABFS input stream prefetching for correct reading. >>> * Vectored IO API for all FSDataInputStream implementations, with >>> high-performance versions for file:// and s3a:// filesystems. >>> file:// through java native io >>> s3a:// parallel GET requests. >>> * This release includes Arm64 binaries. Please can anyone with >>> compatible systems validate these. >>> * and compared to the previous RC, all the major changes are >>> HDFS issues. >>> >>> Note, because the arm64 binaries are built separately on a different >>> platform and JVM, their jar files may not match those of the x86 >>> release -and therefore the maven artifacts. I don't think this is >>> an issue (the ASF actually releases source tarballs, the binaries are >>> there for help only, though with the maven repo that's a bit blurred). >>> >>> The only way to be consistent would actually untar the x86.tar.gz, >>> overwrite its binaries with the arm stuff, retar, sign and push out >>> for the vote. Even automating that would be risky. >>> >>> Please try the release and vote. The vote will run for 5 days. >>> >>> -Steve >>> >> >> - >> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org >> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org >> >> - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
+1 * Verified signature and checksum of the source tarball. * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package -DskipTests -Pnative -Pdist -Dtar`. * Setup pseudo cluster with HDFS and YARN. * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the result. * Run example mr applications and check the result - Pi & wordcount. * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc. Thanks Steve for your work. Best Regards, - He Xiaoqiao On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki wrote: > +1 > > + verified the signature and checksum of the source tarball. > > + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 > with native profile enabled. >+ launched pseudo distributed cluster including kms and httpfs with > Kerberos and SSL enabled. >+ created encryption zone, put and read files via httpfs. >+ ran example MR wordcount over encryption zone. >+ checked the binary of container-executor. > > + built rpm packages by Bigtop (with trivial modifications) on Rocky Linux > 8 (aarch64). >+ ran smoke-tests of hdfs, yarn and mapreduce. > + built site documentation and skimmed the contents. >+ Javadocs are contained. > > Thanks, > Masatake Iwasaki > > On 2023/03/16 4:47, Steve Loughran wrote: > > Apache Hadoop 3.3.5 > > > > Mukund and I have put together a release candidate (RC3) for Hadoop > 3.3.5. > > > > What we would like is for anyone who can to verify the tarballs, > especially > > anyone who can try the arm64 binaries as we want to include them too. > > > > The RC is available at: > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > > > The git tag is release-3.3.5-RC3, commit 706d88266ab > > > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > Change log > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > > > Release notes > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > > > > This is off branch-3.3 and is the first big release since 3.3.2. > > > > Key changes include > > > > * Big update of dependencies to try and keep those reports of > >transitive CVEs under control -both genuine and false positives. > > * HDFS RBF enhancements > > * Critical fix to ABFS input stream prefetching for correct reading. > > * Vectored IO API for all FSDataInputStream implementations, with > >high-performance versions for file:// and s3a:// filesystems. > >file:// through java native io > >s3a:// parallel GET requests. > > * This release includes Arm64 binaries. Please can anyone with > >compatible systems validate these. > > * and compared to the previous RC, all the major changes are > >HDFS issues. > > > > Note, because the arm64 binaries are built separately on a different > > platform and JVM, their jar files may not match those of the x86 > > release -and therefore the maven artifacts. I don't think this is > > an issue (the ASF actually releases source tarballs, the binaries are > > there for help only, though with the maven repo that's a bit blurred). > > > > The only way to be consistent would actually untar the x86.tar.gz, > > overwrite its binaries with the arm stuff, retar, sign and push out > > for the vote. Even automating that would be risky. > > > > Please try the release and vote. The vote will run for 5 days. > > > > -Steve > > > > - > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org > >
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
+1 + verified the signature and checksum of the source tarball. + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 with native profile enabled. + launched pseudo distributed cluster including kms and httpfs with Kerberos and SSL enabled. + created encryption zone, put and read files via httpfs. + ran example MR wordcount over encryption zone. + checked the binary of container-executor. + built rpm packages by Bigtop (with trivial modifications) on Rocky Linux 8 (aarch64). + ran smoke-tests of hdfs, yarn and mapreduce. + built site documentation and skimmed the contents. + Javadocs are contained. Thanks, Masatake Iwasaki On 2023/03/16 4:47, Steve Loughran wrote: Apache Hadoop 3.3.5 Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5. What we would like is for anyone who can to verify the tarballs, especially anyone who can try the arm64 binaries as we want to include them too. The RC is available at: https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ The git tag is release-3.3.5-RC3, commit 706d88266ab The maven artifacts are staged at https://repository.apache.org/content/repositories/orgapachehadoop-1369/ You can find my public key at: https://dist.apache.org/repos/dist/release/hadoop/common/KEYS Change log https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md Release notes https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md This is off branch-3.3 and is the first big release since 3.3.2. Key changes include * Big update of dependencies to try and keep those reports of transitive CVEs under control -both genuine and false positives. * HDFS RBF enhancements * Critical fix to ABFS input stream prefetching for correct reading. * Vectored IO API for all FSDataInputStream implementations, with high-performance versions for file:// and s3a:// filesystems. file:// through java native io s3a:// parallel GET requests. * This release includes Arm64 binaries. Please can anyone with compatible systems validate these. * and compared to the previous RC, all the major changes are HDFS issues. Note, because the arm64 binaries are built separately on a different platform and JVM, their jar files may not match those of the x86 release -and therefore the maven artifacts. I don't think this is an issue (the ASF actually releases source tarballs, the binaries are there for help only, though with the maven repo that's a bit blurred). The only way to be consistent would actually untar the x86.tar.gz, overwrite its binaries with the arm stuff, retar, sign and push out for the vote. Even automating that would be risky. Please try the release and vote. The vote will run for 5 days. -Steve - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
Count me in as well. I am almost done. So, you have 3 potential votes, can be happy now :) Thanx Steve for the efforts!!! -Ayush > On 19-Mar-2023, at 2:46 AM, Chris Nauroth wrote: > > Yes, I'm in progress on verification, so you can expect to get a vote from > me. Thank you, Steve! > > Chris Nauroth > > >> On Sat, Mar 18, 2023 at 9:19 AM Ashutosh Gupta >> wrote: >> Hi Steve >> I will also do it by today/tomorrow. >> Thanks, >> Ashutosh >> On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, > wrote: >>> Thank you for this! >>> Can anyone else with time do a review too? i really want to get this one >>> done, now the HDFS issues are all resolved. >>> I do not want this release to fall by the wayside through lack of votes >>> alone. In fact, I would be very unhappy On Sat, 18 Mar 2023 at 06:47, Viraj Jasani wrote: +1 (non-binding) * Signature/Checksum: ok * Rat check (1.8.0_341): ok - mvn clean apache-rat:check * Built from source (1.8.0_341): ok - mvn clean install -DskipTests * Built tar from source (1.8.0_341): ok - mvn clean package -Pdist -DskipTests -Dtar >> -Dmaven.javadoc.skip=true Containerized deployments: * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper >> 3.7 * Deployed and started JHS, RM, NM * Hbase, hdfs CRUD looks good * Sample RowCount MapReduce job looks good * S3A tests with scale profile looks good On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran wrote: > Apache Hadoop 3.3.5 > Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5. > What we would like is for anyone who can to verify the tarballs, especially > anyone who can try the arm64 binaries as we want to include them too. > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > The git tag is release-3.3.5-RC3, commit 706d88266ab > The maven artifacts are staged at >>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > You can find my public key at: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > Change log >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > Release notes >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > This is off branch-3.3 and is the first big release since 3.3.2. > Key changes include > * Big update of dependencies to try and keep those reports of > transitive CVEs under control -both genuine and false positives. > * HDFS RBF enhancements > * Critical fix to ABFS input stream prefetching for correct reading. > * Vectored IO API for all FSDataInputStream implementations, with > high-performance versions for file:// and s3a:// filesystems. > file:// through java native io > s3a:// parallel GET requests. > * This release includes Arm64 binaries. Please can anyone with > compatible systems validate these. > * and compared to the previous RC, all the major changes are > HDFS issues. > Note, because the arm64 binaries are built separately on a different > platform and JVM, their jar files may not match those of the x86 > release -and therefore the maven artifacts. I don't think this is > an issue (the ASF actually releases source tarballs, the binaries are > there for help only, though with the maven repo that's a bit >> blurred). > The only way to be consistent would actually untar the x86.tar.gz, > overwrite its binaries with the arm stuff, retar, sign and push out > for the vote. Even automating that would be risky. > Please try the release and vote. The vote will run for 5 days. > -Steve - To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
Yes, I'm in progress on verification, so you can expect to get a vote from me. Thank you, Steve! Chris Nauroth On Sat, Mar 18, 2023 at 9:19 AM Ashutosh Gupta wrote: > Hi Steve > > I will also do it by today/tomorrow. > > Thanks, > Ashutosh > > On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, > > wrote: > > > Thank you for this! > > > > Can anyone else with time do a review too? i really want to get this one > > done, now the HDFS issues are all resolved. > > > > I do not want this release to fall by the wayside through lack of votes > > alone. In fact, I would be very unhappy > > > > > > > > On Sat, 18 Mar 2023 at 06:47, Viraj Jasani wrote: > > > > > +1 (non-binding) > > > > > > * Signature/Checksum: ok > > > * Rat check (1.8.0_341): ok > > > - mvn clean apache-rat:check > > > * Built from source (1.8.0_341): ok > > > - mvn clean install -DskipTests > > > * Built tar from source (1.8.0_341): ok > > > - mvn clean package -Pdist -DskipTests -Dtar > -Dmaven.javadoc.skip=true > > > > > > Containerized deployments: > > > * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper > 3.7 > > > * Deployed and started JHS, RM, NM > > > * Hbase, hdfs CRUD looks good > > > * Sample RowCount MapReduce job looks good > > > > > > * S3A tests with scale profile looks good > > > > > > > > > On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran > > > > > > wrote: > > > > > > > Apache Hadoop 3.3.5 > > > > > > > > Mukund and I have put together a release candidate (RC3) for Hadoop > > > 3.3.5. > > > > > > > > What we would like is for anyone who can to verify the tarballs, > > > especially > > > > anyone who can try the arm64 binaries as we want to include them too. > > > > > > > > The RC is available at: > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > > > > > > > The git tag is release-3.3.5-RC3, commit 706d88266ab > > > > > > > > The maven artifacts are staged at > > > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > > > > > > > You can find my public key at: > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > > > Change log > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > > > > > > > Release notes > > > > > > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > > > > > > > > This is off branch-3.3 and is the first big release since 3.3.2. > > > > > > > > Key changes include > > > > > > > > * Big update of dependencies to try and keep those reports of > > > > transitive CVEs under control -both genuine and false positives. > > > > * HDFS RBF enhancements > > > > * Critical fix to ABFS input stream prefetching for correct reading. > > > > * Vectored IO API for all FSDataInputStream implementations, with > > > > high-performance versions for file:// and s3a:// filesystems. > > > > file:// through java native io > > > > s3a:// parallel GET requests. > > > > * This release includes Arm64 binaries. Please can anyone with > > > > compatible systems validate these. > > > > * and compared to the previous RC, all the major changes are > > > > HDFS issues. > > > > > > > > Note, because the arm64 binaries are built separately on a different > > > > platform and JVM, their jar files may not match those of the x86 > > > > release -and therefore the maven artifacts. I don't think this is > > > > an issue (the ASF actually releases source tarballs, the binaries are > > > > there for help only, though with the maven repo that's a bit > blurred). > > > > > > > > The only way to be consistent would actually untar the x86.tar.gz, > > > > overwrite its binaries with the arm stuff, retar, sign and push out > > > > for the vote. Even automating that would be risky. > > > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > > > -Steve > > > > > > > > > >
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
Hi Steve I will also do it by today/tomorrow. Thanks, Ashutosh On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, wrote: > Thank you for this! > > Can anyone else with time do a review too? i really want to get this one > done, now the HDFS issues are all resolved. > > I do not want this release to fall by the wayside through lack of votes > alone. In fact, I would be very unhappy > > > > On Sat, 18 Mar 2023 at 06:47, Viraj Jasani wrote: > > > +1 (non-binding) > > > > * Signature/Checksum: ok > > * Rat check (1.8.0_341): ok > > - mvn clean apache-rat:check > > * Built from source (1.8.0_341): ok > > - mvn clean install -DskipTests > > * Built tar from source (1.8.0_341): ok > > - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true > > > > Containerized deployments: > > * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7 > > * Deployed and started JHS, RM, NM > > * Hbase, hdfs CRUD looks good > > * Sample RowCount MapReduce job looks good > > > > * S3A tests with scale profile looks good > > > > > > On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran > > > > wrote: > > > > > Apache Hadoop 3.3.5 > > > > > > Mukund and I have put together a release candidate (RC3) for Hadoop > > 3.3.5. > > > > > > What we would like is for anyone who can to verify the tarballs, > > especially > > > anyone who can try the arm64 binaries as we want to include them too. > > > > > > The RC is available at: > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > > > > > The git tag is release-3.3.5-RC3, commit 706d88266ab > > > > > > The maven artifacts are staged at > > > > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > > > > > You can find my public key at: > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > > > Change log > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > > > > > Release notes > > > > > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > > > > > > This is off branch-3.3 and is the first big release since 3.3.2. > > > > > > Key changes include > > > > > > * Big update of dependencies to try and keep those reports of > > > transitive CVEs under control -both genuine and false positives. > > > * HDFS RBF enhancements > > > * Critical fix to ABFS input stream prefetching for correct reading. > > > * Vectored IO API for all FSDataInputStream implementations, with > > > high-performance versions for file:// and s3a:// filesystems. > > > file:// through java native io > > > s3a:// parallel GET requests. > > > * This release includes Arm64 binaries. Please can anyone with > > > compatible systems validate these. > > > * and compared to the previous RC, all the major changes are > > > HDFS issues. > > > > > > Note, because the arm64 binaries are built separately on a different > > > platform and JVM, their jar files may not match those of the x86 > > > release -and therefore the maven artifacts. I don't think this is > > > an issue (the ASF actually releases source tarballs, the binaries are > > > there for help only, though with the maven repo that's a bit blurred). > > > > > > The only way to be consistent would actually untar the x86.tar.gz, > > > overwrite its binaries with the arm stuff, retar, sign and push out > > > for the vote. Even automating that would be risky. > > > > > > Please try the release and vote. The vote will run for 5 days. > > > > > > -Steve > > > > > >
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
Thank you for this! Can anyone else with time do a review too? i really want to get this one done, now the HDFS issues are all resolved. I do not want this release to fall by the wayside through lack of votes alone. In fact, I would be very unhappy On Sat, 18 Mar 2023 at 06:47, Viraj Jasani wrote: > +1 (non-binding) > > * Signature/Checksum: ok > * Rat check (1.8.0_341): ok > - mvn clean apache-rat:check > * Built from source (1.8.0_341): ok > - mvn clean install -DskipTests > * Built tar from source (1.8.0_341): ok > - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true > > Containerized deployments: > * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7 > * Deployed and started JHS, RM, NM > * Hbase, hdfs CRUD looks good > * Sample RowCount MapReduce job looks good > > * S3A tests with scale profile looks good > > > On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran > > wrote: > > > Apache Hadoop 3.3.5 > > > > Mukund and I have put together a release candidate (RC3) for Hadoop > 3.3.5. > > > > What we would like is for anyone who can to verify the tarballs, > especially > > anyone who can try the arm64 binaries as we want to include them too. > > > > The RC is available at: > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > > > The git tag is release-3.3.5-RC3, commit 706d88266ab > > > > The maven artifacts are staged at > > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > > > You can find my public key at: > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > > > Change log > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > > > Release notes > > > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > > > > This is off branch-3.3 and is the first big release since 3.3.2. > > > > Key changes include > > > > * Big update of dependencies to try and keep those reports of > > transitive CVEs under control -both genuine and false positives. > > * HDFS RBF enhancements > > * Critical fix to ABFS input stream prefetching for correct reading. > > * Vectored IO API for all FSDataInputStream implementations, with > > high-performance versions for file:// and s3a:// filesystems. > > file:// through java native io > > s3a:// parallel GET requests. > > * This release includes Arm64 binaries. Please can anyone with > > compatible systems validate these. > > * and compared to the previous RC, all the major changes are > > HDFS issues. > > > > Note, because the arm64 binaries are built separately on a different > > platform and JVM, their jar files may not match those of the x86 > > release -and therefore the maven artifacts. I don't think this is > > an issue (the ASF actually releases source tarballs, the binaries are > > there for help only, though with the maven repo that's a bit blurred). > > > > The only way to be consistent would actually untar the x86.tar.gz, > > overwrite its binaries with the arm stuff, retar, sign and push out > > for the vote. Even automating that would be risky. > > > > Please try the release and vote. The vote will run for 5 days. > > > > -Steve > > >
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
+1 (non-binding) * Signature/Checksum: ok * Rat check (1.8.0_341): ok - mvn clean apache-rat:check * Built from source (1.8.0_341): ok - mvn clean install -DskipTests * Built tar from source (1.8.0_341): ok - mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true Containerized deployments: * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7 * Deployed and started JHS, RM, NM * Hbase, hdfs CRUD looks good * Sample RowCount MapReduce job looks good * S3A tests with scale profile looks good On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran wrote: > Apache Hadoop 3.3.5 > > Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5. > > What we would like is for anyone who can to verify the tarballs, especially > anyone who can try the arm64 binaries as we want to include them too. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > The git tag is release-3.3.5-RC3, commit 706d88266ab > > The maven artifacts are staged at > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > You can find my public key at: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > Change log > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > Release notes > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > > This is off branch-3.3 and is the first big release since 3.3.2. > > Key changes include > > * Big update of dependencies to try and keep those reports of > transitive CVEs under control -both genuine and false positives. > * HDFS RBF enhancements > * Critical fix to ABFS input stream prefetching for correct reading. > * Vectored IO API for all FSDataInputStream implementations, with > high-performance versions for file:// and s3a:// filesystems. > file:// through java native io > s3a:// parallel GET requests. > * This release includes Arm64 binaries. Please can anyone with > compatible systems validate these. > * and compared to the previous RC, all the major changes are > HDFS issues. > > Note, because the arm64 binaries are built separately on a different > platform and JVM, their jar files may not match those of the x86 > release -and therefore the maven artifacts. I don't think this is > an issue (the ASF actually releases source tarballs, the binaries are > there for help only, though with the maven repo that's a bit blurred). > > The only way to be consistent would actually untar the x86.tar.gz, > overwrite its binaries with the arm stuff, retar, sign and push out > for the vote. Even automating that would be risky. > > Please try the release and vote. The vote will run for 5 days. > > -Steve >
Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)
and my vote My vote +1 binding I've been using the RCs for a while as my CLI entry point, and testing it through other builds for this RC * Local builds of cloudstore * fs-api-shim * spark * built and ran my cloud integration tests, which now include large CVS file jobs which should show the Azure prefetch bug if it still existed. downloaded the tar, expanded it, ran command line code with it, including cloudstore against the stores. we need to get hadoop-azure and its dependencies onto the path by default, to make abfs io easier. I have the arm binaries building, and did a checknative to make sure all was good stevel@0da162643f99:~/hadoop/patchprocess/hadoop-3.3.5$ bin/hadoop checknative 2023-03-17 13:00:27,107 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native 2023-03-17 13:00:27,112 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library 2023-03-17 13:00:27,121 WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable 2023-03-17 13:00:27,156 INFO nativeio.NativeIO: The native code was built without PMDK support. Native library checking: hadoop: true /home/stevel/hadoop/patchprocess/hadoop-3.3.5/lib/native/libhadoop.so.1.0.0 zlib:true /lib/aarch64-linux-gnu/libz.so.1 zstd : true /lib/aarch64-linux-gnu/libzstd.so.1 bzip2: true /lib/aarch64-linux-gnu/libbz2.so.1 openssl: true /lib/aarch64-linux-gnu/libcrypto.so ISA-L: false libhadoop was built without ISA-L support PMDK:false The native code was built without PMDK support. --- On Wed, 15 Mar 2023 at 19:47, Steve Loughran wrote: > > Apache Hadoop 3.3.5 > > Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5. > > What we would like is for anyone who can to verify the tarballs, especially > anyone who can try the arm64 binaries as we want to include them too. > > The RC is available at: > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/ > > The git tag is release-3.3.5-RC3, commit 706d88266ab > > The maven artifacts are staged at > https://repository.apache.org/content/repositories/orgapachehadoop-1369/ > > You can find my public key at: > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS > > Change log > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md > > Release notes > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md > > This is off branch-3.3 and is the first big release since 3.3.2. > > Key changes include > > * Big update of dependencies to try and keep those reports of > transitive CVEs under control -both genuine and false positives. > * HDFS RBF enhancements > * Critical fix to ABFS input stream prefetching for correct reading. > * Vectored IO API for all FSDataInputStream implementations, with > high-performance versions for file:// and s3a:// filesystems. > file:// through java native io > s3a:// parallel GET requests. > * This release includes Arm64 binaries. Please can anyone with > compatible systems validate these. > * and compared to the previous RC, all the major changes are > HDFS issues. > > Note, because the arm64 binaries are built separately on a different > platform and JVM, their jar files may not match those of the x86 > release -and therefore the maven artifacts. I don't think this is > an issue (the ASF actually releases source tarballs, the binaries are > there for help only, though with the maven repo that's a bit blurred). > > The only way to be consistent would actually untar the x86.tar.gz, > overwrite its binaries with the arm stuff, retar, sign and push out > for the vote. Even automating that would be risky. > > Please try the release and vote. The vote will run for 5 days. > > -Steve >
