Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

go ahead and cut it...I'd forgotten about it.

thanks,

steve

On Fri, 31 Mar 2023 at 06:44, Ayush Saxena  wrote:

> We have a daily build running for 3.3.5:
> https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/
>
> We have already released it, so I feel we can disable it. Will do it
> tomorrow, if nobody objects. In case the one who configured it wants
> to do it early, feel free to do so.
>
> We already have one for branch-3.3 which runs weekly which most
> probably most of us don't follow :)
>
> -Ayush
>
> On Wed, 22 Mar 2023 at 00:20, Steve Loughran
>  wrote:
> >
> > ok, here's my summary, even though most of the binding voters forgot to
> > declare they were on the PMC.
> >
> > +1 binding
> >
> > Steve Loughran
> > Chris Nauroth
> > Masatake Iwasaki
> > Ayush Saxena
> > Xiaoqiao He
> >
> > +1 non-binding
> >
> > Viraj Jasani
> >
> >
> > 0 or -1 votes: none.
> >
> >
> > Accordingly: the release is good!
> >
> > I will send the formal announcement out tomorrow
> >
> > A big thank you to everyone who qualified the RC, I know its a lot of
> work.
> > We can now get this out and *someone else* can plan the followup.
> >
> >
> > steve
> >
> > On Mon, 20 Mar 2023 at 16:01, Chris Nauroth  wrote:
> >
> > > +1
> > >
> > > Thank you for the release candidate, Steve!
> > >
> > > * Verified all checksums.
> > > * Verified all signatures.
> > > * Built from source, including native code on Linux.
> > > * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> > > -Drequire.zstd -DskipTests
> > > * Tests passed.
> > > * mvn --fail-never clean test -Pnative -Dparallel-tests
> > > -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > > * Checked dependency tree to make sure we have all of the expected
> library
> > > updates that are mentioned in the release notes.
> > > * mvn -o dependency:tree
> > > * Confirmed that hadoop-openstack is now just a stub placeholder
> artifact
> > > with no code.
> > > * For ARM verification:
> > > * Ran "file " on all native binaries in the ARM tarball to
> confirm
> > > they actually came out with ARM as the architecture.
> > > * Output of hadoop checknative -a on ARM looks good.
> > > * Ran a MapReduce job with the native bzip2 codec for compression,
> and
> > > it worked fine.
> > > * Ran a MapReduce job with YARN configured to use
> > > LinuxContainerExecutor and verified launching the containers through
> > > container-executor worked.
> > >
> > > Chris Nauroth
> > >
> > >
> > > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena 
> wrote:
> > >
> > > > +1(Binding)
> > > >
> > > > * Built from source (x86 & ARM)
> > > > * Successful Native Build (x86 & ARM)
> > > > * Verified Checksums (x86 & ARM)
> > > > * Verified Signature (x86 & ARM)
> > > > * Checked the output of hadoop version (x86 & ARM)
> > > > * Verified the output of hadoop checknative (x86 & ARM)
> > > > * Ran some basic HDFS shell commands.
> > > > * Ran some basic Yarn shell commands.
> > > > * Played a bit with HDFS Erasure Coding.
> > > > * Ran TeraGen & TeraSort
> > > > * Browed through NN, DN, RM & NM UI
> > > > * Skimmed over the contents of website.
> > > > * Skimmed over the contents of maven repo.
> > > > * Selectively ran some HDFS & CloudStore tests
> > > >
> > > > Thanx Steve for driving the release. Good Luck!!!
> > > >
> > > > -Ayush
> > > >
> > > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He 
> > > wrote:
> > > > >
> > > > > +1
> > > > >
> > > > > * Verified signature and checksum of the source tarball.
> > > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean
> package
> > > > > -DskipTests -Pnative -Pdist -Dtar`.
> > > > > * Setup pseudo cluster with HDFS and YARN.
> > > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check
> the
> > > > > result.
> > > > > * Run example mr applications and check the result - Pi &
> wordcount.
> > > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> > > etc.
> > > > >
> > > > > Thanks Steve for your work.
> > > > >
> > > > > Best Regards,
> > > > > - He Xiaoqiao
> > > > >
> > > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki <
> > > > iwasak...@oss.nttdata.com>
> > > > >> wrote:
> > > > >>
> > > > >> +1
> > > > >>
> > > > >> + verified the signature and checksum of the source tarball.
> > > > >>
> > > > >> + built from the source tarball on Rocky Linux 8 (x86_64) and
> OpenJDK
> > > 8
> > > > >> with native profile enabled.
> > > > >>   + launched pseudo distributed cluster including kms and httpfs
> with
> > > > >> Kerberos and SSL enabled.
> > > > >>   + created encryption zone, put and read files via httpfs.
> > > > >>   + ran example MR wordcount over encryption zone.
> > > > >>   + checked the binary of container-executor.
> > > > >>
> > > > >> + built rpm packages by Bigtop (with trivial modifications) on
> Rocky
> > > > Linux
> > > > >> 8 (aarch64).
> > > > >>   + ran smoke-tests of 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ayush Saxena]

We have a daily build running for 3.3.5:
https://ci-hadoop.apache.org/job/hadoop-qbt-3.3.5-java8-linux-x86_64/

We have already released it, so I feel we can disable it. Will do it
tomorrow, if nobody objects. In case the one who configured it wants
to do it early, feel free to do so.

We already have one for branch-3.3 which runs weekly which most
probably most of us don't follow :)

-Ayush

On Wed, 22 Mar 2023 at 00:20, Steve Loughran
 wrote:
>
> ok, here's my summary, even though most of the binding voters forgot to
> declare they were on the PMC.
>
> +1 binding
>
> Steve Loughran
> Chris Nauroth
> Masatake Iwasaki
> Ayush Saxena
> Xiaoqiao He
>
> +1 non-binding
>
> Viraj Jasani
>
>
> 0 or -1 votes: none.
>
>
> Accordingly: the release is good!
>
> I will send the formal announcement out tomorrow
>
> A big thank you to everyone who qualified the RC, I know its a lot of work.
> We can now get this out and *someone else* can plan the followup.
>
>
> steve
>
> On Mon, 20 Mar 2023 at 16:01, Chris Nauroth  wrote:
>
> > +1
> >
> > Thank you for the release candidate, Steve!
> >
> > * Verified all checksums.
> > * Verified all signatures.
> > * Built from source, including native code on Linux.
> > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> > -Drequire.zstd -DskipTests
> > * Tests passed.
> > * mvn --fail-never clean test -Pnative -Dparallel-tests
> > -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > * Checked dependency tree to make sure we have all of the expected library
> > updates that are mentioned in the release notes.
> > * mvn -o dependency:tree
> > * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> > with no code.
> > * For ARM verification:
> > * Ran "file " on all native binaries in the ARM tarball to confirm
> > they actually came out with ARM as the architecture.
> > * Output of hadoop checknative -a on ARM looks good.
> > * Ran a MapReduce job with the native bzip2 codec for compression, and
> > it worked fine.
> > * Ran a MapReduce job with YARN configured to use
> > LinuxContainerExecutor and verified launching the containers through
> > container-executor worked.
> >
> > Chris Nauroth
> >
> >
> > On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena  wrote:
> >
> > > +1(Binding)
> > >
> > > * Built from source (x86 & ARM)
> > > * Successful Native Build (x86 & ARM)
> > > * Verified Checksums (x86 & ARM)
> > > * Verified Signature (x86 & ARM)
> > > * Checked the output of hadoop version (x86 & ARM)
> > > * Verified the output of hadoop checknative (x86 & ARM)
> > > * Ran some basic HDFS shell commands.
> > > * Ran some basic Yarn shell commands.
> > > * Played a bit with HDFS Erasure Coding.
> > > * Ran TeraGen & TeraSort
> > > * Browed through NN, DN, RM & NM UI
> > > * Skimmed over the contents of website.
> > > * Skimmed over the contents of maven repo.
> > > * Selectively ran some HDFS & CloudStore tests
> > >
> > > Thanx Steve for driving the release. Good Luck!!!
> > >
> > > -Ayush
> > >
> > > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He 
> > wrote:
> > > >
> > > > +1
> > > >
> > > > * Verified signature and checksum of the source tarball.
> > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> > > > -DskipTests -Pnative -Pdist -Dtar`.
> > > > * Setup pseudo cluster with HDFS and YARN.
> > > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> > > > result.
> > > > * Run example mr applications and check the result - Pi & wordcount.
> > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> > etc.
> > > >
> > > > Thanks Steve for your work.
> > > >
> > > > Best Regards,
> > > > - He Xiaoqiao
> > > >
> > > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki <
> > > iwasak...@oss.nttdata.com>
> > > >> wrote:
> > > >>
> > > >> +1
> > > >>
> > > >> + verified the signature and checksum of the source tarball.
> > > >>
> > > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK
> > 8
> > > >> with native profile enabled.
> > > >>   + launched pseudo distributed cluster including kms and httpfs with
> > > >> Kerberos and SSL enabled.
> > > >>   + created encryption zone, put and read files via httpfs.
> > > >>   + ran example MR wordcount over encryption zone.
> > > >>   + checked the binary of container-executor.
> > > >>
> > > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky
> > > Linux
> > > >> 8 (aarch64).
> > > >>   + ran smoke-tests of hdfs, yarn and mapreduce.
> > > >> + built site documentation and skimmed the contents.
> > > >>   +  Javadocs are contained.
> > > >>
> > > >> Thanks,
> > > >> Masatake Iwasaki
> > > >>
> > > >>> On 2023/03/16 4:47, Steve Loughran wrote:
> > > >>> Apache Hadoop 3.3.5
> > > >>>
> > > >>> Mukund and I have put together a release candidate (RC3) for Hadoop
> > > >> 3.3.5.
> > > >>>
> > > >>> What we would like is for 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

ok, here's my summary, even though most of the binding voters forgot to
declare they were on the PMC.

+1 binding

Steve Loughran
Chris Nauroth
Masatake Iwasaki
Ayush Saxena
Xiaoqiao He

+1 non-binding

Viraj Jasani


0 or -1 votes: none.


Accordingly: the release is good!

I will send the formal announcement out tomorrow

A big thank you to everyone who qualified the RC, I know its a lot of work.
We can now get this out and *someone else* can plan the followup.


steve

On Mon, 20 Mar 2023 at 16:01, Chris Nauroth  wrote:

> +1
>
> Thank you for the release candidate, Steve!
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file " on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> Chris Nauroth
>
>
> On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena  wrote:
>
> > +1(Binding)
> >
> > * Built from source (x86 & ARM)
> > * Successful Native Build (x86 & ARM)
> > * Verified Checksums (x86 & ARM)
> > * Verified Signature (x86 & ARM)
> > * Checked the output of hadoop version (x86 & ARM)
> > * Verified the output of hadoop checknative (x86 & ARM)
> > * Ran some basic HDFS shell commands.
> > * Ran some basic Yarn shell commands.
> > * Played a bit with HDFS Erasure Coding.
> > * Ran TeraGen & TeraSort
> > * Browed through NN, DN, RM & NM UI
> > * Skimmed over the contents of website.
> > * Skimmed over the contents of maven repo.
> > * Selectively ran some HDFS & CloudStore tests
> >
> > Thanx Steve for driving the release. Good Luck!!!
> >
> > -Ayush
> >
> > > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He 
> wrote:
> > >
> > > +1
> > >
> > > * Verified signature and checksum of the source tarball.
> > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> > > -DskipTests -Pnative -Pdist -Dtar`.
> > > * Setup pseudo cluster with HDFS and YARN.
> > > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> > > result.
> > > * Run example mr applications and check the result - Pi & wordcount.
> > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> > >
> > > Thanks Steve for your work.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki <
> > iwasak...@oss.nttdata.com>
> > >> wrote:
> > >>
> > >> +1
> > >>
> > >> + verified the signature and checksum of the source tarball.
> > >>
> > >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK
> 8
> > >> with native profile enabled.
> > >>   + launched pseudo distributed cluster including kms and httpfs with
> > >> Kerberos and SSL enabled.
> > >>   + created encryption zone, put and read files via httpfs.
> > >>   + ran example MR wordcount over encryption zone.
> > >>   + checked the binary of container-executor.
> > >>
> > >> + built rpm packages by Bigtop (with trivial modifications) on Rocky
> > Linux
> > >> 8 (aarch64).
> > >>   + ran smoke-tests of hdfs, yarn and mapreduce.
> > >> + built site documentation and skimmed the contents.
> > >>   +  Javadocs are contained.
> > >>
> > >> Thanks,
> > >> Masatake Iwasaki
> > >>
> > >>> On 2023/03/16 4:47, Steve Loughran wrote:
> > >>> Apache Hadoop 3.3.5
> > >>>
> > >>> Mukund and I have put together a release candidate (RC3) for Hadoop
> > >> 3.3.5.
> > >>>
> > >>> What we would like is for anyone who can to verify the tarballs,
> > >> especially
> > >>> anyone who can try the arm64 binaries as we want to include them too.
> > >>>
> > >>> The RC is available at:
> > >>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> > >>>
> > >>> The git tag is release-3.3.5-RC3, commit 706d88266ab
> > >>>
> > >>> The maven artifacts are staged at
> > >>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > >>>
> > >>> You can find my public key at:
> > >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>
> > >>> Change log
> > >>>
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> > >>>
> > >>> Release 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Chris Nauroth]

+1

Thank you for the release candidate, Steve!

* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests passed.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
* Checked dependency tree to make sure we have all of the expected library
updates that are mentioned in the release notes.
* mvn -o dependency:tree
* Confirmed that hadoop-openstack is now just a stub placeholder artifact
with no code.
* For ARM verification:
* Ran "file " on all native binaries in the ARM tarball to confirm
they actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and
it worked fine.
* Ran a MapReduce job with YARN configured to use
LinuxContainerExecutor and verified launching the containers through
container-executor worked.

Chris Nauroth


On Mon, Mar 20, 2023 at 3:45 AM Ayush Saxena  wrote:

> +1(Binding)
>
> * Built from source (x86 & ARM)
> * Successful Native Build (x86 & ARM)
> * Verified Checksums (x86 & ARM)
> * Verified Signature (x86 & ARM)
> * Checked the output of hadoop version (x86 & ARM)
> * Verified the output of hadoop checknative (x86 & ARM)
> * Ran some basic HDFS shell commands.
> * Ran some basic Yarn shell commands.
> * Played a bit with HDFS Erasure Coding.
> * Ran TeraGen & TeraSort
> * Browed through NN, DN, RM & NM UI
> * Skimmed over the contents of website.
> * Skimmed over the contents of maven repo.
> * Selectively ran some HDFS & CloudStore tests
>
> Thanx Steve for driving the release. Good Luck!!!
>
> -Ayush
>
> > On 20-Mar-2023, at 12:54 PM, Xiaoqiao He  wrote:
> >
> > +1
> >
> > * Verified signature and checksum of the source tarball.
> > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> > -DskipTests -Pnative -Pdist -Dtar`.
> > * Setup pseudo cluster with HDFS and YARN.
> > * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> > result.
> > * Run example mr applications and check the result - Pi & wordcount.
> > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
> >
> > Thanks Steve for your work.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> >> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki <
> iwasak...@oss.nttdata.com>
> >> wrote:
> >>
> >> +1
> >>
> >> + verified the signature and checksum of the source tarball.
> >>
> >> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8
> >> with native profile enabled.
> >>   + launched pseudo distributed cluster including kms and httpfs with
> >> Kerberos and SSL enabled.
> >>   + created encryption zone, put and read files via httpfs.
> >>   + ran example MR wordcount over encryption zone.
> >>   + checked the binary of container-executor.
> >>
> >> + built rpm packages by Bigtop (with trivial modifications) on Rocky
> Linux
> >> 8 (aarch64).
> >>   + ran smoke-tests of hdfs, yarn and mapreduce.
> >> + built site documentation and skimmed the contents.
> >>   +  Javadocs are contained.
> >>
> >> Thanks,
> >> Masatake Iwasaki
> >>
> >>> On 2023/03/16 4:47, Steve Loughran wrote:
> >>> Apache Hadoop 3.3.5
> >>>
> >>> Mukund and I have put together a release candidate (RC3) for Hadoop
> >> 3.3.5.
> >>>
> >>> What we would like is for anyone who can to verify the tarballs,
> >> especially
> >>> anyone who can try the arm64 binaries as we want to include them too.
> >>>
> >>> The RC is available at:
> >>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> >>>
> >>> The git tag is release-3.3.5-RC3, commit 706d88266ab
> >>>
> >>> The maven artifacts are staged at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >>>
> >>> You can find my public key at:
> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>
> >>> Change log
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> >>>
> >>> Release notes
> >>>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> >>>
> >>> This is off branch-3.3 and is the first big release since 3.3.2.
> >>>
> >>> Key changes include
> >>>
> >>> * Big update of dependencies to try and keep those reports of
> >>>   transitive CVEs under control -both genuine and false positives.
> >>> * HDFS RBF enhancements
> >>> * Critical fix to ABFS input stream prefetching for correct reading.
> >>> * Vectored IO API for all FSDataInputStream implementations, with
> >>>   high-performance versions for file:// and s3a:// filesystems.
> >>>   file:// through java native io
> >>>   s3a:// parallel GET requests.
> >>> * This release includes Arm64 binaries. Please can anyone with
> >>>   compatible systems validate 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ayush Saxena]

+1(Binding)

* Built from source (x86 & ARM)
* Successful Native Build (x86 & ARM)
* Verified Checksums (x86 & ARM)
* Verified Signature (x86 & ARM)
* Checked the output of hadoop version (x86 & ARM)
* Verified the output of hadoop checknative (x86 & ARM)
* Ran some basic HDFS shell commands.
* Ran some basic Yarn shell commands.
* Played a bit with HDFS Erasure Coding.
* Ran TeraGen & TeraSort
* Browed through NN, DN, RM & NM UI
* Skimmed over the contents of website.
* Skimmed over the contents of maven repo.
* Selectively ran some HDFS & CloudStore tests

Thanx Steve for driving the release. Good Luck!!!

-Ayush

> On 20-Mar-2023, at 12:54 PM, Xiaoqiao He  wrote:
> 
> +1
> 
> * Verified signature and checksum of the source tarball.
> * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> * Setup pseudo cluster with HDFS and YARN.
> * Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> * Run example mr applications and check the result - Pi & wordcount.
> * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
> 
> Thanks Steve for your work.
> 
> Best Regards,
> - He Xiaoqiao
> 
>> On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki 
>> wrote:
>> 
>> +1
>> 
>> + verified the signature and checksum of the source tarball.
>> 
>> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8
>> with native profile enabled.
>>   + launched pseudo distributed cluster including kms and httpfs with
>> Kerberos and SSL enabled.
>>   + created encryption zone, put and read files via httpfs.
>>   + ran example MR wordcount over encryption zone.
>>   + checked the binary of container-executor.
>> 
>> + built rpm packages by Bigtop (with trivial modifications) on Rocky Linux
>> 8 (aarch64).
>>   + ran smoke-tests of hdfs, yarn and mapreduce.
>> + built site documentation and skimmed the contents.
>>   +  Javadocs are contained.
>> 
>> Thanks,
>> Masatake Iwasaki
>> 
>>> On 2023/03/16 4:47, Steve Loughran wrote:
>>> Apache Hadoop 3.3.5
>>> 
>>> Mukund and I have put together a release candidate (RC3) for Hadoop
>> 3.3.5.
>>> 
>>> What we would like is for anyone who can to verify the tarballs,
>> especially
>>> anyone who can try the arm64 binaries as we want to include them too.
>>> 
>>> The RC is available at:
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
>>> 
>>> The git tag is release-3.3.5-RC3, commit 706d88266ab
>>> 
>>> The maven artifacts are staged at
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>>> 
>>> You can find my public key at:
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> 
>>> Change log
>>> 
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
>>> 
>>> Release notes
>>> 
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
>>> 
>>> This is off branch-3.3 and is the first big release since 3.3.2.
>>> 
>>> Key changes include
>>> 
>>> * Big update of dependencies to try and keep those reports of
>>>   transitive CVEs under control -both genuine and false positives.
>>> * HDFS RBF enhancements
>>> * Critical fix to ABFS input stream prefetching for correct reading.
>>> * Vectored IO API for all FSDataInputStream implementations, with
>>>   high-performance versions for file:// and s3a:// filesystems.
>>>   file:// through java native io
>>>   s3a:// parallel GET requests.
>>> * This release includes Arm64 binaries. Please can anyone with
>>>   compatible systems validate these.
>>> * and compared to the previous RC, all the major changes are
>>>   HDFS issues.
>>> 
>>> Note, because the arm64 binaries are built separately on a different
>>> platform and JVM, their jar files may not match those of the x86
>>> release -and therefore the maven artifacts. I don't think this is
>>> an issue (the ASF actually releases source tarballs, the binaries are
>>> there for help only, though with the maven repo that's a bit blurred).
>>> 
>>> The only way to be consistent would actually untar the x86.tar.gz,
>>> overwrite its binaries with the arm stuff, retar, sign and push out
>>> for the vote. Even automating that would be risky.
>>> 
>>> Please try the release and vote. The vote will run for 5 days.
>>> 
>>> -Steve
>>> 
>> 
>> -
>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> 
>> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Xiaoqiao He]

+1

* Verified signature and checksum of the source tarball.
* Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
* Setup pseudo cluster with HDFS and YARN.
* Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
result.
* Run example mr applications and check the result - Pi & wordcount.
* Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.

Thanks Steve for your work.

Best Regards,
- He Xiaoqiao

On Mon, Mar 20, 2023 at 12:04 PM Masatake Iwasaki 
wrote:

> +1
>
> + verified the signature and checksum of the source tarball.
>
> + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8
> with native profile enabled.
>+ launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
>+ created encryption zone, put and read files via httpfs.
>+ ran example MR wordcount over encryption zone.
>+ checked the binary of container-executor.
>
> + built rpm packages by Bigtop (with trivial modifications) on Rocky Linux
> 8 (aarch64).
>+ ran smoke-tests of hdfs, yarn and mapreduce.
> + built site documentation and skimmed the contents.
>+  Javadocs are contained.
>
> Thanks,
> Masatake Iwasaki
>
> On 2023/03/16 4:47, Steve Loughran wrote:
> > Apache Hadoop 3.3.5
> >
> > Mukund and I have put together a release candidate (RC3) for Hadoop
> 3.3.5.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> >
> > The git tag is release-3.3.5-RC3, commit 706d88266ab
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> >
> > Release notes
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > Key changes include
> >
> > * Big update of dependencies to try and keep those reports of
> >transitive CVEs under control -both genuine and false positives.
> > * HDFS RBF enhancements
> > * Critical fix to ABFS input stream prefetching for correct reading.
> > * Vectored IO API for all FSDataInputStream implementations, with
> >high-performance versions for file:// and s3a:// filesystems.
> >file:// through java native io
> >s3a:// parallel GET requests.
> > * This release includes Arm64 binaries. Please can anyone with
> >compatible systems validate these.
> > * and compared to the previous RC, all the major changes are
> >HDFS issues.
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Masatake Iwasaki]

+1

+ verified the signature and checksum of the source tarball.

+ built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8 with 
native profile enabled.
  + launched pseudo distributed cluster including kms and httpfs with Kerberos 
and SSL enabled.
  + created encryption zone, put and read files via httpfs.
  + ran example MR wordcount over encryption zone.
  + checked the binary of container-executor.

+ built rpm packages by Bigtop (with trivial modifications) on Rocky Linux 8 
(aarch64).
  + ran smoke-tests of hdfs, yarn and mapreduce.
+ built site documentation and skimmed the contents.
  +  Javadocs are contained.

Thanks,
Masatake Iwasaki

On 2023/03/16 4:47, Steve Loughran wrote:

Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/

The git tag is release-3.3.5-RC3, commit 706d88266ab

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
   transitive CVEs under control -both genuine and false positives.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
   high-performance versions for file:// and s3a:// filesystems.
   file:// through java native io
   s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
   compatible systems validate these.
* and compared to the previous RC, all the major changes are
   HDFS issues.

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

-Steve



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ayush Saxena]

Count me in as well. I am almost done. So, you have 3 potential votes, can be 
happy now :) 
Thanx Steve for the efforts!!!

-Ayush

> On 19-Mar-2023, at 2:46 AM, Chris Nauroth  wrote:
> 
> Yes, I'm in progress on verification, so you can expect to get a vote from
> me. Thank you, Steve!
> 
> Chris Nauroth
> 
> 
>> On Sat, Mar 18, 2023 at 9:19 AM Ashutosh Gupta 
>> wrote:
>> Hi Steve
>> I will also do it by today/tomorrow.
>> Thanks,
>> Ashutosh
>> On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, > wrote:
>>> Thank you for this!
>>> Can anyone else with time do a review too? i really want to get this one
>>> done, now the HDFS issues are all resolved.
>>> I do not want this release to fall by the wayside through lack of votes
>>> alone. In fact, I would be very unhappy
 On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:
 +1 (non-binding)
 * Signature/Checksum: ok
 * Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
 * Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
 * Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
 Containerized deployments:
 * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper
>> 3.7
 * Deployed and started JHS, RM, NM
 * Hbase, hdfs CRUD looks good
 * Sample RowCount MapReduce job looks good
 * S3A tests with scale profile looks good
 On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
 
 wrote:
> Apache Hadoop 3.3.5
> Mukund and I have put together a release candidate (RC3) for Hadoop
 3.3.5.
> What we would like is for anyone who can to verify the tarballs,
 especially
> anyone who can try the arm64 binaries as we want to include them too.
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> The git tag is release-3.3.5-RC3, commit 706d88266ab
> The maven artifacts are staged at
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> Change log
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> Release notes
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> This is off branch-3.3 and is the first big release since 3.3.2.
> Key changes include
> * Big update of dependencies to try and keep those reports of
> transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
> high-performance versions for file:// and s3a:// filesystems.
> file:// through java native io
> s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
> compatible systems validate these.
> * and compared to the previous RC, all the major changes are
> HDFS issues.
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit
>> blurred).
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
> Please try the release and vote. The vote will run for 5 days.
> -Steve

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Chris Nauroth]

Yes, I'm in progress on verification, so you can expect to get a vote from
me. Thank you, Steve!

Chris Nauroth


On Sat, Mar 18, 2023 at 9:19 AM Ashutosh Gupta 
wrote:

> Hi Steve
>
> I will also do it by today/tomorrow.
>
> Thanks,
> Ashutosh
>
> On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran,  >
> wrote:
>
> > Thank you for this!
> >
> > Can anyone else with time do a review too? i really want to get this one
> > done, now the HDFS issues are all resolved.
> >
> > I do not want this release to fall by the wayside through lack of votes
> > alone. In fact, I would be very unhappy
> >
> >
> >
> > On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:
> >
> > > +1 (non-binding)
> > >
> > > * Signature/Checksum: ok
> > > * Rat check (1.8.0_341): ok
> > >  - mvn clean apache-rat:check
> > > * Built from source (1.8.0_341): ok
> > >  - mvn clean install  -DskipTests
> > > * Built tar from source (1.8.0_341): ok
> > >  - mvn clean package  -Pdist -DskipTests -Dtar
> -Dmaven.javadoc.skip=true
> > >
> > > Containerized deployments:
> > > * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper
> 3.7
> > > * Deployed and started JHS, RM, NM
> > > * Hbase, hdfs CRUD looks good
> > > * Sample RowCount MapReduce job looks good
> > >
> > > * S3A tests with scale profile looks good
> > >
> > >
> > > On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
> > > 
> > > wrote:
> > >
> > > > Apache Hadoop 3.3.5
> > > >
> > > > Mukund and I have put together a release candidate (RC3) for Hadoop
> > > 3.3.5.
> > > >
> > > > What we would like is for anyone who can to verify the tarballs,
> > > especially
> > > > anyone who can try the arm64 binaries as we want to include them too.
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> > > >
> > > > The git tag is release-3.3.5-RC3, commit 706d88266ab
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> > > >
> > > > This is off branch-3.3 and is the first big release since 3.3.2.
> > > >
> > > > Key changes include
> > > >
> > > > * Big update of dependencies to try and keep those reports of
> > > >   transitive CVEs under control -both genuine and false positives.
> > > > * HDFS RBF enhancements
> > > > * Critical fix to ABFS input stream prefetching for correct reading.
> > > > * Vectored IO API for all FSDataInputStream implementations, with
> > > >   high-performance versions for file:// and s3a:// filesystems.
> > > >   file:// through java native io
> > > >   s3a:// parallel GET requests.
> > > > * This release includes Arm64 binaries. Please can anyone with
> > > >   compatible systems validate these.
> > > > * and compared to the previous RC, all the major changes are
> > > >   HDFS issues.
> > > >
> > > > Note, because the arm64 binaries are built separately on a different
> > > > platform and JVM, their jar files may not match those of the x86
> > > > release -and therefore the maven artifacts. I don't think this is
> > > > an issue (the ASF actually releases source tarballs, the binaries are
> > > > there for help only, though with the maven repo that's a bit
> blurred).
> > > >
> > > > The only way to be consistent would actually untar the x86.tar.gz,
> > > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > > for the vote. Even automating that would be risky.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > -Steve
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ashutosh Gupta]

Hi Steve

I will also do it by today/tomorrow.

Thanks,
Ashutosh

On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, 
wrote:

> Thank you for this!
>
> Can anyone else with time do a review too? i really want to get this one
> done, now the HDFS issues are all resolved.
>
> I do not want this release to fall by the wayside through lack of votes
> alone. In fact, I would be very unhappy
>
>
>
> On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:
>
> > +1 (non-binding)
> >
> > * Signature/Checksum: ok
> > * Rat check (1.8.0_341): ok
> >  - mvn clean apache-rat:check
> > * Built from source (1.8.0_341): ok
> >  - mvn clean install  -DskipTests
> > * Built tar from source (1.8.0_341): ok
> >  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >
> > Containerized deployments:
> > * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7
> > * Deployed and started JHS, RM, NM
> > * Hbase, hdfs CRUD looks good
> > * Sample RowCount MapReduce job looks good
> >
> > * S3A tests with scale profile looks good
> >
> >
> > On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
> > 
> > wrote:
> >
> > > Apache Hadoop 3.3.5
> > >
> > > Mukund and I have put together a release candidate (RC3) for Hadoop
> > 3.3.5.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> > >
> > > The git tag is release-3.3.5-RC3, commit 706d88266ab
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.3 and is the first big release since 3.3.2.
> > >
> > > Key changes include
> > >
> > > * Big update of dependencies to try and keep those reports of
> > >   transitive CVEs under control -both genuine and false positives.
> > > * HDFS RBF enhancements
> > > * Critical fix to ABFS input stream prefetching for correct reading.
> > > * Vectored IO API for all FSDataInputStream implementations, with
> > >   high-performance versions for file:// and s3a:// filesystems.
> > >   file:// through java native io
> > >   s3a:// parallel GET requests.
> > > * This release includes Arm64 binaries. Please can anyone with
> > >   compatible systems validate these.
> > > * and compared to the previous RC, all the major changes are
> > >   HDFS issues.
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

Thank you for this!

Can anyone else with time do a review too? i really want to get this one
done, now the HDFS issues are all resolved.

I do not want this release to fall by the wayside through lack of votes
alone. In fact, I would be very unhappy



On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:

> +1 (non-binding)
>
> * Signature/Checksum: ok
> * Rat check (1.8.0_341): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
>  - mvn clean install  -DskipTests
> * Built tar from source (1.8.0_341): ok
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> Containerized deployments:
> * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7
> * Deployed and started JHS, RM, NM
> * Hbase, hdfs CRUD looks good
> * Sample RowCount MapReduce job looks good
>
> * S3A tests with scale profile looks good
>
>
> On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
> 
> wrote:
>
> > Apache Hadoop 3.3.5
> >
> > Mukund and I have put together a release candidate (RC3) for Hadoop
> 3.3.5.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> >
> > The git tag is release-3.3.5-RC3, commit 706d88266ab
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > Key changes include
> >
> > * Big update of dependencies to try and keep those reports of
> >   transitive CVEs under control -both genuine and false positives.
> > * HDFS RBF enhancements
> > * Critical fix to ABFS input stream prefetching for correct reading.
> > * Vectored IO API for all FSDataInputStream implementations, with
> >   high-performance versions for file:// and s3a:// filesystems.
> >   file:// through java native io
> >   s3a:// parallel GET requests.
> > * This release includes Arm64 binaries. Please can anyone with
> >   compatible systems validate these.
> > * and compared to the previous RC, all the major changes are
> >   HDFS issues.
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Viraj Jasani]

+1 (non-binding)

* Signature/Checksum: ok
* Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Containerized deployments:
* Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7
* Deployed and started JHS, RM, NM
* Hbase, hdfs CRUD looks good
* Sample RowCount MapReduce job looks good

* S3A tests with scale profile looks good


On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran 
wrote:

> Apache Hadoop 3.3.5
>
> Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
>
> The git tag is release-3.3.5-RC3, commit 706d88266ab
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
> * and compared to the previous RC, all the major changes are
>   HDFS issues.
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

and my vote

My vote

+1 binding

I've been using the RCs for a while as my CLI entry point, and testing it
through other builds

for this RC
* Local builds of cloudstore
* fs-api-shim
* spark
* built and ran my cloud integration tests, which now include large CVS
file jobs which should show the Azure prefetch bug if it still existed.

downloaded the tar, expanded it, ran command line code with it, including
cloudstore against the stores. we need to get hadoop-azure and its
dependencies onto the path by default, to make abfs io easier.


I have the arm binaries building, and did a checknative to make sure all
was good

stevel@0da162643f99:~/hadoop/patchprocess/hadoop-3.3.5$ bin/hadoop
checknative
2023-03-17 13:00:27,107 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2023-03-17 13:00:27,112 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2023-03-17 13:00:27,121 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2023-03-17 13:00:27,156 INFO nativeio.NativeIO: The native code was built
without PMDK support.
Native library checking:
hadoop:  true
/home/stevel/hadoop/patchprocess/hadoop-3.3.5/lib/native/libhadoop.so.1.0.0
zlib:true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: true /lib/aarch64-linux-gnu/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
PMDK:false The native code was built without PMDK support.

---


On Wed, 15 Mar 2023 at 19:47, Steve Loughran  wrote:

>
> Apache Hadoop 3.3.5
>
> Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
>
> The git tag is release-3.3.5-RC3, commit 706d88266ab
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
> * and compared to the previous RC, all the major changes are
>   HDFS issues.
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

[VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/

The git tag is release-3.3.5-RC3, commit 706d88266ab

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positives.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.
* and compared to the previous RC, all the major changes are
  HDFS issues.

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

thanks.

now looking at a critical kerby CVE (
https://github.com/apache/hadoop/pull/5458) and revisited one for netty
from last week

i am never a fan of last-minute jar updates, but if we don't ship with them
we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18
months

On Mon, 6 Mar 2023 at 23:29, Erik Krogen  wrote:

> > OK. Could you have a go with a (locally built) patch release
>
> Just validated the same on the latest HEAD of branch-3.3.5, which includes
> the two HDFS Jiras I mentioned plus one additional one:
>
> * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
> TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
> 55643692+slfan1...@users.noreply.github.com>]
> * d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
> will throw NPE if path does not exist (#5400) [ZanderXu <
> zande...@apache.org
> >]
> * 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
> Fix NPE when check the block location of empty directory (#5099)
> [zhengchenyu ]
> * 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
> connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
> ste...@cloudera.com>]
>
> On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran  >
> wrote:
>
> >  i looked at that test and wondered if it it was just being brittle to
> > time. I'm not a fan of those -there's one in abfs which is particularly
> bad
> > for me- maybe we could see if the test can be cut as it is quite a slow
> one
> >
> > On Sat, 4 Mar 2023 at 18:28, Viraj Jasani  wrote:
> >
> > > A minor update on ITestS3AConcurrentOps#testParallelRename
> > >
> > > I was previously connected to a vpn due to which bandwidth was getting
> > > throttled earlier. Ran the test again today without vpn and had no
> issues
> > > (earlier only 40% of the overall putObject were able to get completed
> > > within timeout).
> > >
> > >
> > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> >  > > >
> > > wrote:
> > >
> > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
> > > >
> > > > > Thanks Steve. I see now that the branch cut was way back in October
> > so
> > > I
> > > > > definitely understand your frustration here!
> > > > >
> > > > > This made me realize that HDFS-16832
> > > > > , which
> resolves a
> > > > very
> > > > > similar issue as the aforementioned HDFS-16923, is also missing
> from
> > > the
> > > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > > before
> > > > > the initial 3.3.5 RC was made and I didn't notice the branch was
> cut.
> > > My
> > > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > > >
> > > >
> > > > thanks.
> > > >
> > > > >
> > > > > In the meantime, I tested for RC2 that a local cluster of NN +
> > standby
> > > +
> > > > > observer + QJM works as expected for some basic HDFS commands.
> > > > >
> > > >
> > > > OK. Could you have a go with a (locally built) patch release
> > > >
> > > > >
> > > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > > 
> > > > > wrote:
> > > > >
> > > > >> shipping broken hdfs isn't something we'd want to do, but if we
> can
> > be
> > > > >> confident that all other issues can be addressed in RC3 then I'd
> be
> > > > happy.
> > > > >>
> > > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena 
> > wrote:
> > > > >>
> > > > >> > I will highlight that I am completely fed up with doing this
> > > release
> > > > >> and
> > > > >> >> really want to get it out the way -for which I depend on
> support
> > > from
> > > > >> as
> > > > >> >> many other developers as possible.
> > > > >> >
> > > > >> >
> > > > >> > hmm, I can feel the pain. I tried to find if there is any config
> > or
> > > > any
> > > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > > couldn't
> > > > >> find
> > > > >> > any. If someone does a getListing with needLocation and the file
> > > > doesn't
> > > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> > isn't
> > > > just
> > > > >> > the exception, AFAIK Observer reads have some logic around
> > handling
> > > > FNF
> > > > >> > specifically, that it attempts Active NN or something like that
> in
> > > > such
> > > > >> > cases, So, that will be broken as well for this use case.
> > > > >> >
> > > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > > side,
> > > > >> and
> > > > >> > it has already been too much work on your side, so you can argue
> > > that
> > > > it
> > > > >> > might not be a very frequent use case or so. It's your call.
> > > > >> >
> > > > >> > Just sharing, no intentions of saying you should do that, But as
> > an
> > > RM
> > > > >> > "nobody" can force you for a new iteration of a RC, it is gonna
> be
> > > > your
> > > > >> > call and discretion. As far as I know a release can not be
> vetoed
> > by
> > > > 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Erik Krogen]

> OK. Could you have a go with a (locally built) patch release

Just validated the same on the latest HEAD of branch-3.3.5, which includes
the two HDFS Jiras I mentioned plus one additional one:

* 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
55643692+slfan1...@users.noreply.github.com>]
* d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
will throw NPE if path does not exist (#5400) [ZanderXu ]
* 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
Fix NPE when check the block location of empty directory (#5099)
[zhengchenyu ]
* 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
ste...@cloudera.com>]

On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran 
wrote:

>  i looked at that test and wondered if it it was just being brittle to
> time. I'm not a fan of those -there's one in abfs which is particularly bad
> for me- maybe we could see if the test can be cut as it is quite a slow one
>
> On Sat, 4 Mar 2023 at 18:28, Viraj Jasani  wrote:
>
> > A minor update on ITestS3AConcurrentOps#testParallelRename
> >
> > I was previously connected to a vpn due to which bandwidth was getting
> > throttled earlier. Ran the test again today without vpn and had no issues
> > (earlier only 40% of the overall putObject were able to get completed
> > within timeout).
> >
> >
> > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
>  > >
> > wrote:
> >
> > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
> > >
> > > > Thanks Steve. I see now that the branch cut was way back in October
> so
> > I
> > > > definitely understand your frustration here!
> > > >
> > > > This made me realize that HDFS-16832
> > > > , which resolves a
> > > very
> > > > similar issue as the aforementioned HDFS-16923, is also missing from
> > the
> > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > before
> > > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> > My
> > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > >
> > >
> > > thanks.
> > >
> > > >
> > > > In the meantime, I tested for RC2 that a local cluster of NN +
> standby
> > +
> > > > observer + QJM works as expected for some basic HDFS commands.
> > > >
> > >
> > > OK. Could you have a go with a (locally built) patch release
> > >
> > > >
> > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > 
> > > > wrote:
> > > >
> > > >> shipping broken hdfs isn't something we'd want to do, but if we can
> be
> > > >> confident that all other issues can be addressed in RC3 then I'd be
> > > happy.
> > > >>
> > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena 
> wrote:
> > > >>
> > > >> > I will highlight that I am completely fed up with doing this
> > release
> > > >> and
> > > >> >> really want to get it out the way -for which I depend on support
> > from
> > > >> as
> > > >> >> many other developers as possible.
> > > >> >
> > > >> >
> > > >> > hmm, I can feel the pain. I tried to find if there is any config
> or
> > > any
> > > >> > workaround which can dodge this HDFS issue, but unfortunately
> > couldn't
> > > >> find
> > > >> > any. If someone does a getListing with needLocation and the file
> > > doesn't
> > > >> > exist at Observer he is gonna get a NPE rather than a FNF, It
> isn't
> > > just
> > > >> > the exception, AFAIK Observer reads have some logic around
> handling
> > > FNF
> > > >> > specifically, that it attempts Active NN or something like that in
> > > such
> > > >> > cases, So, that will be broken as well for this use case.
> > > >> >
> > > >> > Now, there is no denying the fact there is an issue on the HDFS
> > side,
> > > >> and
> > > >> > it has already been too much work on your side, so you can argue
> > that
> > > it
> > > >> > might not be a very frequent use case or so. It's your call.
> > > >> >
> > > >> > Just sharing, no intentions of saying you should do that, But as
> an
> > RM
> > > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > > your
> > > >> > call and discretion. As far as I know a release can not be vetoed
> by
> > > >> > "anybody" as per the apache by laws.(
> > > >> > https://www.apache.org/legal/release-policy.html#release-approval
> ).
> > > >> Even
> > > >> > our bylaws say that product release requires a Lazy Majority not a
> > > >> > Consensus Approval.
> > > >> >
> > > >> > So, you have a way out. You guys are 2 already and 1 I will give
> > you a
> > > >> > pass, in case you are really in a state: ''Get me out of this
> mess"
> > > >> state,
> > > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > > >> couldn't
> > > >> > reach the end for any of the RC's
> > > >> >
> > > >> > -Ayush
> > > >> >
> > > >> > On Fri, 3 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

 i looked at that test and wondered if it it was just being brittle to
time. I'm not a fan of those -there's one in abfs which is particularly bad
for me- maybe we could see if the test can be cut as it is quite a slow one

On Sat, 4 Mar 2023 at 18:28, Viraj Jasani  wrote:

> A minor update on ITestS3AConcurrentOps#testParallelRename
>
> I was previously connected to a vpn due to which bandwidth was getting
> throttled earlier. Ran the test again today without vpn and had no issues
> (earlier only 40% of the overall putObject were able to get completed
> within timeout).
>
>
> On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran  >
> wrote:
>
> > On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
> >
> > > Thanks Steve. I see now that the branch cut was way back in October so
> I
> > > definitely understand your frustration here!
> > >
> > > This made me realize that HDFS-16832
> > > , which resolves a
> > very
> > > similar issue as the aforementioned HDFS-16923, is also missing from
> the
> > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> before
> > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> My
> > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > >
> >
> > thanks.
> >
> > >
> > > In the meantime, I tested for RC2 that a local cluster of NN + standby
> +
> > > observer + QJM works as expected for some basic HDFS commands.
> > >
> >
> > OK. Could you have a go with a (locally built) patch release
> >
> > >
> > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > 
> > > wrote:
> > >
> > >> shipping broken hdfs isn't something we'd want to do, but if we can be
> > >> confident that all other issues can be addressed in RC3 then I'd be
> > happy.
> > >>
> > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
> > >>
> > >> > I will highlight that I am completely fed up with doing this
> release
> > >> and
> > >> >> really want to get it out the way -for which I depend on support
> from
> > >> as
> > >> >> many other developers as possible.
> > >> >
> > >> >
> > >> > hmm, I can feel the pain. I tried to find if there is any config or
> > any
> > >> > workaround which can dodge this HDFS issue, but unfortunately
> couldn't
> > >> find
> > >> > any. If someone does a getListing with needLocation and the file
> > doesn't
> > >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> > just
> > >> > the exception, AFAIK Observer reads have some logic around handling
> > FNF
> > >> > specifically, that it attempts Active NN or something like that in
> > such
> > >> > cases, So, that will be broken as well for this use case.
> > >> >
> > >> > Now, there is no denying the fact there is an issue on the HDFS
> side,
> > >> and
> > >> > it has already been too much work on your side, so you can argue
> that
> > it
> > >> > might not be a very frequent use case or so. It's your call.
> > >> >
> > >> > Just sharing, no intentions of saying you should do that, But as an
> RM
> > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > your
> > >> > call and discretion. As far as I know a release can not be vetoed by
> > >> > "anybody" as per the apache by laws.(
> > >> > https://www.apache.org/legal/release-policy.html#release-approval).
> > >> Even
> > >> > our bylaws say that product release requires a Lazy Majority not a
> > >> > Consensus Approval.
> > >> >
> > >> > So, you have a way out. You guys are 2 already and 1 I will give
> you a
> > >> > pass, in case you are really in a state: ''Get me out of this mess"
> > >> state,
> > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > >> couldn't
> > >> > reach the end for any of the RC's
> > >> >
> > >> > -Ayush
> > >> >
> > >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani 
> wrote:
> > >> >
> > >> >> While this RC is not going to be final, I just wanted to share the
> > >> results
> > >> >> of the testing I have done so far with RC1 and RC2.
> > >> >>
> > >> >> * Signature: ok
> > >> >> * Checksum : ok
> > >> >> * Rat check (1.8.0_341): ok
> > >> >>  - mvn clean apache-rat:check
> > >> >> * Built from source (1.8.0_341): ok
> > >> >>  - mvn clean install  -DskipTests
> > >> >> * Built tar from source (1.8.0_341): ok
> > >> >>  - mvn clean package  -Pdist -DskipTests -Dtar
> > >> -Dmaven.javadoc.skip=true
> > >> >>
> > >> >> * Built images using the tarball, installed and started all of
> Hdfs,
> > >> JHS
> > >> >> and Yarn components
> > >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter
> Mapreduce
> > >> job
> > >> >> * Hdfs CRUD tests
> > >> >> * MapReduce wordcount job
> > >> >>
> > >> >> * Ran S3A tests with scale profile against us-west-2:
> > >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> > >> >>
> > >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> > >> This
> > >> >> 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Viraj Jasani]

A minor update on ITestS3AConcurrentOps#testParallelRename

I was previously connected to a vpn due to which bandwidth was getting
throttled earlier. Ran the test again today without vpn and had no issues
(earlier only 40% of the overall putObject were able to get completed
within timeout).


On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran 
wrote:

> On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
>
> > Thanks Steve. I see now that the branch cut was way back in October so I
> > definitely understand your frustration here!
> >
> > This made me realize that HDFS-16832
> > , which resolves a
> very
> > similar issue as the aforementioned HDFS-16923, is also missing from the
> > RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> > the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > branch-3.3.5, so they are ready if/when an RC3 is cut.
> >
>
> thanks.
>
> >
> > In the meantime, I tested for RC2 that a local cluster of NN + standby +
> > observer + QJM works as expected for some basic HDFS commands.
> >
>
> OK. Could you have a go with a (locally built) patch release
>
> >
> > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> 
> > wrote:
> >
> >> shipping broken hdfs isn't something we'd want to do, but if we can be
> >> confident that all other issues can be addressed in RC3 then I'd be
> happy.
> >>
> >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
> >>
> >> > I will highlight that I am completely fed up with doing this  release
> >> and
> >> >> really want to get it out the way -for which I depend on support from
> >> as
> >> >> many other developers as possible.
> >> >
> >> >
> >> > hmm, I can feel the pain. I tried to find if there is any config or
> any
> >> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> >> find
> >> > any. If someone does a getListing with needLocation and the file
> doesn't
> >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> just
> >> > the exception, AFAIK Observer reads have some logic around handling
> FNF
> >> > specifically, that it attempts Active NN or something like that in
> such
> >> > cases, So, that will be broken as well for this use case.
> >> >
> >> > Now, there is no denying the fact there is an issue on the HDFS side,
> >> and
> >> > it has already been too much work on your side, so you can argue that
> it
> >> > might not be a very frequent use case or so. It's your call.
> >> >
> >> > Just sharing, no intentions of saying you should do that, But as an RM
> >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> your
> >> > call and discretion. As far as I know a release can not be vetoed by
> >> > "anybody" as per the apache by laws.(
> >> > https://www.apache.org/legal/release-policy.html#release-approval).
> >> Even
> >> > our bylaws say that product release requires a Lazy Majority not a
> >> > Consensus Approval.
> >> >
> >> > So, you have a way out. You guys are 2 already and 1 I will give you a
> >> > pass, in case you are really in a state: ''Get me out of this mess"
> >> state,
> >> > my basic validations on x86 & Aarch64 both are passing as of now,
> >> couldn't
> >> > reach the end for any of the RC's
> >> >
> >> > -Ayush
> >> >
> >> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
> >> >
> >> >> While this RC is not going to be final, I just wanted to share the
> >> results
> >> >> of the testing I have done so far with RC1 and RC2.
> >> >>
> >> >> * Signature: ok
> >> >> * Checksum : ok
> >> >> * Rat check (1.8.0_341): ok
> >> >>  - mvn clean apache-rat:check
> >> >> * Built from source (1.8.0_341): ok
> >> >>  - mvn clean install  -DskipTests
> >> >> * Built tar from source (1.8.0_341): ok
> >> >>  - mvn clean package  -Pdist -DskipTests -Dtar
> >> -Dmaven.javadoc.skip=true
> >> >>
> >> >> * Built images using the tarball, installed and started all of Hdfs,
> >> JHS
> >> >> and Yarn components
> >> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> >> job
> >> >> * Hdfs CRUD tests
> >> >> * MapReduce wordcount job
> >> >>
> >> >> * Ran S3A tests with scale profile against us-west-2:
> >> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >> >>
> >> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
> >> This
> >> >> is
> >> >> consistently failing, looks like a recent regression.
> >> >> I was also able to repro on trunk, will create Jira.
> >> >>
> >> >>
> >> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> >> 
> >> >> wrote:
> >> >>
> >> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> >> 3.3.5.
> >> >> >
> >> >> > We need anyone who can to verify the source and binary artifacts,
> >> >> > including those JARs staged on maven, the site documentation and
> the
> >> >> arm64
> >> >> > tar file.
> >> >> >
> >> >> > The 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:

> Thanks Steve. I see now that the branch cut was way back in October so I
> definitely understand your frustration here!
>
> This made me realize that HDFS-16832
> , which resolves a very
> similar issue as the aforementioned HDFS-16923, is also missing from the
> RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> branch-3.3.5, so they are ready if/when an RC3 is cut.
>

thanks.

>
> In the meantime, I tested for RC2 that a local cluster of NN + standby +
> observer + QJM works as expected for some basic HDFS commands.
>

OK. Could you have a go with a (locally built) patch release

>
> On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran 
> wrote:
>
>> shipping broken hdfs isn't something we'd want to do, but if we can be
>> confident that all other issues can be addressed in RC3 then I'd be happy.
>>
>> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
>>
>> > I will highlight that I am completely fed up with doing this  release
>> and
>> >> really want to get it out the way -for which I depend on support from
>> as
>> >> many other developers as possible.
>> >
>> >
>> > hmm, I can feel the pain. I tried to find if there is any config or any
>> > workaround which can dodge this HDFS issue, but unfortunately couldn't
>> find
>> > any. If someone does a getListing with needLocation and the file doesn't
>> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
>> > the exception, AFAIK Observer reads have some logic around handling FNF
>> > specifically, that it attempts Active NN or something like that in such
>> > cases, So, that will be broken as well for this use case.
>> >
>> > Now, there is no denying the fact there is an issue on the HDFS side,
>> and
>> > it has already been too much work on your side, so you can argue that it
>> > might not be a very frequent use case or so. It's your call.
>> >
>> > Just sharing, no intentions of saying you should do that, But as an RM
>> > "nobody" can force you for a new iteration of a RC, it is gonna be your
>> > call and discretion. As far as I know a release can not be vetoed by
>> > "anybody" as per the apache by laws.(
>> > https://www.apache.org/legal/release-policy.html#release-approval).
>> Even
>> > our bylaws say that product release requires a Lazy Majority not a
>> > Consensus Approval.
>> >
>> > So, you have a way out. You guys are 2 already and 1 I will give you a
>> > pass, in case you are really in a state: ''Get me out of this mess"
>> state,
>> > my basic validations on x86 & Aarch64 both are passing as of now,
>> couldn't
>> > reach the end for any of the RC's
>> >
>> > -Ayush
>> >
>> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
>> >
>> >> While this RC is not going to be final, I just wanted to share the
>> results
>> >> of the testing I have done so far with RC1 and RC2.
>> >>
>> >> * Signature: ok
>> >> * Checksum : ok
>> >> * Rat check (1.8.0_341): ok
>> >>  - mvn clean apache-rat:check
>> >> * Built from source (1.8.0_341): ok
>> >>  - mvn clean install  -DskipTests
>> >> * Built tar from source (1.8.0_341): ok
>> >>  - mvn clean package  -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>> >>
>> >> * Built images using the tarball, installed and started all of Hdfs,
>> JHS
>> >> and Yarn components
>> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
>> job
>> >> * Hdfs CRUD tests
>> >> * MapReduce wordcount job
>> >>
>> >> * Ran S3A tests with scale profile against us-west-2:
>> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>> >>
>> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
>> This
>> >> is
>> >> consistently failing, looks like a recent regression.
>> >> I was also able to repro on trunk, will create Jira.
>> >>
>> >>
>> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> >> 
>> >> wrote:
>> >>
>> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> >> 3.3.5.
>> >> >
>> >> > We need anyone who can to verify the source and binary artifacts,
>> >> > including those JARs staged on maven, the site documentation and the
>> >> arm64
>> >> > tar file.
>> >> >
>> >> > The RC is available at:
>> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >> >
>> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >> >
>> >> > The maven artifacts are staged at
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >> >
>> >> > You can find my public key at:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > Change log
>> >> >
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >> >
>> >> > Release notes
>> >> >
>> >> >
>> >>
>> 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Erik Krogen]

Thanks Steve. I see now that the branch cut was way back in October so I
definitely understand your frustration here!

This made me realize that HDFS-16832
, which resolves a very
similar issue as the aforementioned HDFS-16923, is also missing from the
RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
branch-3.3.5, so they are ready if/when an RC3 is cut.

In the meantime, I tested for RC2 that a local cluster of NN + standby +
observer + QJM works as expected for some basic HDFS commands.

On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran 
wrote:

> shipping broken hdfs isn't something we'd want to do, but if we can be
> confident that all other issues can be addressed in RC3 then I'd be happy.
>
> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
>
> > I will highlight that I am completely fed up with doing this  release and
> >> really want to get it out the way -for which I depend on support from as
> >> many other developers as possible.
> >
> >
> > hmm, I can feel the pain. I tried to find if there is any config or any
> > workaround which can dodge this HDFS issue, but unfortunately couldn't
> find
> > any. If someone does a getListing with needLocation and the file doesn't
> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> > the exception, AFAIK Observer reads have some logic around handling FNF
> > specifically, that it attempts Active NN or something like that in such
> > cases, So, that will be broken as well for this use case.
> >
> > Now, there is no denying the fact there is an issue on the HDFS side, and
> > it has already been too much work on your side, so you can argue that it
> > might not be a very frequent use case or so. It's your call.
> >
> > Just sharing, no intentions of saying you should do that, But as an RM
> > "nobody" can force you for a new iteration of a RC, it is gonna be your
> > call and discretion. As far as I know a release can not be vetoed by
> > "anybody" as per the apache by laws.(
> > https://www.apache.org/legal/release-policy.html#release-approval). Even
> > our bylaws say that product release requires a Lazy Majority not a
> > Consensus Approval.
> >
> > So, you have a way out. You guys are 2 already and 1 I will give you a
> > pass, in case you are really in a state: ''Get me out of this mess"
> state,
> > my basic validations on x86 & Aarch64 both are passing as of now,
> couldn't
> > reach the end for any of the RC's
> >
> > -Ayush
> >
> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
> >
> >> While this RC is not going to be final, I just wanted to share the
> results
> >> of the testing I have done so far with RC1 and RC2.
> >>
> >> * Signature: ok
> >> * Checksum : ok
> >> * Rat check (1.8.0_341): ok
> >>  - mvn clean apache-rat:check
> >> * Built from source (1.8.0_341): ok
> >>  - mvn clean install  -DskipTests
> >> * Built tar from source (1.8.0_341): ok
> >>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >>
> >> * Built images using the tarball, installed and started all of Hdfs, JHS
> >> and Yarn components
> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
> job
> >> * Hdfs CRUD tests
> >> * MapReduce wordcount job
> >>
> >> * Ran S3A tests with scale profile against us-west-2:
> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
> >>
> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
> >> is
> >> consistently failing, looks like a recent regression.
> >> I was also able to repro on trunk, will create Jira.
> >>
> >>
> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
> >> 
> >> wrote:
> >>
> >> > Mukund and I have put together a release candidate (RC2) for Hadoop
> >> 3.3.5.
> >> >
> >> > We need anyone who can to verify the source and binary artifacts,
> >> > including those JARs staged on maven, the site documentation and the
> >> arm64
> >> > tar file.
> >> >
> >> > The RC is available at:
> >> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >> >
> >> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >> >
> >> > The maven artifacts are staged at
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >> >
> >> > You can find my public key at:
> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> >
> >> > Change log
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >> >
> >> > Release notes
> >> >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >> >
> >> > This is off branch-3.3 and is the first big release since 3.3.2.
> >> >
> >> > As to what changed since the RC1 attempt last week
> >> >
> >> >
> >> >1. Version fixup in JIRA (credit due to 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

shipping broken hdfs isn't something we'd want to do, but if we can be
confident that all other issues can be addressed in RC3 then I'd be happy.

On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:

> I will highlight that I am completely fed up with doing this  release and
>> really want to get it out the way -for which I depend on support from as
>> many other developers as possible.
>
>
> hmm, I can feel the pain. I tried to find if there is any config or any
> workaround which can dodge this HDFS issue, but unfortunately couldn't find
> any. If someone does a getListing with needLocation and the file doesn't
> exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> the exception, AFAIK Observer reads have some logic around handling FNF
> specifically, that it attempts Active NN or something like that in such
> cases, So, that will be broken as well for this use case.
>
> Now, there is no denying the fact there is an issue on the HDFS side, and
> it has already been too much work on your side, so you can argue that it
> might not be a very frequent use case or so. It's your call.
>
> Just sharing, no intentions of saying you should do that, But as an RM
> "nobody" can force you for a new iteration of a RC, it is gonna be your
> call and discretion. As far as I know a release can not be vetoed by
> "anybody" as per the apache by laws.(
> https://www.apache.org/legal/release-policy.html#release-approval). Even
> our bylaws say that product release requires a Lazy Majority not a
> Consensus Approval.
>
> So, you have a way out. You guys are 2 already and 1 I will give you a
> pass, in case you are really in a state: ''Get me out of this mess" state,
> my basic validations on x86 & Aarch64 both are passing as of now, couldn't
> reach the end for any of the RC's
>
> -Ayush
>
> On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
>
>> While this RC is not going to be final, I just wanted to share the results
>> of the testing I have done so far with RC1 and RC2.
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_341): ok
>>  - mvn clean apache-rat:check
>> * Built from source (1.8.0_341): ok
>>  - mvn clean install  -DskipTests
>> * Built tar from source (1.8.0_341): ok
>>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Built images using the tarball, installed and started all of Hdfs, JHS
>> and Yarn components
>> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
>> * Hdfs CRUD tests
>> * MapReduce wordcount job
>>
>> * Ran S3A tests with scale profile against us-west-2:
>> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>>
>> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
>> is
>> consistently failing, looks like a recent regression.
>> I was also able to repro on trunk, will create Jira.
>>
>>
>> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> 
>> wrote:
>>
>> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> 3.3.5.
>> >
>> > We need anyone who can to verify the source and binary artifacts,
>> > including those JARs staged on maven, the site documentation and the
>> arm64
>> > tar file.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >
>> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >
>> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >
>> > As to what changed since the RC1 attempt last week
>> >
>> >
>> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> build
>> >issues in maven 3.9.0)
>> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> (#5429)
>> >
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Steve and Mukund
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Ayush Saxena]

>
> I will highlight that I am completely fed up with doing this  release and
> really want to get it out the way -for which I depend on support from as
> many other developers as possible.


hmm, I can feel the pain. I tried to find if there is any config or any
workaround which can dodge this HDFS issue, but unfortunately couldn't find
any. If someone does a getListing with needLocation and the file doesn't
exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
the exception, AFAIK Observer reads have some logic around handling FNF
specifically, that it attempts Active NN or something like that in such
cases, So, that will be broken as well for this use case.

Now, there is no denying the fact there is an issue on the HDFS side, and
it has already been too much work on your side, so you can argue that it
might not be a very frequent use case or so. It's your call.

Just sharing, no intentions of saying you should do that, But as an RM
"nobody" can force you for a new iteration of a RC, it is gonna be your
call and discretion. As far as I know a release can not be vetoed by
"anybody" as per the apache by laws.(
https://www.apache.org/legal/release-policy.html#release-approval). Even
our bylaws say that product release requires a Lazy Majority not a
Consensus Approval.

So, you have a way out. You guys are 2 already and 1 I will give you a
pass, in case you are really in a state: ''Get me out of this mess" state,
my basic validations on x86 & Aarch64 both are passing as of now, couldn't
reach the end for any of the RC's

-Ayush

On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:

> While this RC is not going to be final, I just wanted to share the results
> of the testing I have done so far with RC1 and RC2.
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
>  - mvn clean install  -DskipTests
> * Built tar from source (1.8.0_341): ok
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> * Built images using the tarball, installed and started all of Hdfs, JHS
> and Yarn components
> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
> * Hdfs CRUD tests
> * MapReduce wordcount job
>
> * Ran S3A tests with scale profile against us-west-2:
> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>
> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
> consistently failing, looks like a recent regression.
> I was also able to repro on trunk, will create Jira.
>
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran  >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> >issues in maven 3.9.0)
> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Viraj Jasani]

While this RC is not going to be final, I just wanted to share the results
of the testing I have done so far with RC1 and RC2.

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

* Built images using the tarball, installed and started all of Hdfs, JHS
and Yarn components
* Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
* Hdfs CRUD tests
* MapReduce wordcount job

* Ran S3A tests with scale profile against us-west-2:
mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale

ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This is
consistently failing, looks like a recent regression.
I was also able to repro on trunk, will create Jira.


On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran 
wrote:

> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
>1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
>issues in maven 3.9.0)
>4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

well, lets see what others say.

we don't want to ship stuff with serious regression to hdfs.

I will highlight that I am completely fed up with doing this  release and
really want to get it out the way -for which I depend on support from as
many other developers as possible.

Erik, right now what you can help by doing is test all the rest of the
release knowing that this issue exists and seeing if you can identify
anything else. That way this update will be the sole blocker and we can get
through that next RC with nothing else surfacing.

I had noticed that the arm64 release somehow missed out the native binaries
and was going to investigate that but didn't consider that a blocker… I was
just going to cut that artefact and, post Darcy, create a new arm64 release
using all the jars of the x86 build but replacing the x86 native libs with
the arm versions. This helps ensure that the JAR files in the wild all
match, which strikes me as a good thing.

Can I also encourage people in the HFS team to put their hand up and
volunteer for leading the next release, with a goal of getting something
out later this year.



On Thu, 2 Mar 2023 at 00:27, Erik Krogen  wrote:

> Hi folks, apologies for being late to the release conversation, but I think
> we need to get HDFS-16923
>  into
> 3.3.5. HDFS-16732 ,
> which
> also went into 3.3.5, introduced an issue whereby Observer NameNodes will
> throw NPE upon any getListing call on a directory that doesn't exist. It
> will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
> for this and it has been merged to trunk/branch-3.3 as of a few minutes
> ago. I'd like to see about merging to branch-3.3.5 as well.
>
> Thanks for the consideration and sorry for not bringing this up in RC1 or
> earlier.
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran  >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> >issues in maven 3.9.0)
> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Erik Krogen]

Hi folks, apologies for being late to the release conversation, but I think
we need to get HDFS-16923
 into
3.3.5. HDFS-16732 , which
also went into 3.3.5, introduced an issue whereby Observer NameNodes will
throw NPE upon any getListing call on a directory that doesn't exist. It
will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
for this and it has been merged to trunk/branch-3.3 as of a few minutes
ago. I'd like to see about merging to branch-3.3.5 as well.

Thanks for the consideration and sorry for not bringing this up in RC1 or
earlier.

On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran 
wrote:

> Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.
>
> We need anyone who can to verify the source and binary artifacts,
> including those JARs staged on maven, the site documentation and the arm64
> tar file.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>
> The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> As to what changed since the RC1 attempt last week
>
>
>1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
>issues in maven 3.9.0)
>4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>

[VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.

We need anyone who can to verify the source and binary artifacts,
including those JARs staged on maven, the site documentation and the arm64
tar file.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/

The git tag is release-3.3.5-RC2, commit 72f8c2a4888

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

As to what changed since the RC1 attempt last week


   1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
   2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
   3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
   issues in maven 3.9.0)
   4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)


Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

Steve and Mukund

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

>
>  And i
> think we need to change the PR template to mention transitive updates in
> the license bit too


Not sure if that is gonna help, People might ignore that or check that in
overconfidence. No harm though..

BTW Ozone has some cool stuff to handle this, it was added here:
https://github.com/apache/ozone/pull/2199

It checks for each PR, if the changes bring any new transitive dependency
or not and if it does, it flags that and then licence and all can be
managed. Worth exploring

-Ayush

On Sat, 25 Feb 2023 at 01:09, Steve Loughran 
wrote:

>  need this pr in too, https://github.com/apache/hadoop/pull/5429
>
>1. cuts back on some transitive dependencies from hadoop-aliyun
>2. fixes LICENSE-bin to be correct
>
> #2 is the blocker...and it looks like 3.2.x will also need fixup as well as
> the later ones -hadoop binaries have shipped without that file being up to
> date, but at least all the transitive stuff is correctly licensed. And i
> think we need to change the PR template to mention transitive updates in
> the license bit too
>
> if this goes in, I will do the rebuild on monday UK time
>
> On Thu, 23 Feb 2023 at 11:18, Steve Loughran  wrote:
>
> >
> > And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
> > maven releases (3.9.0)
> >
> > on a new local build with maven updated on homebrew (which i needed for
> > spark). so a code change too. That issue doesn't surface on our
> > release dockers, but will hit other people. especially over time. Going
> to
> > revert HADOOP-18590. Publish SBOM artifacts (#5281)
> >
> >
> >
> > On Thu, 23 Feb 2023 at 10:29, Steve Loughran 
> wrote:
> >
> >> ok, let me cancel, update those jiras and kick off again. that will save
> >> anyone else having to do their homework
> >>
> >> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
> >> wrote:
> >>
> >>> I'm now -1 as I found the wrong information on the top page (index.md).
> >>>
> >>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in
> HDFS
> >>> Router Based Federation.
> >>>
> >>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5
> before,
> >>> though it is actually not in branch-3.3. I corrected the fix version
> and
> >>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
> >>> Unfortunately, it won't be fixed soon. I should have let you know at
> that
> >>> time, sorry.  Supporting Observer NameNode in RBF is a prominent
> feature.
> >>> So I think we have to delete the description from the top page not to
> >>> confuse Hadoop users.
> >>>
> >>> - Takanobu
> >>>
> >>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
> >>>
> >>> > Thanks for driving the release, Steve and Mukund.
> >>> >
> >>> > I found that there were some jiras with wrong fix versions.
> >>> >
> >>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
> >>> > - HDFS-16845
> >>> > - HADOOP-18345
> >>> >
> >>> > The fix versions didn't include 3.3.5, but actually, it is in
> 3.3.5-RC1
> >>> > (and it is not in release-3.3.4) :
> >>> > - HADOOP-17276
> >>> > - HDFS-13293
> >>> > - HDFS-15630
> >>> > - HDFS-16266
> >>> > - HADOOP-18003
> >>> > - HDFS-16310
> >>> > - HADOOP-18014
> >>> >
> >>> > I corrected all the wrong fix versions just now. I'm not sure we
> should
> >>> > revote it since it only affects the changelog.
> >>> >
> >>> > - Takanobu
> >>> >
> >>> > 2023年2月21日(火) 22:43 Steve Loughran :
> >>> >
> >>> >> Apache Hadoop 3.3.5
> >>> >>
> >>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
> >>> 3.3.5.
> >>> >>
> >>> >> What we would like is for anyone who can to verify the tarballs,
> >>> >> especially
> >>> >> anyone who can try the arm64 binaries as we want to include them
> too.
> >>> >>
> >>> >> The RC is available at:
> >>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
> >>> >>
> >>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
> >>> >>
> >>> >> The maven artifacts are staged at
> >>> >>
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
> >>> >>
> >>> >> You can find my public key at:
> >>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>> >>
> >>> >> Change log
> >>> >>
> >>> >>
> >>>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
> >>> >>
> >>> >> Release notes
> >>> >>
> >>> >>
> >>>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
> >>> >>
> >>> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>> >>
> >>> >> Key changes include
> >>> >>
> >>> >> * Big update of dependencies to try and keep those reports of
> >>> >>   transitive CVEs under control -both genuine and false positives.
> >>> >> * HDFS RBF enhancements
> >>> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >>> >> * Vectored IO API for all FSDataInputStream implementations, with
> >>> >>   high-performance versions for file:// and s3a:// filesystems.
> >>> >>   

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

 need this pr in too, https://github.com/apache/hadoop/pull/5429

   1. cuts back on some transitive dependencies from hadoop-aliyun
   2. fixes LICENSE-bin to be correct

#2 is the blocker...and it looks like 3.2.x will also need fixup as well as
the later ones -hadoop binaries have shipped without that file being up to
date, but at least all the transitive stuff is correctly licensed. And i
think we need to change the PR template to mention transitive updates in
the license bit too

if this goes in, I will do the rebuild on monday UK time

On Thu, 23 Feb 2023 at 11:18, Steve Loughran  wrote:

>
> And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
> maven releases (3.9.0)
>
> on a new local build with maven updated on homebrew (which i needed for
> spark). so a code change too. That issue doesn't surface on our
> release dockers, but will hit other people. especially over time. Going to
> revert HADOOP-18590. Publish SBOM artifacts (#5281)
>
>
>
> On Thu, 23 Feb 2023 at 10:29, Steve Loughran  wrote:
>
>> ok, let me cancel, update those jiras and kick off again. that will save
>> anyone else having to do their homework
>>
>> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
>> wrote:
>>
>>> I'm now -1 as I found the wrong information on the top page (index.md).
>>>
>>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
>>> Router Based Federation.
>>>
>>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
>>> though it is actually not in branch-3.3. I corrected the fix version and
>>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
>>> Unfortunately, it won't be fixed soon. I should have let you know at that
>>> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
>>> So I think we have to delete the description from the top page not to
>>> confuse Hadoop users.
>>>
>>> - Takanobu
>>>
>>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>>>
>>> > Thanks for driving the release, Steve and Mukund.
>>> >
>>> > I found that there were some jiras with wrong fix versions.
>>> >
>>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
>>> > - HDFS-16845
>>> > - HADOOP-18345
>>> >
>>> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
>>> > (and it is not in release-3.3.4) :
>>> > - HADOOP-17276
>>> > - HDFS-13293
>>> > - HDFS-15630
>>> > - HDFS-16266
>>> > - HADOOP-18003
>>> > - HDFS-16310
>>> > - HADOOP-18014
>>> >
>>> > I corrected all the wrong fix versions just now. I'm not sure we should
>>> > revote it since it only affects the changelog.
>>> >
>>> > - Takanobu
>>> >
>>> > 2023年2月21日(火) 22:43 Steve Loughran :
>>> >
>>> >> Apache Hadoop 3.3.5
>>> >>
>>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
>>> 3.3.5.
>>> >>
>>> >> What we would like is for anyone who can to verify the tarballs,
>>> >> especially
>>> >> anyone who can try the arm64 binaries as we want to include them too.
>>> >>
>>> >> The RC is available at:
>>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>>> >>
>>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
>>> >>
>>> >> The maven artifacts are staged at
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>>> >>
>>> >> You can find my public key at:
>>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >>
>>> >> Change log
>>> >>
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>>> >>
>>> >> Release notes
>>> >>
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>>> >>
>>> >> This is off branch-3.3 and is the first big release since 3.3.2.
>>> >>
>>> >> Key changes include
>>> >>
>>> >> * Big update of dependencies to try and keep those reports of
>>> >>   transitive CVEs under control -both genuine and false positives.
>>> >> * HDFS RBF enhancements
>>> >> * Critical fix to ABFS input stream prefetching for correct reading.
>>> >> * Vectored IO API for all FSDataInputStream implementations, with
>>> >>   high-performance versions for file:// and s3a:// filesystems.
>>> >>   file:// through java native io
>>> >>   s3a:// parallel GET requests.
>>> >> * This release includes Arm64 binaries. Please can anyone with
>>> >>   compatible systems validate these.
>>> >>
>>> >> Note, because the arm64 binaries are built separately on a different
>>> >> platform and JVM, their jar files may not match those of the x86
>>> >> release -and therefore the maven artifacts. I don't think this is
>>> >> an issue (the ASF actually releases source tarballs, the binaries are
>>> >> there for help only, though with the maven repo that's a bit blurred).
>>> >>
>>> >> The only way to be consistent would actually untar the x86.tar.gz,
>>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>>> >> for the vote. Even automating that would be risky.
>>> >>
>>> >> Please try the release 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
maven releases (3.9.0)

on a new local build with maven updated on homebrew (which i needed for
spark). so a code change too. That issue doesn't surface on our
release dockers, but will hit other people. especially over time. Going to
revert HADOOP-18590. Publish SBOM artifacts (#5281)



On Thu, 23 Feb 2023 at 10:29, Steve Loughran  wrote:

> ok, let me cancel, update those jiras and kick off again. that will save
> anyone else having to do their homework
>
> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
> wrote:
>
>> I'm now -1 as I found the wrong information on the top page (index.md).
>>
>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
>> Router Based Federation.
>>
>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
>> though it is actually not in branch-3.3. I corrected the fix version and
>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
>> Unfortunately, it won't be fixed soon. I should have let you know at that
>> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
>> So I think we have to delete the description from the top page not to
>> confuse Hadoop users.
>>
>> - Takanobu
>>
>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>>
>> > Thanks for driving the release, Steve and Mukund.
>> >
>> > I found that there were some jiras with wrong fix versions.
>> >
>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
>> > - HDFS-16845
>> > - HADOOP-18345
>> >
>> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
>> > (and it is not in release-3.3.4) :
>> > - HADOOP-17276
>> > - HDFS-13293
>> > - HDFS-15630
>> > - HDFS-16266
>> > - HADOOP-18003
>> > - HDFS-16310
>> > - HADOOP-18014
>> >
>> > I corrected all the wrong fix versions just now. I'm not sure we should
>> > revote it since it only affects the changelog.
>> >
>> > - Takanobu
>> >
>> > 2023年2月21日(火) 22:43 Steve Loughran :
>> >
>> >> Apache Hadoop 3.3.5
>> >>
>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
>> 3.3.5.
>> >>
>> >> What we would like is for anyone who can to verify the tarballs,
>> >> especially
>> >> anyone who can try the arm64 binaries as we want to include them too.
>> >>
>> >> The RC is available at:
>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>> >>
>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
>> >>
>> >> The maven artifacts are staged at
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>> >>
>> >> You can find my public key at:
>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >>
>> >> Change log
>> >>
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>> >>
>> >> Release notes
>> >>
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>> >>
>> >> This is off branch-3.3 and is the first big release since 3.3.2.
>> >>
>> >> Key changes include
>> >>
>> >> * Big update of dependencies to try and keep those reports of
>> >>   transitive CVEs under control -both genuine and false positives.
>> >> * HDFS RBF enhancements
>> >> * Critical fix to ABFS input stream prefetching for correct reading.
>> >> * Vectored IO API for all FSDataInputStream implementations, with
>> >>   high-performance versions for file:// and s3a:// filesystems.
>> >>   file:// through java native io
>> >>   s3a:// parallel GET requests.
>> >> * This release includes Arm64 binaries. Please can anyone with
>> >>   compatible systems validate these.
>> >>
>> >> Note, because the arm64 binaries are built separately on a different
>> >> platform and JVM, their jar files may not match those of the x86
>> >> release -and therefore the maven artifacts. I don't think this is
>> >> an issue (the ASF actually releases source tarballs, the binaries are
>> >> there for help only, though with the maven repo that's a bit blurred).
>> >>
>> >> The only way to be consistent would actually untar the x86.tar.gz,
>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>> >> for the vote. Even automating that would be risky.
>> >>
>> >> Please try the release and vote. The vote will run for 5 days.
>> >>
>> >> Steve and Mukund
>> >>
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

ok, let me cancel, update those jiras and kick off again. that will save
anyone else having to do their homework

On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma  wrote:

> I'm now -1 as I found the wrong information on the top page (index.md).
>
> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
> Router Based Federation.
>
> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
> though it is actually not in branch-3.3. I corrected the fix version and
> created HDFS-16889 to backport them to branch-3.3 about a month ago.
> Unfortunately, it won't be fixed soon. I should have let you know at that
> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
> So I think we have to delete the description from the top page not to
> confuse Hadoop users.
>
> - Takanobu
>
> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>
> > Thanks for driving the release, Steve and Mukund.
> >
> > I found that there were some jiras with wrong fix versions.
> >
> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
> > - HDFS-16845
> > - HADOOP-18345
> >
> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
> > (and it is not in release-3.3.4) :
> > - HADOOP-17276
> > - HDFS-13293
> > - HDFS-15630
> > - HDFS-16266
> > - HADOOP-18003
> > - HDFS-16310
> > - HADOOP-18014
> >
> > I corrected all the wrong fix versions just now. I'm not sure we should
> > revote it since it only affects the changelog.
> >
> > - Takanobu
> >
> > 2023年2月21日(火) 22:43 Steve Loughran :
> >
> >> Apache Hadoop 3.3.5
> >>
> >> Mukund and I have put together a release candidate (RC1) for Hadoop
> 3.3.5.
> >>
> >> What we would like is for anyone who can to verify the tarballs,
> >> especially
> >> anyone who can try the arm64 binaries as we want to include them too.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
> >>
> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
> >>
> >> The maven artifacts are staged at
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
> >>
> >> Release notes
> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >>   transitive CVEs under control -both genuine and false positives.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >>   high-performance versions for file:// and s3a:// filesystems.
> >>   file:// through java native io
> >>   s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >>   compatible systems validate these.
> >>
> >> Note, because the arm64 binaries are built separately on a different
> >> platform and JVM, their jar files may not match those of the x86
> >> release -and therefore the maven artifacts. I don't think this is
> >> an issue (the ASF actually releases source tarballs, the binaries are
> >> there for help only, though with the maven repo that's a bit blurred).
> >>
> >> The only way to be consistent would actually untar the x86.tar.gz,
> >> overwrite its binaries with the arm stuff, retar, sign and push out
> >> for the vote. Even automating that would be risky.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> Steve and Mukund
> >>
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Takanobu Asanuma]

I'm now -1 as I found the wrong information on the top page (index.md).

> 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
Router Based Federation.

The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
though it is actually not in branch-3.3. I corrected the fix version and
created HDFS-16889 to backport them to branch-3.3 about a month ago.
Unfortunately, it won't be fixed soon. I should have let you know at that
time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
So I think we have to delete the description from the top page not to
confuse Hadoop users.

- Takanobu

2023年2月23日(木) 17:17 Takanobu Asanuma :

> Thanks for driving the release, Steve and Mukund.
>
> I found that there were some jiras with wrong fix versions.
>
> The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
> - HDFS-16845
> - HADOOP-18345
>
> The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
> (and it is not in release-3.3.4) :
> - HADOOP-17276
> - HDFS-13293
> - HDFS-15630
> - HDFS-16266
> - HADOOP-18003
> - HDFS-16310
> - HADOOP-18014
>
> I corrected all the wrong fix versions just now. I'm not sure we should
> revote it since it only affects the changelog.
>
> - Takanobu
>
> 2023年2月21日(火) 22:43 Steve Loughran :
>
>> Apache Hadoop 3.3.5
>>
>> Mukund and I have put together a release candidate (RC1) for Hadoop 3.3.5.
>>
>> What we would like is for anyone who can to verify the tarballs,
>> especially
>> anyone who can try the arm64 binaries as we want to include them too.
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>>
>> The git tag is release-3.3.5-RC1, commit 274f91a3259
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Change log
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>>
>> Release notes
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>>
>> This is off branch-3.3 and is the first big release since 3.3.2.
>>
>> Key changes include
>>
>> * Big update of dependencies to try and keep those reports of
>>   transitive CVEs under control -both genuine and false positives.
>> * HDFS RBF enhancements
>> * Critical fix to ABFS input stream prefetching for correct reading.
>> * Vectored IO API for all FSDataInputStream implementations, with
>>   high-performance versions for file:// and s3a:// filesystems.
>>   file:// through java native io
>>   s3a:// parallel GET requests.
>> * This release includes Arm64 binaries. Please can anyone with
>>   compatible systems validate these.
>>
>> Note, because the arm64 binaries are built separately on a different
>> platform and JVM, their jar files may not match those of the x86
>> release -and therefore the maven artifacts. I don't think this is
>> an issue (the ASF actually releases source tarballs, the binaries are
>> there for help only, though with the maven repo that's a bit blurred).
>>
>> The only way to be consistent would actually untar the x86.tar.gz,
>> overwrite its binaries with the arm stuff, retar, sign and push out
>> for the vote. Even automating that would be risky.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> Steve and Mukund
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Takanobu Asanuma]

Thanks for driving the release, Steve and Mukund.

I found that there were some jiras with wrong fix versions.

The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
- HDFS-16845
- HADOOP-18345

The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
(and it is not in release-3.3.4) :
- HADOOP-17276
- HDFS-13293
- HDFS-15630
- HDFS-16266
- HADOOP-18003
- HDFS-16310
- HADOOP-18014

I corrected all the wrong fix versions just now. I'm not sure we should
revote it since it only affects the changelog.

- Takanobu

2023年2月21日(火) 22:43 Steve Loughran :

> Apache Hadoop 3.3.5
>
> Mukund and I have put together a release candidate (RC1) for Hadoop 3.3.5.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>
> The git tag is release-3.3.5-RC1, commit 274f91a3259
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Steve and Mukund
>

[VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC1) for Hadoop 3.3.5.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/

The git tag is release-3.3.5-RC1, commit 274f91a3259

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1368/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positives.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

Steve and Mukund

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

thanks

pulling in a few of the recent changes which seem needed/important, now
wondering about the javadocs.

i will add a new probe for this in our automated release ant bulld so we
can't cut a release without that
https://github.com/steveloughran/validate-hadoop-client-artifacts

On Mon, 2 Jan 2023 at 15:47, Masatake Iwasaki 
wrote:

> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>
are we adding some new dependencies from somewhere then? i never even knew
there was a com.sun.json module

hey, imagine if there was a single, standard, json library with a minimal
O/J mapping (strings, numbers, arrays and maps) -we'd be able to cut out
all of jackson, gson, jettison and maybe even avoid the eternal
jackson-databind CVE homework


>[INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
>[ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>Please check the following and either correct the build or update
>the allowed list with reasoning.
>
>com/
>com/sun/
>com/sun/jersey/
>com/sun/jersey/json/
>...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> Excluding them on hive-jdbc  might be the fix.
>
>[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
>...
>
>
ahh, covered in https://issues.apache.org/jira/browse/HADOOP-17563  ... the
maven shade plugin needs to be updated to handle the multi-JVM JAR

>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> >+ launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> >+ created encryption zone, put and read files via httpfs.
> >+ ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >  # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > + skimmed the site contents.
> >- Javadocs are not contained (under r3.3.5/api).
> >  # The issue can be reproduced even if I built site docs from the
> source.
> >
> > Masatake Iwasaki
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Mukund Madhav Thakur]

>
>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
>
I think it is because of what Ayush said. They are marked as done but not
fixed.
Also, some were marked with wrong fix versions, initially, we thought 3.3.9
but then decided to change to 3.3.5.
I have updated them and hopefully, in RC1 it will be fine.

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

I haven't got a chance to deep dive into HADOOP-18324
 which is claimed to be
the reason for these failures. Most probably will try to check next week if
it is still there.
>From the PR uploaded on HDFS-16853
 it looks like changing
or tweaking the cleanup logic itself rather than with playing with tests or
MiniDfsCluster, So, the clean up logic has issues but I still need to check
what is the impact of that, If I have a service and that terminates in a
non test setup, will the restart be an issue like these tests are facing,
my initial hunch was No. But I need to carefully check and see what is the
impact and what other issues it can cause. the original logic ain't
something which can be decoded with just a few seconds of cursory look.

++ @Owen O'Malley  is the original author of the
Hadoop Jira, maybe he can share some pointers about that.

-Ayush

On Thu, 5 Jan 2023 at 07:04, Chris Nauroth  wrote:

> Is it a problem limited to MiniDFSCluster, or is it a broader problem of
> RPC client resource cleanup? The patch is changing connection close
> cleanup, so I assumed the latter. If so, then it could potentially impact
> applications integrating with the RPC clients.
>
> If the problem is limited to MiniDFSCluster and restarts within a single
> JVM, then I agree the impact is smaller. Then, we'd want to consider what
> downstream projects have tests that do restarts on a MiniDFSCluster.
>
> Chris Nauroth
>
>
> On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena  wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> >> mentioned anywhere in change log or release notes. Are they actually
> >> up-to-date?
> >
> >
> > I don't think there is any issue with the ReleaseNotes generation as such
> > but with the Resolution type of this ticket, It ain't marked as Fixed but
> > Done. The other ticket which is marked Done is also not part of the
> release
> > notes. [1]
> >
> > if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >
> > Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> > can happen outside a MiniDfsCluster setup? Just guessing from the
> > description in the ticket. It looked like when we did a restart of the
> > Namenode in the MiniDfsCluster, I guess that would be in the same single
> > JVM, and that is why a previous blocked thread caused issues with the
> > restart. That is what I understood, I haven't checked the code though.
> >
> > Second, In the same context, Being curious If this lands up being a
> > MiniDfsCluster only issue, do we still consider this a release blocker?
> Not
> > saying in a way it won't be serious, MiniDfsCluster is very widely used
> by
> > downstream projects and all, so just wanted to know
> >
> > Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> > veto, I am not sure if it will get done any time soon, so if the use case
> > is something required, I would suggest handling it at Hadoop itself. It
> > seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> > with 3.3.5 and it passed. Other than that Hive officially support only
> > Hadoop-3.3.1 and that too only in the last 4.x release[2]
> >
> >
> > [1]
> >
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> > [2] https://issues.apache.org/jira/browse/HIVE-24484
> >
> > -Ayush
> >
> > On Tue, 3 Jan 2023 at 23:51, Chris Nauroth  wrote:
> >
> >> -1, because if I'm understanding the potential impact of HDFS-16853
> >> correctly, then it's serious enough to fix before a release. (I could
> >> change my vote if someone wants to make a case that it's not that
> >> serious.)
> >>
> >> Otherwise, this RC was looking good:
> >>
> >> * Verified all checksums.
> >> * Verified all signatures.
> >> * Built from source, including native code on Linux.
> >> * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >> -Drequire.zstd -DskipTests
> >> * Tests passed.
> >> * mvn --fail-never clean test -Pnative -Dparallel-tests
> >> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >> * Checked dependency tree to make sure we have all of the expected
> library
> >> updates that are mentioned in the release notes.
> >> * mvn -o dependency:tree
> >> * Farewell, S3Guard.
> >> * Confirmed that hadoop-openstack is now just a stub placeholder
> artifact
> >> with no code.
> >> * For ARM verification:
> >> * Ran "file " on all native binaries in the ARM tarball to
> confirm
> >> they actually came out with ARM as the architecture.
> >> * Output of hadoop 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Chris Nauroth]

Is it a problem limited to MiniDFSCluster, or is it a broader problem of
RPC client resource cleanup? The patch is changing connection close
cleanup, so I assumed the latter. If so, then it could potentially impact
applications integrating with the RPC clients.

If the problem is limited to MiniDFSCluster and restarts within a single
JVM, then I agree the impact is smaller. Then, we'd want to consider what
downstream projects have tests that do restarts on a MiniDFSCluster.

Chris Nauroth


On Wed, Jan 4, 2023 at 4:22 PM Ayush Saxena  wrote:

> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> mentioned anywhere in change log or release notes. Are they actually
>> up-to-date?
>
>
> I don't think there is any issue with the ReleaseNotes generation as such
> but with the Resolution type of this ticket, It ain't marked as Fixed but
> Done. The other ticket which is marked Done is also not part of the release
> notes. [1]
>
> if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>
> Chris, I just had a very quick look at HDFS-16853, I am not sure if this
> can happen outside a MiniDfsCluster setup? Just guessing from the
> description in the ticket. It looked like when we did a restart of the
> Namenode in the MiniDfsCluster, I guess that would be in the same single
> JVM, and that is why a previous blocked thread caused issues with the
> restart. That is what I understood, I haven't checked the code though.
>
> Second, In the same context, Being curious If this lands up being a
> MiniDfsCluster only issue, do we still consider this a release blocker? Not
> saying in a way it won't be serious, MiniDfsCluster is very widely used by
> downstream projects and all, so just wanted to know
>
> Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
> veto, I am not sure if it will get done any time soon, so if the use case
> is something required, I would suggest handling it at Hadoop itself. It
> seems to be centric to Hive-3.x, I tried compiling the Hive master branch
> with 3.3.5 and it passed. Other than that Hive officially support only
> Hadoop-3.3.1 and that too only in the last 4.x release[2]
>
>
> [1]
> https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
> [2] https://issues.apache.org/jira/browse/HIVE-24484
>
> -Ayush
>
> On Tue, 3 Jan 2023 at 23:51, Chris Nauroth  wrote:
>
>> -1, because if I'm understanding the potential impact of HDFS-16853
>> correctly, then it's serious enough to fix before a release. (I could
>> change my vote if someone wants to make a case that it's not that
>> serious.)
>>
>> Otherwise, this RC was looking good:
>>
>> * Verified all checksums.
>> * Verified all signatures.
>> * Built from source, including native code on Linux.
>> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
>> -Drequire.zstd -DskipTests
>> * Tests passed.
>> * mvn --fail-never clean test -Pnative -Dparallel-tests
>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
>> * Checked dependency tree to make sure we have all of the expected library
>> updates that are mentioned in the release notes.
>> * mvn -o dependency:tree
>> * Farewell, S3Guard.
>> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
>> with no code.
>> * For ARM verification:
>> * Ran "file " on all native binaries in the ARM tarball to confirm
>> they actually came out with ARM as the architecture.
>> * Output of hadoop checknative -a on ARM looks good.
>> * Ran a MapReduce job with the native bzip2 codec for compression, and
>> it worked fine.
>> * Ran a MapReduce job with YARN configured to use
>> LinuxContainerExecutor and verified launching the containers through
>> container-executor worked.
>>
>> My local setup didn't have the test failures mentioned by Viraj, though
>> there was some flakiness with a few HDFS snapshot tests timing out.
>>
>> Regarding Hive and Bouncy Castle, there is an existing issue and pull
>> request tracking an upgrade attempt. It's looking like some amount of code
>> changes are required:
>>
>> https://issues.apache.org/jira/browse/HIVE-26648
>> https://github.com/apache/hive/pull/3744
>>
>> Chris Nauroth
>>
>>
>> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun  wrote:
>>
>> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
>> > mentioned anywhere in change log or release notes. Are they actually
>> > up-to-date?
>> >
>> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
>> >  wrote:
>> > >
>> > > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due
>> to
>> > dependency change.
>> > >
>> > > For HBase, classes under com/sun/jersey/json/* 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

>
> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?


I don't think there is any issue with the ReleaseNotes generation as such
but with the Resolution type of this ticket, It ain't marked as Fixed but
Done. The other ticket which is marked Done is also not part of the release
notes. [1]

if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>

Chris, I just had a very quick look at HDFS-16853, I am not sure if this
can happen outside a MiniDfsCluster setup? Just guessing from the
description in the ticket. It looked like when we did a restart of the
Namenode in the MiniDfsCluster, I guess that would be in the same single
JVM, and that is why a previous blocked thread caused issues with the
restart. That is what I understood, I haven't checked the code though.

Second, In the same context, Being curious If this lands up being a
MiniDfsCluster only issue, do we still consider this a release blocker? Not
saying in a way it won't be serious, MiniDfsCluster is very widely used by
downstream projects and all, so just wanted to know

Regarding the Hive & Bouncy castle. The PR seems to have a valid binding
veto, I am not sure if it will get done any time soon, so if the use case
is something required, I would suggest handling it at Hadoop itself. It
seems to be centric to Hive-3.x, I tried compiling the Hive master branch
with 3.3.5 and it passed. Other than that Hive officially support only
Hadoop-3.3.1 and that too only in the last 4.x release[2]


[1]
https://issues.apache.org/jira/browse/HADOOP-11867?jql=project%20%3D%20HADOOP%20AND%20resolution%20%3D%20Done%20AND%20fixVersion%20%3D%203.3.5%20ORDER%20BY%20resolution%20DESC
[2] https://issues.apache.org/jira/browse/HIVE-24484

-Ayush

On Tue, 3 Jan 2023 at 23:51, Chris Nauroth  wrote:

> -1, because if I'm understanding the potential impact of HDFS-16853
> correctly, then it's serious enough to fix before a release. (I could
> change my vote if someone wants to make a case that it's not that serious.)
>
> Otherwise, this RC was looking good:
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
> * Farewell, S3Guard.
> * Confirmed that hadoop-openstack is now just a stub placeholder artifact
> with no code.
> * For ARM verification:
> * Ran "file " on all native binaries in the ARM tarball to confirm
> they actually came out with ARM as the architecture.
> * Output of hadoop checknative -a on ARM looks good.
> * Ran a MapReduce job with the native bzip2 codec for compression, and
> it worked fine.
> * Ran a MapReduce job with YARN configured to use
> LinuxContainerExecutor and verified launching the containers through
> container-executor worked.
>
> My local setup didn't have the test failures mentioned by Viraj, though
> there was some flakiness with a few HDFS snapshot tests timing out.
>
> Regarding Hive and Bouncy Castle, there is an existing issue and pull
> request tracking an upgrade attempt. It's looking like some amount of code
> changes are required:
>
> https://issues.apache.org/jira/browse/HIVE-26648
> https://github.com/apache/hive/pull/3744
>
> Chris Nauroth
>
>
> On Tue, Jan 3, 2023 at 8:57 AM Chao Sun  wrote:
>
> > Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> > mentioned anywhere in change log or release notes. Are they actually
> > up-to-date?
> >
> > On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
> >  wrote:
> > >
> > > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> > dependency change.
> > >
> > > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are
> not
> > expected in hbase-shaded-with-hadoop-check-invariants.
> > > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> > HBASE-27292.
> > >
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> > >
> > >[INFO] --- exec-maven-plugin:1.6.0:exec
> > (check-jar-contents-for-stuff-with-hadoop) @
> > hbase-shaded-with-hadoop-check-invariants ---
> > >[ERROR] Found artifact with unexpected contents:
> >
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> > >Please check the following and either correct the 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Chris Nauroth]

-1, because if I'm understanding the potential impact of HDFS-16853
correctly, then it's serious enough to fix before a release. (I could
change my vote if someone wants to make a case that it's not that serious.)

Otherwise, this RC was looking good:

* Verified all checksums.
* Verified all signatures.
* Built from source, including native code on Linux.
* mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
-Drequire.zstd -DskipTests
* Tests passed.
* mvn --fail-never clean test -Pnative -Dparallel-tests
-Drequire.snappy -Drequire.zstd -Drequire.openssl
-Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
* Checked dependency tree to make sure we have all of the expected library
updates that are mentioned in the release notes.
* mvn -o dependency:tree
* Farewell, S3Guard.
* Confirmed that hadoop-openstack is now just a stub placeholder artifact
with no code.
* For ARM verification:
* Ran "file " on all native binaries in the ARM tarball to confirm
they actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and
it worked fine.
* Ran a MapReduce job with YARN configured to use
LinuxContainerExecutor and verified launching the containers through
container-executor worked.

My local setup didn't have the test failures mentioned by Viraj, though
there was some flakiness with a few HDFS snapshot tests timing out.

Regarding Hive and Bouncy Castle, there is an existing issue and pull
request tracking an upgrade attempt. It's looking like some amount of code
changes are required:

https://issues.apache.org/jira/browse/HIVE-26648
https://github.com/apache/hive/pull/3744

Chris Nauroth


On Tue, Jan 3, 2023 at 8:57 AM Chao Sun  wrote:

> Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
> mentioned anywhere in change log or release notes. Are they actually
> up-to-date?
>
> On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
>  wrote:
> >
> > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >
> > For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> > Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
> >
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
> >
> >[INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
> >[ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
> >Please check the following and either correct the build or update
> >the allowed list with reasoning.
> >
> >com/
> >com/sun/
> >com/sun/jersey/
> >com/sun/jersey/json/
> >...
> >
> >
> > For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> > Excluding them on hive-jdbc  might be the fix.
> >
> >[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
> >...
> >
> >
> > On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > > Thanks for your great effort for the new release, Steve and Mukund.
> > >
> > > +1 while it would be nice if we can address missed Javadocs.
> > >
> > > + verified the signature and checksum.
> > > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> > >+ launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> > >+ created encryption zone, put and read files via httpfs.
> > >+ ran example MR wordcount over encryption zone.
> > > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> > >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> > >  # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > > + skimmed the site contents.
> > >- Javadocs are not contained (under r3.3.5/api).
> > >  # The issue can be reproduced even if I built site docs from the
> source.
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/12/22 4:28, Steve Loughran wrote:
> > >> Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> > >>
> > >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> > >> expect people to be able to test it 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Chao Sun]

Hmm I'm looking at HADOOP-11867 related stuff but couldn't find it
mentioned anywhere in change log or release notes. Are they actually
up-to-date?

On Mon, Jan 2, 2023 at 7:48 AM Masatake Iwasaki
 wrote:
>
> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to 
> > dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not 
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in 
> HBASE-27292.
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>[INFO] --- exec-maven-plugin:1.6.0:exec 
> (check-jar-contents-for-stuff-with-hadoop) @ 
> hbase-shaded-with-hadoop-check-invariants ---
>[ERROR] Found artifact with unexpected contents: 
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>Please check the following and either correct the build or update
>the allowed list with reasoning.
>
>com/
>com/sun/
>com/sun/jersey/
>com/sun/jersey/json/
>...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to 
> be problematic.
> Excluding them on hive-jdbc  might be the fix.
>
>[ERROR] Failed to execute goal 
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project 
> hive-jdbc: Error creating shaded jar: Problem shading JAR 
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
>  entry 
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
>  java.lang.IllegalArgumentException: Unsupported class file major version 59 
> -> [Help 1]
>...
>
>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native 
> > profile enabled.
> >+ launched pseudo distributed cluster including kms and httpfs with 
> > Kerberos and SSL enabled.
> >+ created encryption zone, put and read files via httpfs.
> >+ ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both 
> > x86_64 and aarch64).
> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to 
> > dependency change.
> >  # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 
> > worked.
> > + skimmed the site contents.
> >- Javadocs are not contained (under r3.3.5/api).
> >  # The issue can be reproduced even if I built site docs from the 
> > source.
> >
> > Masatake Iwasaki
> >
> > On 2022/12/22 4:28, Steve Loughran wrote:
> >> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
> >>
> >> Given the time of year it's a bit unrealistic to run a 5 day vote and
> >> expect people to be able to test it thoroughly enough to make this the one
> >> we can ship.
> >>
> >> What we would like is for anyone who can to verify the tarballs, and test
> >> the binaries, especially anyone who can try the arm64 binaries. We've got
> >> the building of those done and now the build file will incorporate them
> >> into the release -but neither of us have actually tested it yet. Maybe I
> >> should try it on my pi400 over xmas.
> >>
> >> The maven artifacts are up on the apache staging repo -they are the ones
> >> from x86 build. Building and testing downstream apps will be incredibly
> >> helpful.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >>
> >> The git tag is release-3.3.5-RC0, commit 3262495904d
> >>
> >> The maven artifacts are staged at
> >> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >>
> >> Release notes
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >>transitive CVEs under control -both genuine and false positive.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >>high-performance versions for file:// and s3a:// filesystems.
> >>file:// through java native io
> >>s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >>compatible systems validate these.
> >>
> >>
> >> Please try 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Masatake Iwasaki]

   - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to 
dependency change.


For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not 
expected in hbase-shaded-with-hadoop-check-invariants.
Updating hbase-shaded/pom.xml is expected to be the fix as done in HBASE-27292.
https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277

  [INFO] --- exec-maven-plugin:1.6.0:exec 
(check-jar-contents-for-stuff-with-hadoop) @ 
hbase-shaded-with-hadoop-check-invariants ---
  [ERROR] Found artifact with unexpected contents: 
'/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
  Please check the following and either correct the build or update
  the allowed list with reasoning.
  
  com/

  com/sun/
  com/sun/jersey/
  com/sun/jersey/json/
  ...
  


For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem to be 
problematic.
Excluding them on hive-jdbc  might be the fix.

  [ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on project 
hive-jdbc: Error creating shaded jar: Problem shading JAR 
/home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
 entry 
META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
 java.lang.IllegalArgumentException: Unsupported class file major version 59 -> 
[Help 1]
  ...


On 2023/01/02 22:02, Masatake Iwasaki wrote:

Thanks for your great effort for the new release, Steve and Mukund.

+1 while it would be nice if we can address missed Javadocs.

+ verified the signature and checksum.
+ built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile 
enabled.
   + launched pseudo distributed cluster including kms and httpfs with Kerberos 
and SSL enabled.
   + created encryption zone, put and read files via httpfs.
   + ran example MR wordcount over encryption zone.
+ built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both 
x86_64 and aarch64).
   - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to 
dependency change.
     # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
+ skimmed the site contents.
   - Javadocs are not contained (under r3.3.5/api).
     # The issue can be reproduced even if I built site docs from the source.

Masatake Iwasaki

On 2022/12/22 4:28, Steve Loughran wrote:

Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.

Given the time of year it's a bit unrealistic to run a 5 day vote and
expect people to be able to test it thoroughly enough to make this the one
we can ship.

What we would like is for anyone who can to verify the tarballs, and test
the binaries, especially anyone who can try the arm64 binaries. We've got
the building of those done and now the build file will incorporate them
into the release -but neither of us have actually tested it yet. Maybe I
should try it on my pi400 over xmas.

The maven artifacts are up on the apache staging repo -they are the ones
from x86 build. Building and testing downstream apps will be incredibly
helpful.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/

The git tag is release-3.3.5-RC0, commit 3262495904d

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1365/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
   transitive CVEs under control -both genuine and false positive.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
   high-performance versions for file:// and s3a:// filesystems.
   file:// through java native io
   s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
   compatible systems validate these.


Please try the release and vote on it, even though i don't know what is a
good timeline here...i'm actually going on holiday in early jan. Mukund is
around and so can drive the process while I'm offline.

Assuming we do have another iteration, the RC1 will not be before mid jan
for that reason

Steve (and mukund)



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: 

Re: [VOTE] Release Apache Hadoop 3.3.5

[Masatake Iwasaki]

Thanks for your great effort for the new release, Steve and Mukund.

+1 while it would be nice if we can address missed Javadocs.

+ verified the signature and checksum.
+ built from source tarball on Rocky Linux 8 and OpenJDK 8 with native profile 
enabled.
  + launched pseudo distributed cluster including kms and httpfs with Kerberos 
and SSL enabled.
  + created encryption zone, put and read files via httpfs.
  + ran example MR wordcount over encryption zone.
+ built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8 (both 
x86_64 and aarch64).
  - building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to dependency 
change.
# while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4 worked.
+ skimmed the site contents.
  - Javadocs are not contained (under r3.3.5/api).
# The issue can be reproduced even if I built site docs from the source.

Masatake Iwasaki

On 2022/12/22 4:28, Steve Loughran wrote:

Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.

Given the time of year it's a bit unrealistic to run a 5 day vote and
expect people to be able to test it thoroughly enough to make this the one
we can ship.

What we would like is for anyone who can to verify the tarballs, and test
the binaries, especially anyone who can try the arm64 binaries. We've got
the building of those done and now the build file will incorporate them
into the release -but neither of us have actually tested it yet. Maybe I
should try it on my pi400 over xmas.

The maven artifacts are up on the apache staging repo -they are the ones
from x86 build. Building and testing downstream apps will be incredibly
helpful.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/

The git tag is release-3.3.5-RC0, commit 3262495904d

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1365/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
   transitive CVEs under control -both genuine and false positive.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
   high-performance versions for file:// and s3a:// filesystems.
   file:// through java native io
   s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
   compatible systems validate these.


Please try the release and vote on it, even though i don't know what is a
good timeline here...i'm actually going on holiday in early jan. Mukund is
around and so can drive the process while I'm offline.

Assuming we do have another iteration, the RC1 will not be before mid jan
for that reason

Steve (and mukund)



-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.5

[Ayush Saxena]

Mostly or may be all of those failures are due to HADOOP-18324
, there is a Jira
tracking issues with TestLeaseRecovery2 linked to that as well HDFS-16853


-Ayush

On Wed, 28 Dec 2022 at 09:13, Viraj Jasani  wrote:

> -0 (non-binding)
>
> Output of hadoop-vote.sh:
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_341): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
>  - mvn clean install  -DskipTests
> * Built tar from source (1.8.0_341): ok
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> Manual testing on local mini cluster:
> * Basic CRUD tests on Hdfs looks good
> * Sample MapReduce job looks good
> * S3A tests look good with scale profile (ITestS3AContractUnbuffer is
> flaky, but when run individually, it passes)
>
> Full build with all modules UT results for branch-3.3.5 latest HEAD are
> available on
>
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-3.3.5-java8-linux-x86_64/
>
> From the above build, there are some consistently failing tests, out of
> which only TestDataNodeRollingUpgrade passed locally, whereas rest of the
> tests are consistently failing locally as well, we might want to fix (or
> ignore, if required) them:
>
>
> org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshot#testSnapshotsOnErasureCodingDirAfterNNRestart
>
> org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart#testFileLengthWithHSyncAndClusterRestartWithOutDNsRegister
>
> org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart
>
> org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart2
>
> org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryWithRenameAfterNameNodeRestart
>
> org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade#testWithLayoutChangeAndFinalize
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot#testSnapshotOpsOnRootReservedPath
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap#testReadRenamedSnapshotFileWithCheckpoint
>
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion#testApplyEditLogForDeletion
>
>
>
> On Wed, Dec 21, 2022 at 11:29 AM Steve Loughran
> 
> wrote:
>
> > Mukund and I have put together a release candidate (RC0) for Hadoop
> 3.3.5.
> >
> > Given the time of year it's a bit unrealistic to run a 5 day vote and
> > expect people to be able to test it thoroughly enough to make this the
> one
> > we can ship.
> >
> > What we would like is for anyone who can to verify the tarballs, and test
> > the binaries, especially anyone who can try the arm64 binaries. We've got
> > the building of those done and now the build file will incorporate them
> > into the release -but neither of us have actually tested it yet. Maybe I
> > should try it on my pi400 over xmas.
> >
> > The maven artifacts are up on the apache staging repo -they are the ones
> > from x86 build. Building and testing downstream apps will be incredibly
> > helpful.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
> >
> > The git tag is release-3.3.5-RC0, commit 3262495904d
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1365/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > Key changes include
> >
> > * Big update of dependencies to try and keep those reports of
> >   transitive CVEs under control -both genuine and false positive.
> > * HDFS RBF enhancements
> > * Critical fix to ABFS input stream prefetching for correct reading.
> > * Vectored IO API for all FSDataInputStream implementations, with
> >   high-performance versions for file:// and s3a:// filesystems.
> >   file:// through java native io
> >   s3a:// parallel GET requests.
> > * This release includes Arm64 binaries. Please can anyone with
> >   compatible systems validate these.
> >
> >
> > Please try the release and vote on it, even though i don't know what is a
> > good timeline here...i'm actually going on holiday in early jan. Mukund
> is
> > around and so can drive the process while I'm offline.
> >
> > Assuming we do have another iteration, the RC1 will not be before mid jan
> > for that reason
> >
> > Steve (and mukund)
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Viraj Jasani]

-0 (non-binding)

Output of hadoop-vote.sh:

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_341): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_341): ok
 - mvn clean install  -DskipTests
* Built tar from source (1.8.0_341): ok
 - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

Manual testing on local mini cluster:
* Basic CRUD tests on Hdfs looks good
* Sample MapReduce job looks good
* S3A tests look good with scale profile (ITestS3AContractUnbuffer is
flaky, but when run individually, it passes)

Full build with all modules UT results for branch-3.3.5 latest HEAD are
available on
https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-3.3.5-java8-linux-x86_64/

>From the above build, there are some consistently failing tests, out of
which only TestDataNodeRollingUpgrade passed locally, whereas rest of the
tests are consistently failing locally as well, we might want to fix (or
ignore, if required) them:

org.apache.hadoop.hdfs.TestErasureCodingPolicyWithSnapshot#testSnapshotsOnErasureCodingDirAfterNNRestart
org.apache.hadoop.hdfs.TestFileLengthOnClusterRestart#testFileLengthWithHSyncAndClusterRestartWithOutDNsRegister
org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart
org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryAfterNameNodeRestart2
org.apache.hadoop.hdfs.TestLeaseRecovery2#testHardLeaseRecoveryWithRenameAfterNameNodeRestart
org.apache.hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade#testWithLayoutChangeAndFinalize
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot#testSnapshotOpsOnRootReservedPath
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap#testReadRenamedSnapshotFileWithCheckpoint
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion#testApplyEditLogForDeletion



On Wed, Dec 21, 2022 at 11:29 AM Steve Loughran 
wrote:

> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>
> Given the time of year it's a bit unrealistic to run a 5 day vote and
> expect people to be able to test it thoroughly enough to make this the one
> we can ship.
>
> What we would like is for anyone who can to verify the tarballs, and test
> the binaries, especially anyone who can try the arm64 binaries. We've got
> the building of those done and now the build file will incorporate them
> into the release -but neither of us have actually tested it yet. Maybe I
> should try it on my pi400 over xmas.
>
> The maven artifacts are up on the apache staging repo -they are the ones
> from x86 build. Building and testing downstream apps will be incredibly
> helpful.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>
> The git tag is release-3.3.5-RC0, commit 3262495904d
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positive.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
>
>
> Please try the release and vote on it, even though i don't know what is a
> good timeline here...i'm actually going on holiday in early jan. Mukund is
> around and so can drive the process while I'm offline.
>
> Assuming we do have another iteration, the RC1 will not be before mid jan
> for that reason
>
> Steve (and mukund)
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Chris Nauroth]

I'm not quite ready to vote yet, pending some additional testing.

However, I wanted to give a quick update that ARM support is looking good
from my perspective. I focused on verifying the native bits that would need
to be different for ARM vs. x64. Here is what I did:
* Ran "file " on all native binaries in the ARM tarball to confirm they
actually came out with ARM as the architecture.
* Output of hadoop checknative -a on ARM looks good.
* Ran a MapReduce job with the native bzip2 codec for compression, and it
worked fine.
* Ran a MapReduce job with YARN configured to use LinuxContainerExecutor
and verified launching the containers through container-executor worked.

Chris Nauroth


On Wed, Dec 21, 2022 at 11:29 AM Steve Loughran 
wrote:

> Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.
>
> Given the time of year it's a bit unrealistic to run a 5 day vote and
> expect people to be able to test it thoroughly enough to make this the one
> we can ship.
>
> What we would like is for anyone who can to verify the tarballs, and test
> the binaries, especially anyone who can try the arm64 binaries. We've got
> the building of those done and now the build file will incorporate them
> into the release -but neither of us have actually tested it yet. Maybe I
> should try it on my pi400 over xmas.
>
> The maven artifacts are up on the apache staging repo -they are the ones
> from x86 build. Building and testing downstream apps will be incredibly
> helpful.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/
>
> The git tag is release-3.3.5-RC0, commit 3262495904d
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1365/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positive.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
>
>
> Please try the release and vote on it, even though i don't know what is a
> good timeline here...i'm actually going on holiday in early jan. Mukund is
> around and so can drive the process while I'm offline.
>
> Assuming we do have another iteration, the RC1 will not be before mid jan
> for that reason
>
> Steve (and mukund)
>

[VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.

Given the time of year it's a bit unrealistic to run a 5 day vote and
expect people to be able to test it thoroughly enough to make this the one
we can ship.

What we would like is for anyone who can to verify the tarballs, and test
the binaries, especially anyone who can try the arm64 binaries. We've got
the building of those done and now the build file will incorporate them
into the release -but neither of us have actually tested it yet. Maybe I
should try it on my pi400 over xmas.

The maven artifacts are up on the apache staging repo -they are the ones
from x86 build. Building and testing downstream apps will be incredibly
helpful.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/

The git tag is release-3.3.5-RC0, commit 3262495904d

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1365/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positive.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.


Please try the release and vote on it, even though i don't know what is a
good timeline here...i'm actually going on holiday in early jan. Mukund is
around and so can drive the process while I'm offline.

Assuming we do have another iteration, the RC1 will not be before mid jan
for that reason

Steve (and mukund)