Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

[Viraj Jasani]

+1 (non-binding)

* Signature: ok
* Checksum: ok
* Rat check (1.8.0_362): ok
- mvn clean apache-rat:check
* Built from source (1.8.0_362): ok
- mvn clean install -DskipTests
* Built tar from source (1.8.0_362): ok
- mvn clean package -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true

* Brought up pseudo-distributed mode cluster with hdfs, zk 3.7 and hbase 2.5
* Basic RBF tests look good
* S3A tests with input stream prefetch enabled: looks good


On Sun, Jun 18, 2023 at 5:53 PM Wei-Chiu Chuang  wrote:

> I am inviting anyone to try and vote on this release candidate.
>
> Note:
> This is exactly the same as RC0, except the CHANGELOG.
>
> The RC is available at:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)
>
> Git tag: release-3.3.6-RC1
> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
>
> Maven artifacts is built by x86 machine and are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1380/
>
> My public key:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Changelog:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
>
> Release notes:
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
>
> This is a relatively small release (by Hadoop standard) containing about
> 120 commits.
> Please give it a try, this RC vote will run for 7 days.
>
>
> Feature highlights:
>
> SBOM artifacts
> 
> Starting from this release, Hadoop publishes Software Bill of Materials
> (SBOM) using
> CycloneDX Maven plugin. For more information about SBOM, please go to
> [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
>
> HDFS RBF: RDBMS based token storage support
> 
> HDFS Router-Router Based Federation now supports storing delegation tokens
> on MySQL,
> [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
> which improves token operation through over the original Zookeeper-based
> implementation.
>
>
> New File System APIs
> 
> [HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
> number of
> HDFS-specific APIs to Hadoop Common to make it possible for certain
> applications that
> depend on HDFS semantics to run on other Hadoop compatible file systems.
>
> In particular, recoverLease() and isFileClosed() are exposed through
> LeaseRecoverable
> interface. While setSafeMode() is exposed through SafeMode interface.
>

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1065/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17055) Export HAState as a metric from Namenode for monitoring

[Xing Lin (Jira)]

Xing Lin created HDFS-17055:
---

 Summary: Export HAState as a metric from Namenode for monitoring
 Key: HDFS-17055
 URL: https://issues.apache.org/jira/browse/HDFS-17055
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: hdfs
Affects Versions: 3.4.0, 3.3.9
Reporter: Xing Lin


We'd like measure the uptime for Namenodes: percentage of time when we have the 
active/standby/observer node available (up and running). We could monitor the 
namenode from an external service, such as ZKFC. But that would require the 
external service to be available 100% itself. And when this third-party 
external monitoring service is down, we won't have info on whether our 
Namenodes are still up.

We propose to take a different approach: we will emit Namenode state directly 
from namenode itself. Whenever we miss a data point for this metric, we 
consider the corresponding namenode to be down/not available. In other words, 
we assume the metric collection/monitoring infrastructure to be 100% reliable.

One implementation detail: in hadoop, we have the _NameNodeMetrics_ class, 
which is used to emit all metrics for {_}NameNode.java{_}. However, we don't 
think that is a good place to emit NameNode HAState. HAState is stored in 
NameNode.java and we should directly emit it from NameNode.java. Otherwise, we 
basically duplicate this info in two classes and we would have to keep them in 
sync. Besides, _NameNodeMetrics_ class does not have a reference to the 
_NameNode_ object which it belongs to. An _NameNodeMetrics_ is created by a 
_static_ function _initMetrics()_ in {_}NameNode.java{_}. We shouldn't emit HA 
state from FSNameSystem.java either, as it is initialized from NameNode.java 
and all state transitions are implemented in NameNode.java.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

[Mukund Madhav Thakur]

Hi Wei-Chiu,
Thanks for driving the release.

+1 (binding)
Verified checksum and signature.
Built from source successfully.
Ran aws Itests
Ran azure Itests
Compiled hadoop-api-shim
Compiled google cloud storage.


I see two two test failures in GCS connector as seen by you. Created issue.
https://github.com/GoogleCloudDataproc/hadoop-connectors/issues/1014



On Wed, Jun 21, 2023 at 12:55 PM Wei-Chiu Chuang
 wrote:

> I am using Maven 3.6.3 on Mac (x86), JDK 1.8.0_341
> No issue for me.
>
> On Wed, Jun 21, 2023 at 5:48 AM Xiaoqiao He  wrote:
>
> > Addendum:
> > A. Build the release from sources using: `mvn clean install package
> > -DskipTests=true -Dmaven.javadoc.skip=true`
> > B. It works well when using the same command to build from the
> > source branch trunk.
> >
> > On Wed, Jun 21, 2023 at 8:44 PM Xiaoqiao He 
> wrote:
> >
> > > Hi,
> > >
> > > I met a fatal error when building from source on local Mac OSX. It
> > > could reproduce stably.
> > > Not sure if it is related to my local environment. Try to dig it, but
> not
> > > any conclusion right now.
> > > Will feedback once find reasons.
> > >
> > > Appendix system environment, some more stack information refer to the
> > > attachment please.
> > > OS:Bsduname:Darwin 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28
> > > 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
> > > rlimit: STACK 8192k, CORE 0k, NPROC 2784, NOFILE 10240, AS infinity
> > > load average:13.99 12.30 8.96
> > >
> > > CPU:total 12 (initial active 12) (6 cores per cpu, 2 threads per core)
> > > family 6 model 158 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3,
> > > ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref,
> > > lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx
> > >
> > > vm_info: Java HotSpot(TM) 64-Bit Server VM (25.202-b08) for bsd-amd64
> JRE
> > > (1.8.0_202-b08), built on Dec 15 2018 20:16:16 by "java_re" with gcc
> > 4.2.1
> > > (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
> > >
> > > mvn version: Apache Maven 3.6.0
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > >
> > > On Wed, Jun 21, 2023 at 10:09 AM Tak Lon (Stephen) Wu <
> tak...@apache.org
> > >
> > > wrote:
> > >
> > >> +1 (non-binding), and thanks a lot for driving the vote.
> > >>
> > >> * Signature of sources and binaries: ok
> > >> * Checksum of sources and binaries: ok
> > >> * Rat check (1.8.0_362): okie
> > >>  - mvn clean apache-rat:check
> > >> * Built from source (1.8.0_362): ok
> > >>  - mvn clean install -DskipTests
> > >> * Run Pseudo-Distributed mode with HDFS and YARN (1.8.0_362): ok
> > >> * Run Shell command (mkdir/put/ls/get) (1.8.0_362) : ok
> > >> * Run MR examples applications and check the result (1.8.0_362): ok
> > >>  - bin/hadoop jar
> > >> share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input
> > >> output 'dfs[a-z.]+'
> > >>
> > >> -Stephen
> > >>
> > >> On Tue, Jun 20, 2023 at 6:56 PM Masatake Iwasaki <
> > >> iwasak...@oss.nttdata.com>
> > >> wrote:
> > >>
> > >> > +1
> > >> >
> > >> > + verified the signature and checksum of the source tarball.
> > >> > + built from the source tarball on Rocky Linux 8 (x86_64) and
> OpenJDK
> > 8
> > >> > with native profile enabled.
> > >> >+ launched pseudo distributed cluster including kms and httpfs
> with
> > >> > Kerberos and SSL enabled.
> > >> >+ created encryption zone, put and read files via httpfs.
> > >> > + built RPM packages by Bigtop (modified to use ZooKeepr 3.6) on
> Rocky
> > >> > Linux 8 (x86_64).
> > >> >+ built HBase and Hive against Hadoop 3.3.6.
> > >> >+ ran smoke-tests of hdfs, yarn, mapreduce, hbase and hive.
> > >> > + skimmed the contents of site documentation.
> > >> >
> > >> > Thanks,
> > >> > Masatake Iwasaki
> > >> >
> > >> > On 2023/06/21 8:07, Wei-Chiu Chuang wrote:
> > >> > > Bumping this thread to the top.
> > >> > > If you are verifying the release, please vote on this thread. RC0
> > and
> > >> RC1
> > >> > > are exactly the same. The only material difference is the
> Changelog.
> > >> > >
> > >> > > Thanks!!
> > >> > >
> > >> > > On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang <
> weic...@apache.org
> > >
> > >> > wrote:
> > >> > >
> > >> > >> I am inviting anyone to try and vote on this release candidate.
> > >> > >>
> > >> > >> Note:
> > >> > >> This is exactly the same as RC0, except the CHANGELOG.
> > >> > >>
> > >> > >> The RC is available at:
> > >> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for
> > amd64)
> > >> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for
> > arm64)
> > >> > >>
> > >> > >> Git tag: release-3.3.6-RC1
> > >> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
> > >> > >>
> > >> > >> Maven artifacts is built by x86 machine and are staged at
> > >> > >>
> > >> >
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1380/
> > >> > >>
> > >> > >> My public key:
> > >> > >> 

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/

[Jun 20, 2023, 5:23:11 PM] (github) YARN-11505. [Federation] Add Steps To Set 
up a Test Cluster. (#5751)
[Jun 20, 2023, 5:23:51 PM] (github) YARN-11516. Improve 
FederationStateStoreFacade#existsApplicationHomeSubCluster/existsReservationHomeSubCluster
 Log Level. (#5755)
[Jun 20, 2023, 5:25:46 PM] (github) YARN-11510. [Federation] Fix 
NodeManager#TestFederationInterceptor Flaky Unit Test. (#5733)




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/blanks-eol.txt
 [14M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-hadolint.txt
 [20K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/results-javadoc-javadoc-root.txt
 [244K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
 [32K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
 [24K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1264/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 [72K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

[Wei-Chiu Chuang]

I am using Maven 3.6.3 on Mac (x86), JDK 1.8.0_341
No issue for me.

On Wed, Jun 21, 2023 at 5:48 AM Xiaoqiao He  wrote:

> Addendum:
> A. Build the release from sources using: `mvn clean install package
> -DskipTests=true -Dmaven.javadoc.skip=true`
> B. It works well when using the same command to build from the
> source branch trunk.
>
> On Wed, Jun 21, 2023 at 8:44 PM Xiaoqiao He  wrote:
>
> > Hi,
> >
> > I met a fatal error when building from source on local Mac OSX. It
> > could reproduce stably.
> > Not sure if it is related to my local environment. Try to dig it, but not
> > any conclusion right now.
> > Will feedback once find reasons.
> >
> > Appendix system environment, some more stack information refer to the
> > attachment please.
> > OS:Bsduname:Darwin 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28
> > 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
> > rlimit: STACK 8192k, CORE 0k, NPROC 2784, NOFILE 10240, AS infinity
> > load average:13.99 12.30 8.96
> >
> > CPU:total 12 (initial active 12) (6 cores per cpu, 2 threads per core)
> > family 6 model 158 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3,
> > ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref,
> > lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx
> >
> > vm_info: Java HotSpot(TM) 64-Bit Server VM (25.202-b08) for bsd-amd64 JRE
> > (1.8.0_202-b08), built on Dec 15 2018 20:16:16 by "java_re" with gcc
> 4.2.1
> > (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
> >
> > mvn version: Apache Maven 3.6.0
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> >
> > On Wed, Jun 21, 2023 at 10:09 AM Tak Lon (Stephen) Wu  >
> > wrote:
> >
> >> +1 (non-binding), and thanks a lot for driving the vote.
> >>
> >> * Signature of sources and binaries: ok
> >> * Checksum of sources and binaries: ok
> >> * Rat check (1.8.0_362): okie
> >>  - mvn clean apache-rat:check
> >> * Built from source (1.8.0_362): ok
> >>  - mvn clean install -DskipTests
> >> * Run Pseudo-Distributed mode with HDFS and YARN (1.8.0_362): ok
> >> * Run Shell command (mkdir/put/ls/get) (1.8.0_362) : ok
> >> * Run MR examples applications and check the result (1.8.0_362): ok
> >>  - bin/hadoop jar
> >> share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input
> >> output 'dfs[a-z.]+'
> >>
> >> -Stephen
> >>
> >> On Tue, Jun 20, 2023 at 6:56 PM Masatake Iwasaki <
> >> iwasak...@oss.nttdata.com>
> >> wrote:
> >>
> >> > +1
> >> >
> >> > + verified the signature and checksum of the source tarball.
> >> > + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK
> 8
> >> > with native profile enabled.
> >> >+ launched pseudo distributed cluster including kms and httpfs with
> >> > Kerberos and SSL enabled.
> >> >+ created encryption zone, put and read files via httpfs.
> >> > + built RPM packages by Bigtop (modified to use ZooKeepr 3.6) on Rocky
> >> > Linux 8 (x86_64).
> >> >+ built HBase and Hive against Hadoop 3.3.6.
> >> >+ ran smoke-tests of hdfs, yarn, mapreduce, hbase and hive.
> >> > + skimmed the contents of site documentation.
> >> >
> >> > Thanks,
> >> > Masatake Iwasaki
> >> >
> >> > On 2023/06/21 8:07, Wei-Chiu Chuang wrote:
> >> > > Bumping this thread to the top.
> >> > > If you are verifying the release, please vote on this thread. RC0
> and
> >> RC1
> >> > > are exactly the same. The only material difference is the Changelog.
> >> > >
> >> > > Thanks!!
> >> > >
> >> > > On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang  >
> >> > wrote:
> >> > >
> >> > >> I am inviting anyone to try and vote on this release candidate.
> >> > >>
> >> > >> Note:
> >> > >> This is exactly the same as RC0, except the CHANGELOG.
> >> > >>
> >> > >> The RC is available at:
> >> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for
> amd64)
> >> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for
> arm64)
> >> > >>
> >> > >> Git tag: release-3.3.6-RC1
> >> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
> >> > >>
> >> > >> Maven artifacts is built by x86 machine and are staged at
> >> > >>
> >> >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1380/
> >> > >>
> >> > >> My public key:
> >> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >> > >>
> >> > >> Changelog:
> >> > >>
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
> >> > >>
> >> > >> Release notes:
> >> > >>
> >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
> >> > >>
> >> > >> This is a relatively small release (by Hadoop standard) containing
> >> about
> >> > >> 120 commits.
> >> > >> Please give it a try, this RC vote will run for 7 days.
> >> > >>
> >> > >>
> >> > >> Feature highlights:
> >> > >>
> >> > >> SBOM artifacts
> >> > >> 
> >> > >> Starting from this release, Hadoop publishes Software Bill of
> >> Materials
> >> > >> (SBOM) using
> >> > >> CycloneDX Maven plugin. For more 

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

[Xiaoqiao He]

Addendum:
A. Build the release from sources using: `mvn clean install package
-DskipTests=true -Dmaven.javadoc.skip=true`
B. It works well when using the same command to build from the
source branch trunk.

On Wed, Jun 21, 2023 at 8:44 PM Xiaoqiao He  wrote:

> Hi,
>
> I met a fatal error when building from source on local Mac OSX. It
> could reproduce stably.
> Not sure if it is related to my local environment. Try to dig it, but not
> any conclusion right now.
> Will feedback once find reasons.
>
> Appendix system environment, some more stack information refer to the
> attachment please.
> OS:Bsduname:Darwin 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28
> 20:28:54 PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
> rlimit: STACK 8192k, CORE 0k, NPROC 2784, NOFILE 10240, AS infinity
> load average:13.99 12.30 8.96
>
> CPU:total 12 (initial active 12) (6 cores per cpu, 2 threads per core)
> family 6 model 158 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3,
> ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref,
> lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx
>
> vm_info: Java HotSpot(TM) 64-Bit Server VM (25.202-b08) for bsd-amd64 JRE
> (1.8.0_202-b08), built on Dec 15 2018 20:16:16 by "java_re" with gcc 4.2.1
> (Based on Apple Inc. build 5658) (LLVM build 2336.11.00)
>
> mvn version: Apache Maven 3.6.0
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Wed, Jun 21, 2023 at 10:09 AM Tak Lon (Stephen) Wu 
> wrote:
>
>> +1 (non-binding), and thanks a lot for driving the vote.
>>
>> * Signature of sources and binaries: ok
>> * Checksum of sources and binaries: ok
>> * Rat check (1.8.0_362): okie
>>  - mvn clean apache-rat:check
>> * Built from source (1.8.0_362): ok
>>  - mvn clean install -DskipTests
>> * Run Pseudo-Distributed mode with HDFS and YARN (1.8.0_362): ok
>> * Run Shell command (mkdir/put/ls/get) (1.8.0_362) : ok
>> * Run MR examples applications and check the result (1.8.0_362): ok
>>  - bin/hadoop jar
>> share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input
>> output 'dfs[a-z.]+'
>>
>> -Stephen
>>
>> On Tue, Jun 20, 2023 at 6:56 PM Masatake Iwasaki <
>> iwasak...@oss.nttdata.com>
>> wrote:
>>
>> > +1
>> >
>> > + verified the signature and checksum of the source tarball.
>> > + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8
>> > with native profile enabled.
>> >+ launched pseudo distributed cluster including kms and httpfs with
>> > Kerberos and SSL enabled.
>> >+ created encryption zone, put and read files via httpfs.
>> > + built RPM packages by Bigtop (modified to use ZooKeepr 3.6) on Rocky
>> > Linux 8 (x86_64).
>> >+ built HBase and Hive against Hadoop 3.3.6.
>> >+ ran smoke-tests of hdfs, yarn, mapreduce, hbase and hive.
>> > + skimmed the contents of site documentation.
>> >
>> > Thanks,
>> > Masatake Iwasaki
>> >
>> > On 2023/06/21 8:07, Wei-Chiu Chuang wrote:
>> > > Bumping this thread to the top.
>> > > If you are verifying the release, please vote on this thread. RC0 and
>> RC1
>> > > are exactly the same. The only material difference is the Changelog.
>> > >
>> > > Thanks!!
>> > >
>> > > On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang 
>> > wrote:
>> > >
>> > >> I am inviting anyone to try and vote on this release candidate.
>> > >>
>> > >> Note:
>> > >> This is exactly the same as RC0, except the CHANGELOG.
>> > >>
>> > >> The RC is available at:
>> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
>> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)
>> > >>
>> > >> Git tag: release-3.3.6-RC1
>> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
>> > >>
>> > >> Maven artifacts is built by x86 machine and are staged at
>> > >>
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1380/
>> > >>
>> > >> My public key:
>> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > >>
>> > >> Changelog:
>> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
>> > >>
>> > >> Release notes:
>> > >>
>> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
>> > >>
>> > >> This is a relatively small release (by Hadoop standard) containing
>> about
>> > >> 120 commits.
>> > >> Please give it a try, this RC vote will run for 7 days.
>> > >>
>> > >>
>> > >> Feature highlights:
>> > >>
>> > >> SBOM artifacts
>> > >> 
>> > >> Starting from this release, Hadoop publishes Software Bill of
>> Materials
>> > >> (SBOM) using
>> > >> CycloneDX Maven plugin. For more information about SBOM, please go to
>> > >> [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
>> > >>
>> > >> HDFS RBF: RDBMS based token storage support
>> > >> 
>> > >> HDFS Router-Router Based Federation now supports storing delegation
>> > tokens
>> > >> on MySQL,
>> > >> [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)

Re: [VOTE] Release Apache Hadoop 3.3.6 RC1

[Xiaoqiao He]

Hi,

I met a fatal error when building from source on local Mac OSX. It
could reproduce stably.
Not sure if it is related to my local environment. Try to dig it, but not
any conclusion right now.
Will feedback once find reasons.

Appendix system environment, some more stack information refer to the
attachment please.
OS:Bsduname:Darwin 21.2.0 Darwin Kernel Version 21.2.0: Sun Nov 28 20:28:54
PST 2021; root:xnu-8019.61.5~1/RELEASE_X86_64 x86_64
rlimit: STACK 8192k, CORE 0k, NPROC 2784, NOFILE 10240, AS infinity
load average:13.99 12.30 8.96

CPU:total 12 (initial active 12) (6 cores per cpu, 2 threads per core)
family 6 model 158 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3,
ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref,
lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.202-b08) for bsd-amd64 JRE
(1.8.0_202-b08), built on Dec 15 2018 20:16:16 by "java_re" with gcc 4.2.1
(Based on Apple Inc. build 5658) (LLVM build 2336.11.00)

mvn version: Apache Maven 3.6.0

Best Regards,
- He Xiaoqiao


On Wed, Jun 21, 2023 at 10:09 AM Tak Lon (Stephen) Wu 
wrote:

> +1 (non-binding), and thanks a lot for driving the vote.
>
> * Signature of sources and binaries: ok
> * Checksum of sources and binaries: ok
> * Rat check (1.8.0_362): okie
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_362): ok
>  - mvn clean install -DskipTests
> * Run Pseudo-Distributed mode with HDFS and YARN (1.8.0_362): ok
> * Run Shell command (mkdir/put/ls/get) (1.8.0_362) : ok
> * Run MR examples applications and check the result (1.8.0_362): ok
>  - bin/hadoop jar
> share/hadoop/mapreduce/hadoop-mapreduce-examples-3.3.6.jar grep input
> output 'dfs[a-z.]+'
>
> -Stephen
>
> On Tue, Jun 20, 2023 at 6:56 PM Masatake Iwasaki <
> iwasak...@oss.nttdata.com>
> wrote:
>
> > +1
> >
> > + verified the signature and checksum of the source tarball.
> > + built from the source tarball on Rocky Linux 8 (x86_64) and OpenJDK 8
> > with native profile enabled.
> >+ launched pseudo distributed cluster including kms and httpfs with
> > Kerberos and SSL enabled.
> >+ created encryption zone, put and read files via httpfs.
> > + built RPM packages by Bigtop (modified to use ZooKeepr 3.6) on Rocky
> > Linux 8 (x86_64).
> >+ built HBase and Hive against Hadoop 3.3.6.
> >+ ran smoke-tests of hdfs, yarn, mapreduce, hbase and hive.
> > + skimmed the contents of site documentation.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > On 2023/06/21 8:07, Wei-Chiu Chuang wrote:
> > > Bumping this thread to the top.
> > > If you are verifying the release, please vote on this thread. RC0 and
> RC1
> > > are exactly the same. The only material difference is the Changelog.
> > >
> > > Thanks!!
> > >
> > > On Sun, Jun 18, 2023 at 5:52 PM Wei-Chiu Chuang 
> > wrote:
> > >
> > >> I am inviting anyone to try and vote on this release candidate.
> > >>
> > >> Note:
> > >> This is exactly the same as RC0, except the CHANGELOG.
> > >>
> > >> The RC is available at:
> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)
> > >>
> > >> Git tag: release-3.3.6-RC1
> > >> https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1
> > >>
> > >> Maven artifacts is built by x86 machine and are staged at
> > >>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1380/
> > >>
> > >> My public key:
> > >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>
> > >> Changelog:
> > >> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md
> > >>
> > >> Release notes:
> > >>
> https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md
> > >>
> > >> This is a relatively small release (by Hadoop standard) containing
> about
> > >> 120 commits.
> > >> Please give it a try, this RC vote will run for 7 days.
> > >>
> > >>
> > >> Feature highlights:
> > >>
> > >> SBOM artifacts
> > >> 
> > >> Starting from this release, Hadoop publishes Software Bill of
> Materials
> > >> (SBOM) using
> > >> CycloneDX Maven plugin. For more information about SBOM, please go to
> > >> [SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).
> > >>
> > >> HDFS RBF: RDBMS based token storage support
> > >> 
> > >> HDFS Router-Router Based Federation now supports storing delegation
> > tokens
> > >> on MySQL,
> > >> [HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
> > >> which improves token operation through over the original
> Zookeeper-based
> > >> implementation.
> > >>
> > >>
> > >> New File System APIs
> > >> 
> > >> [HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671)
> > moved
> > >> a number of
> > >> HDFS-specific APIs to Hadoop Common to make it possible for certain
> > >> applications that
> > >> depend on 

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Steve Loughran]

1. we should patch gcs, to ignore those new keys.
2. i may be able to validate RC1 this weekend



On Fri, 16 Jun 2023 at 03:20, Wei-Chiu Chuang 
wrote:

> Overall so far so good.
>
> hadoop-api-shim:
> built, tested successfully.
>
> cloudstore:
> built successfully.
>
> Spark:
> built successfully. Passed hadoop-cloud tests.
>
> Ozone:
> One test failure due to unrelated Ozone issue. This test is being disabled
> in the latest Ozone code.
>
> org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
> to load library ozone_rocksdb_tools from both java.library.path &
> resource file libozone_rocksdb_t
> ools.so from jar.
> at
> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
>
>
> Google gcs:
> There are two test failures. The tests were added recently by HADOOP-18724
>  in Hadoop 3.3.6. This
> is okay. Not production code problem. Can be addressed in GCS code.
>
> [ERROR] Errors:
> [ERROR]
>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
> et/contract-test/testFloatingPointLength "fs.option.openfile.length"
> [ERROR]
>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
> ucket/contract-test/testOpenFileApplyAsyncRead "fs.option.openfile.length"
>
>
>
>
>
> On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
> wrote:
>
> > The hbase-filesystem tests passed after reverting HADOOP-18596
> >  and HADOOP-18633
> >  from my local tree.
> > So I think it's a matter of the default behavior being changed. It's not
> > the end of the world. I think we can address it by adding an incompatible
> > change flag and a release note.
> >
> > On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
> > wrote:
> >
> >> Cross referenced git history and jira. Changelog needs some update
> >>
> >> Not in the release
> >>
> >>1. HDFS-16858 
> >>
> >>
> >>1. HADOOP-18532 
> >>2.
> >>   1. HDFS-16861 
> >>  2.
> >> 1. HDFS-16866
> >> 
> >> 2.
> >>1. HADOOP-18320
> >>
> >>2.
> >>
> >> Updated fixed version. Will generate. new Changelog in the next RC.
> >>
> >> Was able to build HBase and hbase-filesystem without any code change.
> >>
> >> hbase has one unit test failure. This one is reproducible even with
> >> Hadoop 3.3.5, so maybe a red herring. Local env or something.
> >>
> >> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> >> 9.007 s <<< FAILURE! - in
> >> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> >> [ERROR]
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> >>  Time elapsed: 3.13 s  <<< ERROR!
> >> java.lang.OutOfMemoryError: Java heap space
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)
> >> at
> >>
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness(TestSyncTimeRangeTracker.java:156)
> >>
> >> hbase-filesystem has three test failures in TestHBOSSContractDistCp, and
> >> is not reproducible with Hadoop 3.3.5.
> >> [ERROR] Failures: [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip:976->Assert.fail:88
> >> 10 errors in file of length 10
> >> [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureNoChange:270->AbstractContractDistCpTest.assertCounterInRange:290->Assert.assertTrue:41->Assert.fail:88
> >> Files Skipped value 0 too below minimum 1
> >> [ERROR]
> >>
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureToRemote:259->AbstractContractDistCpTest.distCpUpdateDeepDirectoryStructure:334->AbstractContractDistCpTest.assertCounterInRange:294->Assert.assertTrue:41->Assert.fail:88
> >> Files Copied value 2 above maximum 1
> >> [INFO]
> >> [ERROR] Tests run: 240, Failures: 3, Errors: 0, Skipped: 58
> >>
> >>
> >> Ozone
> >> test in progress. Will report back.
> >>
> >>
> >> On Tue, Jun 13, 2023 at 11:27 PM Wei-Chiu Chuang 
> >> wrote:
> >>
> >>> I am inviting anyone to try and vote on this release candidate.
> >>>
> >>> Note:
> >>> This is built off branch-3.3.6 plus PR#5741 (aws sdk update) and
> PR#5740
> >>> (LICENSE file update)
> >>>
> >>> The RC is available at:
> >>>