Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Ayush Saxena]

Hi Masatake,
That ticket is just a backport of:
https://issues.apache.org/jira/browse/HADOOP-17612 and the only java code
change is in test [1], which is causing issues for you. That ticket isn't
marked incompatible either, so maybe that is why it got pulled back.

btw. the original PR does mention it to be compatible [2].

Are you trying to compile ZK-3.5 with Hadoop-3.3.6 or Hadoop-3.3.6 with
ZK-3.5, If later and then if the compilation fails, then it shouldn't be an
incompatible change, right? or do we need to maintain compat that way as
well?

+1 in maintaining compatibility, incompatible changes should be avoided as
far as possible unless excessively necessary even in trunk or like we need
to do it for some "relevant" security issue or so in those thirdparty libs.
The RC1 vote is already up, do you plan to get this change excluded from
that?

Regarding the test, If you pass ``null`` instead of that DisconnectReason,
then also the test test passes, but I am pretty sure you would get
NoSuchMethod error for closeAll after that because that closeAll ain't
there is Zk-3.5, ZOOKEEPER-3439 removed it

-Ayush

PS. From this doc: https://zookeeper.apache.org/releases.html, even ZK-3.6
line is also EOL, not sure how those guys operate :-)

[1]
https://github.com/apache/hadoop/pull/3241/files#diff-b273546d6f060e617553eaa49da69039d2c655a77d42022779c2281d0f6cd08eR135
[2] https://github.com/apache/hadoop/pull/3241#issuecomment-889185103

On Mon, 19 Jun 2023 at 06:44, Masatake Iwasaki 
wrote:

> I got compilation error against ZooKeeper 3.5 due to HADOOP-18515.
> It should be marked as incompatible change?
> https://issues.apache.org/jira/browse/HADOOP-18515
>
> ::
>
>[ERROR]
> /home/rocky/srcs/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.6-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverControllerStress.java:[135,40]
> cannot find symbol
>  symbol:   variable DisconnectReason
>  location: class org.apache.zookeeper.server.ServerCnxn
>
> While ZooKeeper 3.5 is already EoL, it would be nice to keep compatibility
> in a patch release
> especially if only test code is the cause.
>
> Thanks,
> Masatake Iwasaki
>
> On 2023/06/18 4:57, Wei-Chiu Chuang wrote:
> > I was going to do another RC in case something comes up.
> > But it looks like the only thing that needs to be fixed is the Changelog.
> >
> >
> > 1. HADOOP-18596 
> >
> > HADOOP-18633 
> > are related to cloud store semantics, and I don't want to make a
> judgement
> > call on it. As far as I can tell its effect can be addressed by
> supplying a
> > config option in the application code.
> > It looks like the feature improves fault tolerance by ensuring files are
> > synchronized if modification time is different between the source and
> > destination. So to me it's the better behavior.
> >
> > I can make a RC1 over the weekend to fix the Changelog but that's
> probably
> > the only thing that's going to have.
> > On Sat, Jun 17, 2023 at 2:00 AM Xiaoqiao He 
> wrote:
> >
> >> Thanks Wei-Chiu for driving this release. The next RC will be prepared,
> >> right?
> >> If true, I would like to try and vote on the next RC.
> >> Just notice that some JIRAs are not included and need to revert some
> PRs to
> >> pass HBase verification which are mentioned above.
> >>
> >> Best Regards,
> >> - He Xiaoqiao
> >>
> >>
> >> On Fri, Jun 16, 2023 at 9:20 AM Wei-Chiu Chuang
> >>  wrote:
> >>
> >>> Overall so far so good.
> >>>
> >>> hadoop-api-shim:
> >>> built, tested successfully.
> >>>
> >>> cloudstore:
> >>> built successfully.
> >>>
> >>> Spark:
> >>> built successfully. Passed hadoop-cloud tests.
> >>>
> >>> Ozone:
> >>> One test failure due to unrelated Ozone issue. This test is being
> >> disabled
> >>> in the latest Ozone code.
> >>>
> >>> org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
> >>> to load library ozone_rocksdb_tools from both java.library.path &
> >>> resource file libozone_rocksdb_t
> >>> ools.so from jar.
> >>>  at
> >>>
> >>
> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
> >>>
> >>>
> >>> Google gcs:
> >>> There are two test failures. The tests were added recently by
> >> HADOOP-18724
> >>>  in Hadoop 3.3.6.
> >> This
> >>> is okay. Not production code problem. Can be addressed in GCS code.
> >>>
> >>> [ERROR] Errors:
> >>> [ERROR]
> >>>
> >>>
> >>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
> >>> » IllegalArgument Unknown mandatory key for
> gs://fake-in-memory-test-buck
> >>> et/contract-test/testFloatingPointLength "fs.option.openfile.length"
> >>> [ERROR]
> >>>
> >>>
> >>
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
> >>> » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Masatake Iwasaki]

I got compilation error against ZooKeeper 3.5 due to HADOOP-18515.
It should be marked as incompatible change?
https://issues.apache.org/jira/browse/HADOOP-18515

::

  [ERROR] 
/home/rocky/srcs/bigtop/build/hadoop/rpm/BUILD/hadoop-3.3.6-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverControllerStress.java:[135,40]
 cannot find symbol
symbol:   variable DisconnectReason
location: class org.apache.zookeeper.server.ServerCnxn

While ZooKeeper 3.5 is already EoL, it would be nice to keep compatibility in a 
patch release
especially if only test code is the cause.

Thanks,
Masatake Iwasaki

On 2023/06/18 4:57, Wei-Chiu Chuang wrote:

I was going to do another RC in case something comes up.
But it looks like the only thing that needs to be fixed is the Changelog.


1. HADOOP-18596 

HADOOP-18633 
are related to cloud store semantics, and I don't want to make a judgement
call on it. As far as I can tell its effect can be addressed by supplying a
config option in the application code.
It looks like the feature improves fault tolerance by ensuring files are
synchronized if modification time is different between the source and
destination. So to me it's the better behavior.

I can make a RC1 over the weekend to fix the Changelog but that's probably
the only thing that's going to have.
On Sat, Jun 17, 2023 at 2:00 AM Xiaoqiao He  wrote:


Thanks Wei-Chiu for driving this release. The next RC will be prepared,
right?
If true, I would like to try and vote on the next RC.
Just notice that some JIRAs are not included and need to revert some PRs to
pass HBase verification which are mentioned above.

Best Regards,
- He Xiaoqiao


On Fri, Jun 16, 2023 at 9:20 AM Wei-Chiu Chuang
 wrote:


Overall so far so good.

hadoop-api-shim:
built, tested successfully.

cloudstore:
built successfully.

Spark:
built successfully. Passed hadoop-cloud tests.

Ozone:
One test failure due to unrelated Ozone issue. This test is being

disabled

in the latest Ozone code.

org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
to load library ozone_rocksdb_tools from both java.library.path &
resource file libozone_rocksdb_t
ools.so from jar.
 at


org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)



Google gcs:
There are two test failures. The tests were added recently by

HADOOP-18724

 in Hadoop 3.3.6.

This

is okay. Not production code problem. Can be addressed in GCS code.

[ERROR] Errors:
[ERROR]



TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403

» IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
et/contract-test/testFloatingPointLength "fs.option.openfile.length"
[ERROR]



TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341

» IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
ucket/contract-test/testOpenFileApplyAsyncRead

"fs.option.openfile.length"






On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
wrote:


The hbase-filesystem tests passed after reverting HADOOP-18596
 and HADOOP-18633
 from my local

tree.

So I think it's a matter of the default behavior being changed. It's

not

the end of the world. I think we can address it by adding an

incompatible

change flag and a release note.

On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
wrote:


Cross referenced git history and jira. Changelog needs some update

Not in the release

1. HDFS-16858 


1. HADOOP-18532 <

https://issues.apache.org/jira/browse/HADOOP-18532>

2.
   1. HDFS-16861 
 2.
1. HADOOP-18320

2.

Updated fixed version. Will generate. new Changelog in the next RC.

Was able to build HBase and hbase-filesystem without any code change.

hbase has one unit test failure. This one is reproducible even with
Hadoop 3.3.5, so maybe a red herring. Local env or something.

[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time

elapsed:

9.007 s <<< FAILURE! - in
org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
[ERROR]




org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness

  Time elapsed: 3.13 s  <<< ERROR!
java.lang.OutOfMemoryError: Java heap space
at




org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)

at




org.apache.hadoop

[VOTE] Release Apache Hadoop 3.3.6 RC1

[Wei-Chiu Chuang]

I am inviting anyone to try and vote on this release candidate.

Note:
This is exactly the same as RC0, except the CHANGELOG.

The RC is available at:
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/ (for amd64)
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-arm64/ (for arm64)

Git tag: release-3.3.6-RC1
https://github.com/apache/hadoop/releases/tag/release-3.3.6-RC1

Maven artifacts is built by x86 machine and are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1380/

My public key:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Changelog:
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/CHANGELOG.md

Release notes:
https://home.apache.org/~weichiu/hadoop-3.3.6-RC1-amd64/RELEASENOTES.md

This is a relatively small release (by Hadoop standard) containing about
120 commits.
Please give it a try, this RC vote will run for 7 days.


Feature highlights:

SBOM artifacts

Starting from this release, Hadoop publishes Software Bill of Materials
(SBOM) using
CycloneDX Maven plugin. For more information about SBOM, please go to
[SBOM](https://cwiki.apache.org/confluence/display/COMDEV/SBOM).

HDFS RBF: RDBMS based token storage support

HDFS Router-Router Based Federation now supports storing delegation tokens
on MySQL,
[HADOOP-18535](https://issues.apache.org/jira/browse/HADOOP-18535)
which improves token operation through over the original Zookeeper-based
implementation.


New File System APIs

[HADOOP-18671](https://issues.apache.org/jira/browse/HADOOP-18671) moved a
number of
HDFS-specific APIs to Hadoop Common to make it possible for certain
applications that
depend on HDFS semantics to run on other Hadoop compatible file systems.

In particular, recoverLease() and isFileClosed() are exposed through
LeaseRecoverable
interface. While setSafeMode() is exposed through SafeMode interface.

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1062/

No changes


ERROR: File 'out/email-report.txt' does not exist

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[Apache Jenkins Server]

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/

No changes




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.yarn.server.applicationhistoryservice.webapp.TestAHSWebServices 
   hadoop.mapreduce.v2.TestUberAM 
   hadoop.mapreduce.v2.TestMRJobsWithProfiler 
   hadoop.mapreduce.v2.TestMRJobs 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/blanks-eol.txt
 [14M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-hadolint.txt
 [20K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/xml.txt
 [24K]

   javadoc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/results-javadoc-javadoc-root.txt
 [244K]

   unit:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt
 [32K]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1261/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
 [72K]

Powered by Apache Yetus 0.14.0-SNAPSHOT   https://yetus.apache.org

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org