Re: [DISCUSS] what needs to be done to switch to java17

[Ashutosh Gupta]

Thanks, everyone, for starting this discussion. This sounds like a good
plan to start with. As I was working on the JUnit 4 to 5 upgrade, I paused
for a while as I got occupied with other stuff. But I would be happy to
complete it as part of the Java switch process.

On Tue, Jul 30, 2024 at 4:21 AM slfan1989  wrote:

> Thank you very much for initiating this discussion! I am also very much
> looking forward to JDK 17. I have observed that using --add-opens= is quite
> common in other Apache projects. One thing I know is that it is necessary
> to upgrade JUnit 4 to JUnit 5. There are some JIRA issues currently being
> worked on for this. For YARN MapReduce, some pull requests have already
> been completed for this feature. Upgrading Jersey is a very challenging
> project. Currently, our upgrade strategy is to perform it all at once. If
> we can find a way to break Jersey down into individual modules for
> upgrading, it might be much easier.
>
> I have successfully debugged most of the unit tests for YARN's
> ResourceManager and NodeManager.(Upgrade Jersey 2.4.1)  Additionally, we
> are using Hadoop compiled with JDK 17 internally (mainly on the current
> trunk branch with --add-opens=). We have replaced over 600 NM instances,
> and this part is functioning normally. Therefore, I believe we are
> essentially ready to upgrade to JDK 17.
>
> My idea is to create a separate branch specifically for upgrading to JDK
> 17. After debugging and ensuring everything works, we can then merge this
> branch into the trunk branch.
>
> By the way, there's a JIRA that we have had to revert twice
> (HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin.
>
> [1] https://issues.apache.org/jira/browse/HADOOP-19071
>
> - Shilun Fan
>
> On Tue, 30 Jul 2024 at 06:37, Shilun Fan wrote:
>
> > Depends on what we mean by switching to JDK-17, Compile time support
> > or Runtime support, We don't have compile time support for JDK-11 too,
> > it is just runtime, We have a daily build as well for JDK-11, which
> > has now some genuine failures now, need to check [1], as of now the
> > version is hardcoded here [2], unless we change here it but just
> > change our local JAVA_HOME, it will compile with JDK-17 as well I
> > believe
>
> > So, If we are to chase the runtime support we can setup a regular CI
> > with JDK-17 like we have one for JDK-11, there were some test failures
> > reported as part of HADOOP-18716, most of them should be fixable by
> > some dependency upgrades or some hacks like adding one of these
> > ```
> > --add-opens=java.base/java.net=ALL-UNNAMED
> > --add-opens=java.base/java.util=ALL-UNNAMED
> > --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
> > --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
> > --add-opens=java.base/java.lang=ALL-UNNAMED
> > --add-opens=java.base/java.util.regex=ALL-UNNAMED
> > --add-opens=java.base/java.io=ALL-UNNAMED
> > ```
>
> > That is some typical JDK-17 hack
>
> > For the compile support, that is we change the value at [2] to 17, I
> > think then some bigger problems will surface, One I know is Jersey
> > creates some issues for sure, We need to upgrade Jersey, As per this
> > [3] to Jersey 2.35+ or do some hack to use a patched jersey version
> > with minimal change which can unblock the JDK-17 initiative
>
> > -Ayush
>
>
> > [1]
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/709/
> > [2]
>
> https://github.com/apache/hadoop/blob/038636a1b5250e06622cac7ee11b12965c9e/hadoop-project/pom.xml#L160
> > [3] https://github.com/eclipse-ee4j/jersey/wiki/Road-Map
>
> On Tue, 30 Jul 2024 at 00:53, PJ Fanning wrote:
> >
> > I think this is worth considering. I think it would require a minor
> > release like 3.5.0 as opposed to considering it for future 3.4.x patch
> > releases.
> > I tend to build locally with Java 11, by default and I haven't hit
> > major issues building Hadoop. There may be some gotcha somewhere but
> > it is likely to be easy enough to fix.
> >
> > It's probably only a matter of time before some important dependency
> > jars for Hadoop require Java 11 or 17.
> >
> > On Mon, 29 Jul 2024 at 20:04, Steve Loughran
> > wrote:
> > >
> > > A lot of projects are moving off java8. making java17 the new baseline
> > >
> > > what do we need to there that is blocker rather than just "nice"'?
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Re: [VOTE] Release Apache Hadoop 3.3.6 RC0

[Ashutosh Gupta]

Hi

Thanks Wei-Chiu for driving the release.

+1 (non-binding)

* Builds from source looks good.
* Checksums and signatures look good.
* Running basic HDFS commands and running simple MapReduce jobs looks good.
* hadoop-tools/hadoop-aws UTs and ITs looks good

Thanks,
Ash

On Tue, Jun 20, 2023 at 5:18 PM Mukund Madhav Thakur
 wrote:

> Hi Wei-Chiu,
> Thanks for driving the release.
>
> +1 (binding)
> Verified checksum and signature.
> Built from source successfully.
> Ran aws Itests
> Ran azure Itests
> Compiled hadoop-api-shim
> Compiled google cloud storage.
>
>
> I did see the two test failures in GCS connector as well but those are
> harmless.
>
>
>
> On Thu, Jun 15, 2023 at 8:21 PM Wei-Chiu Chuang
>  wrote:
>
> > Overall so far so good.
> >
> > hadoop-api-shim:
> > built, tested successfully.
> >
> > cloudstore:
> > built successfully.
> >
> > Spark:
> > built successfully. Passed hadoop-cloud tests.
> >
> > Ozone:
> > One test failure due to unrelated Ozone issue. This test is being
> disabled
> > in the latest Ozone code.
> >
> > org.apache.hadoop.hdds.utils.NativeLibraryNotLoadedException: Unable
> > to load library ozone_rocksdb_tools from both java.library.path &
> > resource file libozone_rocksdb_t
> > ools.so from jar.
> > at
> >
> org.apache.hadoop.hdds.utils.db.managed.ManagedSSTDumpTool.(ManagedSSTDumpTool.java:49)
> >
> >
> > Google gcs:
> > There are two test failures. The tests were added recently by
> HADOOP-18724
> >  in Hadoop 3.3.6.
> This
> > is okay. Not production code problem. Can be addressed in GCS code.
> >
> > [ERROR] Errors:
> > [ERROR]
> >
> >
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testFloatingPointLength:403
> > » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-buck
> > et/contract-test/testFloatingPointLength "fs.option.openfile.length"
> > [ERROR]
> >
> >
> TestInMemoryGoogleContractOpen>AbstractContractOpenTest.testOpenFileApplyAsyncRead:341
> > » IllegalArgument Unknown mandatory key for gs://fake-in-memory-test-b
> > ucket/contract-test/testOpenFileApplyAsyncRead
> "fs.option.openfile.length"
> >
> >
> >
> >
> >
> > On Wed, Jun 14, 2023 at 5:01 PM Wei-Chiu Chuang 
> > wrote:
> >
> > > The hbase-filesystem tests passed after reverting HADOOP-18596
> > >  and HADOOP-18633
> > >  from my local
> tree.
> > > So I think it's a matter of the default behavior being changed. It's
> not
> > > the end of the world. I think we can address it by adding an
> incompatible
> > > change flag and a release note.
> > >
> > > On Wed, Jun 14, 2023 at 3:55 PM Wei-Chiu Chuang 
> > > wrote:
> > >
> > >> Cross referenced git history and jira. Changelog needs some update
> > >>
> > >> Not in the release
> > >>
> > >>1. HDFS-16858 
> > >>
> > >>
> > >>1. HADOOP-18532 <
> https://issues.apache.org/jira/browse/HADOOP-18532>
> > >>2.
> > >>   1. HDFS-16861  >
> > >>  2.
> > >> 1. HDFS-16866
> > >> 
> > >> 2.
> > >>1. HADOOP-18320
> > >>
> > >>2.
> > >>
> > >> Updated fixed version. Will generate. new Changelog in the next RC.
> > >>
> > >> Was able to build HBase and hbase-filesystem without any code change.
> > >>
> > >> hbase has one unit test failure. This one is reproducible even with
> > >> Hadoop 3.3.5, so maybe a red herring. Local env or something.
> > >>
> > >> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time
> elapsed:
> > >> 9.007 s <<< FAILURE! - in
> > >> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker
> > >> [ERROR]
> > >>
> >
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness
> > >>  Time elapsed: 3.13 s  <<< ERROR!
> > >> java.lang.OutOfMemoryError: Java heap space
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker$RandomTestData.(TestSyncTimeRangeTracker.java:91)
> > >> at
> > >>
> >
> org.apache.hadoop.hbase.regionserver.TestSyncTimeRangeTracker.testConcurrentIncludeTimestampCorrectness(TestSyncTimeRangeTracker.java:156)
> > >>
> > >> hbase-filesystem has three test failures in TestHBOSSContractDistCp,
> and
> > >> is not reproducible with Hadoop 3.3.5.
> > >> [ERROR] Failures: [ERROR]
> > >>
> >
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testDistCpUpdateCheckFileSkip:976->Assert.fail:88
> > >> 10 errors in file of length 10
> > >> [ERROR]
> > >>
> >
> TestHBOSSContractDistCp>AbstractContractDistCpTest.testUpdateDeepDirectoryStructureNoChange:270->AbstractContractDistCpTest.assertCounterInRange:290->Assert.assertTrue:41->Assert.fail:88
> > >> Files Ski

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Ashutosh Gupta]

Hi Steve

I will also do it by today/tomorrow.

Thanks,
Ashutosh

On Sat, 18 Mar, 2023, 4:07 pm Steve Loughran, 
wrote:

> Thank you for this!
>
> Can anyone else with time do a review too? i really want to get this one
> done, now the HDFS issues are all resolved.
>
> I do not want this release to fall by the wayside through lack of votes
> alone. In fact, I would be very unhappy
>
>
>
> On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:
>
> > +1 (non-binding)
> >
> > * Signature/Checksum: ok
> > * Rat check (1.8.0_341): ok
> >  - mvn clean apache-rat:check
> > * Built from source (1.8.0_341): ok
> >  - mvn clean install  -DskipTests
> > * Built tar from source (1.8.0_341): ok
> >  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
> >
> > Containerized deployments:
> > * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7
> > * Deployed and started JHS, RM, NM
> > * Hbase, hdfs CRUD looks good
> > * Sample RowCount MapReduce job looks good
> >
> > * S3A tests with scale profile looks good
> >
> >
> > On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
> > 
> > wrote:
> >
> > > Apache Hadoop 3.3.5
> > >
> > > Mukund and I have put together a release candidate (RC3) for Hadoop
> > 3.3.5.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> > >
> > > The git tag is release-3.3.5-RC3, commit 706d88266ab
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.3 and is the first big release since 3.3.2.
> > >
> > > Key changes include
> > >
> > > * Big update of dependencies to try and keep those reports of
> > >   transitive CVEs under control -both genuine and false positives.
> > > * HDFS RBF enhancements
> > > * Critical fix to ABFS input stream prefetching for correct reading.
> > > * Vectored IO API for all FSDataInputStream implementations, with
> > >   high-performance versions for file:// and s3a:// filesystems.
> > >   file:// through java native io
> > >   s3a:// parallel GET requests.
> > > * This release includes Arm64 binaries. Please can anyone with
> > >   compatible systems validate these.
> > > * and compared to the previous RC, all the major changes are
> > >   HDFS issues.
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > -Steve
> > >
> >
>

Re: [ANNOUNCE] New Hadoop PMC member - Mukund Thakur

[Ashutosh Gupta]

Congratulations Mukund. Well deserved!

On Wed, 8 Feb, 2023, 11:07 pm Mukund Madhav Thakur,
 wrote:

> Thanks Everyone :)
>
> On Wed, Feb 8, 2023 at 4:28 PM Sunil Govindan  wrote:
>
> > Congratulations Mukund!!!
> >
> > Thanks
> > Sunil
> >
> > On Wed, Feb 8, 2023 at 1:48 AM Vinayakumar B 
> > wrote:
> >
> > > On behalf of the Apache Hadoop PMC, I am pleased to announce that
> Mukund
> > > Thakur (mthakur) has accepted the PMC's invitation to become a PMC
> member
> > > on
> > > the project. We appreciate all of Mukund’s generous contributions thus
> > far
> > > and look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Mukund !
> > >
> > > -Vinayakumar B
> > > --
> > > -Vinay
> > >
> >
>

[jira] [Created] (HDFS-16801) TestObserverNode failing intermittently

[Ashutosh Gupta (Jira)]

Ashutosh Gupta created HDFS-16801:
-

 Summary: TestObserverNode failing intermittently
 Key: HDFS-16801
 URL: https://issues.apache.org/jira/browse/HDFS-16801
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namanode, test
Reporter: Ashutosh Gupta


TestObserverNode failing intermittently

 
{code:java}
[ERROR] 
testMkdirsRaceWithObserverRead(org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode)
  Time elapsed: 311.199 s  <<< ERROR!
java.net.ConnectException: Call From 75e69096caae/172.17.0.2 to localhost:12852 
failed on connection exception: java.net.ConnectException: Connection refused; 
For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.GeneratedConstructorAccessor118.newInstance(Unknown 
Source)
at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:948)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:863)
at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1649)
at org.apache.hadoop.ipc.Client.call(Client.java:1590)
at org.apache.hadoop.ipc.Client.call(Client.java:1487)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:258)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine2$Invoker.invoke(ProtobufRpcEngine2.java:139)
at com.sun.proxy.$Proxy31.mkdirs(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:677)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider$ObserverReadInvocationHandler.invoke(ObserverReadProxyProvider.java:518)
at com.sun.proxy.$Proxy32.mkdirs(Unknown Source)
at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:437)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeMethod(RetryInvocationHandler.java:170)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invoke(RetryInvocationHandler.java:162)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler$Call.invokeOnce(RetryInvocationHandler.java:100)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:366)
at com.sun.proxy.$Proxy32.mkdirs(Unknown Source)
at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2558)
at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2534)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1484)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1481)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1498)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.mkdir(DistributedFileSystem.java:1457)
at 
org.apache.hadoop.hdfs.server.namenode.ha.TestObserverNode.testMkdirsRaceWithObserverRead(TestObserverNode.java:561)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
 

[jira] [Created] (HDFS-16792) Add -newEditsOnly option to name node initializeSharedEdits command to format unformatted journal nodes for master replacement

[Ashutosh Gupta (Jira)]

Ashutosh Gupta created HDFS-16792:
-

 Summary: Add -newEditsOnly option to name node 
initializeSharedEdits command to format unformatted journal nodes for master 
replacement
 Key: HDFS-16792
 URL: https://issues.apache.org/jira/browse/HDFS-16792
 Project: Hadoop HDFS
  Issue Type: New Feature
  Components: journal-node
Affects Versions: 3.3.4, 3.3.3
Reporter: Ashutosh Gupta
Assignee: Ashutosh Gupta


Add -newEditsOnly option to name node initializeSharedEdits command to format 
unformatted journal nodes for master replacement.

 

The current hadoop has limitations on formatting journal nodes. First, it 
requires all journal nodes to be present when formatting name node. After that, 
new journal nodes could not be formatted unless we reformat all of them and 
destroy all HDFS edit logs.

 

$ hdfs namenode -initializeSharedEdits -newEditsOnly

There are two cases here:

1) the replaced master instance has name node running on it, then the above 
command will only be run once on the existing name node.

2) the replaced master instance does not have a name node running on it, then 
the above command will be run twice. But the above command is idempotent as 
long as the same command will not be issued at the same time, which could be 
guaranteed by instance controller when reconfiguring master instances. The 
reason is that all hadoop name node commands such as format, bootstrap, and 
initializeSharedEdits are not designed to be thread safe and they rely on an 
external mechanism, which is a manual work for hadoop admin for on-premise 
clusters, to make sure that they are not executed concurrently. We still keep 
the same behavior of the initializeSharedEdits command to minimize the 
code/logic changes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16784) replace readLock with writeLock in #DatanodeAdminBackoffMonitor.scanDatanodeStorage

[Ashutosh Gupta (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Gupta resolved HDFS-16784.
---
Resolution: Not A Problem

> replace readLock with writeLock in 
> #DatanodeAdminBackoffMonitor.scanDatanodeStorage
> ---
>
> Key: HDFS-16784
> URL: https://issues.apache.org/jira/browse/HDFS-16784
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>  Labels: pull-request-available
> Attachments: image-2022-09-28-15-37-24-622.png
>
>
> In #DatanodeAdminBackoffMonitor.scanDatanodeStorage, it uses a read lock to 
> protect the function #isBlockReplicatedOk.
> But we found that the function #isBlockReplicatedOk may update the 
> #neededReconstruction under certain conditions.
> !image-2022-09-28-15-37-24-622.png!
> Should we replace the read lock with write lock?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.3.4

[Ashutosh Gupta]

+1 (non-binding)

* Builds from source looks good.
* Checksums and signatures are correct.
* Running basic HDFS commands and running simple MapReduce jobs looks good.
* Skimmed through the contents of site documentation and it looks good.

Thanks Steve for driving this release.

Ashutosh


On Wed, Aug 3, 2022 at 9:39 PM Chris Nauroth  wrote:

> +1 (binding)
>
> * Verified all checksums.
> * Verified all signatures.
> * Built from source, including native code on Linux.
> * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> -Drequire.zstd -DskipTests
> * Tests passed.
> * mvn --fail-never clean test -Pnative -Dparallel-tests
> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> * Checked dependency tree to make sure we have all of the expected library
> updates that are mentioned in the release notes.
> * mvn -o dependency:tree
>
> I saw a LibHDFS test failure, but I know it's something flaky that's
> already tracked in a JIRA issue. The release looks good. Steve, thank you
> for driving this.
>
> Chris Nauroth
>
>
> On Wed, Aug 3, 2022 at 11:27 AM Steve Loughran  >
> wrote:
>
> > my vote for this is +1, binding.
> >
> > obviously I`m biased, but i do not want to have to issue any more interim
> > releases before the feature release off branch-3.3, so I am trying to be
> > ruthless.
> >
> > my client vaidator ant project has a more targets to help with releasing,
> > and now builds a lot mor of my local projects
> > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > all good as far as my test coverage goes, with these projects validating
> > the staged dependencies.
> >
> > now, who else can review
> >
> > On Fri, 29 Jul 2022 at 19:47, Steve Loughran 
> wrote:
> >
> > >
> > >
> > > I have put together a release candidate (RC1) for Hadoop 3.3.4
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
> > >
> > > The git tag is release-3.3.4-RC1, commit a585a73c3e0
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1358/
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md
> > >
> > > There's a very small number of changes, primarily critical
> code/packaging
> > > issues and security fixes.
> > >
> > > See the release notes for details.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > steve
> > >
> >
>

Re: Is there a 3.3.4 release planned?

[Ashutosh Gupta]

Hi Lei

Releasing 3.3.4 is already in process.

- Ashutosh

On Mon, 25 Jul, 2022, 4:37 pm Lei Zhang,  wrote:

> Hi,
>
> Is there a 3.3.4 release planned?
>
> Lei Zhang
>

Re: [VOTE] Release Apache Hadoop 3.2.4 - RC0

[Ashutosh Gupta]

+1(non-binding)

* Builds from source look good.
* Checksums and signatures are correct.
* Running basic HDFS and MapReduce commands looks good.

> * TestAMRMProxy - Not able to reproduce in local
> * TestFsck - I can see failure only I can see is
 TestFsck.testFsckListCorruptSnapshotFiles which passed after applying
HDFS-15038
> * TestSLSStreamAMSynth - Not able to reproduce in local
> * TestServiceAM - Not able to reproduce in local

Thanks Masatake for driving this release.

On Thu, Jul 21, 2022 at 5:51 AM Masatake Iwasaki 
wrote:

> Hi developers,
>
> I'm still waiting for your vote.
> I'm considering the intermittent test failures mentioned by Chris are not
> blocker.
> Please file a JIRA and let me know if you find a blocker issue.
>
> I will appreciate your help for the release process.
>
> Regards,
> Masatake Iwasaki
>
> On 2022/07/20 14:50, Masatake Iwasaki wrote:
> >> TestServiceAM
> >
> > I can see the reported failure of TestServiceAM in some "Apache Hadoop
> qbt Report: branch-3.2+JDK8 on Linux/x86_64".
> > 3.3.0 and above might be fixed by YARN-8867 which added guard using
> GenericTestUtils#waitFor for stabilizing the
> testContainersReleasedWhenPreLaunchFails.
> > YARN 8867 did not modified other code under hadoop-yarn-services.
> > If it is the case, TestServiceAM can be tagged as flaky in branch-3.2.
> >
> >
> > On 2022/07/20 14:21, Masatake Iwasaki wrote:
> >> Thanks for testing the RC0, Chris.
> >>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>
> >> I could not reproduce the test failures on my local.
> >>
> >> For TestFsck, if the failed test case is
> testFsckListCorruptSnapshotFiles,
> >> cherry-picking HDFS-15038 (fixing only test code) could be the fix.
> >>
> >> The failure of TestSLSStreamAMSynth looks frequently reported by
> >> "Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64".
> >> It could be tagged as known flaky test.
> >>
> >> On 2022/07/20 9:15, Chris Nauroth wrote:
> >>> -0 (binding)
> >>>
> >>> * Verified all checksums.
> >>> * Verified all signatures.
> >>> * Built from source, including native code on Linux.
> >>>  * mvn clean package -Pnative -Psrc -Drequire.openssl
> -Drequire.snappy
> >>> -Drequire.zstd -DskipTests
> >>> * Tests mostly passed, but see below.
> >>>  * mvn --fail-never clean test -Pnative -Dparallel-tests
> >>> -Drequire.snappy -Drequire.zstd -Drequire.openssl
> >>> -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> >>>
> >>> The following are new test failures for me on 3.2.4:
> >>> * TestAMRMProxy
> >>> * TestFsck
> >>> * TestSLSStreamAMSynth
> >>> * TestServiceAM
> >>>
> >>> The following tests also failed, but they also fail for me on 3.2.3, so
> >>> they aren't likely to be related to this release candidate:
> >>> * TestCapacitySchedulerNodeLabelUpdate
> >>> * TestFrameworkUploader
> >>> * TestSLSGenericSynth
> >>> * TestSLSRunner
> >>> * test_libhdfs_threaded_hdfspp_test_shim_static
> >>>
> >>> I'm not voting a full -1, because I haven't done any root cause
> analysis on
> >>> these new test failures. I don't know if it's a quirk to my
> environment,
> >>> though I'm using the start-build-env.sh Docker container, so any build
> >>> dependencies should be consistent. I'd be comfortable moving ahead if
> >>> others are seeing these tests pass.
> >>>
> >>> Chris Nauroth
> >>>
> >>>
> >>> On Thu, Jul 14, 2022 at 7:57 AM Masatake Iwasaki <
> iwasak...@oss.nttdata.com>
> >>> wrote:
> >>>
>  +1 from myself.
> 
>  * skimmed the contents of site documentation.
> 
>  * built the source tarball on Rocky Linux 8 (x86_64) by OpenJDK 8 with
>  `-Pnative`.
> 
>  * launched pseudo distributed cluster including kms and httpfs with
>  Kerberos and SSL enabled.
> 
>  * created encryption zone, put and read files via httpfs.
>  * ran example MR wordcount over encryption zone.
> 
>  * launched 3-node docker cluster with NN-HA and RM-HA enabled and ran
> some
>  example MR jobs.
> 
>  * built HBase 2.4.11, Hive 3.1.2 and Spark 3.1.2 against Hadoop 3.2.4
> RC0
>  on CentOS 7 (x86_64) by using Bigtop branch-3.1 and ran
> smoke-tests.
>  https://github.com/apache/bigtop/pull/942
> 
>  * Hive needs updating exclusion rule to address HADOOP-18088
> (migration
>  to reload4j).
> 
>  * built Spark 3.3.0 against Hadoop 3.2.4 RC0 using the staging
> repository::
> 
>    
>   staged
>   staged-releases
>   
> 
> https://repository.apache.org/content/repositories/orgapachehadoop-1354
>  
>   
> true
>   
>   
> true
>   
> 
> 
>  Thanks,
>  Masatake Iwasaki
> 
>  On 2022/07/13 1:14, Masatake Iwasaki wrote:
> > Hi all,
> >
> > Here's Hadoop 3.2.4 release candida

[jira] [Reopened] (HDFS-16256) Minor fixes in HDFS Fedbalance document

[Ashutosh Gupta (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Gupta reopened HDFS-16256:
---

> Minor fixes in HDFS Fedbalance document
> ---
>
> Key: HDFS-16256
> URL: https://issues.apache.org/jira/browse/HDFS-16256
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Akira Ajisaka
>    Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: newbie, pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 1. "Command submit has 4 options:" is not true. Now it has actually 6 
> options. It should be updated to something like "Command submit has the 
> following options".
> 2. 
> {code}
> ### Configuration Options
> 
> {code}
> In the above code, the "" is not needed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-16256) Minor fixes in HDFS Fedbalance document

[Ashutosh Gupta (Jira)]

 [ 
https://issues.apache.org/jira/browse/HDFS-16256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashutosh Gupta resolved HDFS-16256.
---
Resolution: Fixed

> Minor fixes in HDFS Fedbalance document
> ---
>
> Key: HDFS-16256
> URL: https://issues.apache.org/jira/browse/HDFS-16256
> Project: Hadoop HDFS
>  Issue Type: Sub-task
>  Components: documentation
>Reporter: Akira Ajisaka
>    Assignee: Ashutosh Gupta
>Priority: Minor
>  Labels: newbie, pull-request-available
> Fix For: 3.4.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> 1. "Command submit has 4 options:" is not true. Now it has actually 6 
> options. It should be updated to something like "Command submit has the 
> following options".
> 2. 
> {code}
> ### Configuration Options
> 
> {code}
> In the above code, the "" is not needed.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0

[Ashutosh Gupta]

+1(non-binding)

* Builds from source look good.
* Checksums and signatures are correct.
* Running basic HDFS and MapReduce commands looks good.

Thanks Masatake.

Thanks & Regards,
Ashutosh


On Fri, May 27, 2022 at 2:15 PM Masatake Iwasaki <
iwasak...@oss.nttdata.co.jp> wrote:

> +1 from myself.
>
> * launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
>* created encryption zone, put and read files via httpfs.
>* ran example MR wordcount over encryption zone.
>
> * skimmed the contents of site documentation.
>
> * built HBase 1.5.0, Hive 2.3.6 and Spark 2.4.5 against Hadoop 2.10.2 RC0
>on CentOS 7 (x86_64) by using Bigtop branch-1.5 and ran smoke-tests.
>https://github.com/apache/bigtop/pull/906
>
> Masatake Iwasaki
>
> On 2022/05/25 11:40, Masatake Iwasaki wrote:
> > Hi all,
> >
> > Here's Hadoop 2.10.2 release candidate #0:
> >
> > The RC is available at:
> >https://home.apache.org/~iwasakims/hadoop-2.10.2-RC0/
> >
> > The RC tag is at:
> >https://github.com/apache/hadoop/releases/tag/release-2.10.2-RC0
> >
> > The Maven artifacts are staged at:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1350
> >
> > You can find my public key at:
> >https://downloads.apache.org/hadoop/common/KEYS
> >
> > Please evaluate the RC and vote.
> > The vote will be open for (at least) 5 days.
> >
> > Thanks,
> > Masatake Iwasaki
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDFS-16576) Remove unused Imports in Hadoop HDFS project

[Ashutosh Gupta (Jira)]

Ashutosh Gupta created HDFS-16576:
-

 Summary: Remove unused Imports in Hadoop HDFS project
 Key: HDFS-16576
 URL: https://issues.apache.org/jira/browse/HDFS-16576
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Ashutosh Gupta


h3. Optimize Imports to keep code clean
 # Remove any unused imports



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-16410) Insecure Xml parsing in Java

[Ashutosh Gupta (Jira)]

Ashutosh Gupta created HDFS-16410:
-

 Summary: Insecure Xml parsing in Java 
 Key: HDFS-16410
 URL: https://issues.apache.org/jira/browse/HDFS-16410
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.3.1
Reporter: Ashutosh Gupta


Insecure Xml parsing in Java 

https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/offlineEditsViewer/OfflineEditsXmlLoader.java#L88



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org