Re: [DISCUSS] Hadoop 2019 Release Planning

[Steve Loughran]

I'm thinking of doing a backport of most of the hadoop-aws changes to
branch-3.2, for the next 3.2.x release; they are all self contained and
will benefit many (they will need to cope with the older mockito version,
but I have to deal with that in-house already).

one change is the new openFile() builder API. I'd like to wrap that up with
a little improvement https://issues.apache.org/jira/browse/HADOOP-16759;
That way for all releases with the API, it's consistent.

(that withStatus() feature gives extra performance and ensures that
etag/version can be used to get the explicit version you want.)

On Tue, Jan 7, 2020 at 2:18 AM Akira Ajisaka  wrote:

> >  I am interested on 3.3 release ..will act as RM .will update the wiki as
> well..
>
> Thanks Brahma for your reply. I'll help you as co-RM.
> We will send announcements (cutting branches, code freeze, and so on) in
> another thread.
>
> Thanks,
> Akira
>
> On Tue, Jan 7, 2020 at 4:32 AM Wangda Tan  wrote:
>
> > Hi guys,
> >
> > Thanks for the update and for volunteering to be RM.
> >
> > I just did a quick check:
> > 3.1.4 has 52 patches resolved. (3.1.3 Released on Oct 21)
> > 3.2.2 has 46 patches resolved. (3.2.1 Released on Sep 22)
> > 3.3.0 has .. many patches sitting here so we definitely need a release.
> >
> > If Akira and Brahma you guys can be co-RMs for 3.3.0 that would be great.
> >
> > Hadoop 3.2.1 is released on Sep 22 which is 3+ months ago, and I saw
> > community started to have large prod deployment on 3.2.x, Gabor if you
> have
> > bandwidth to help releases, I think we can do 3.2.2 first then 3.1.4.
> >
> > Thoughts?
> > - Wangda
> >
> > On Mon, Jan 6, 2020 at 5:50 AM Brahma Reddy Battula 
> > wrote:
> >
> >> Thanks Akira for resuming this..
> >>
> >>  I am interested on 3.3 release ..will act as RM .will update the wiki
> as
> >> well..
> >>
> >>
> >>
> >> On Mon, 6 Jan 2020 at 6:08 PM, Gabor Bota  .invalid>
> >> wrote:
> >>
> >>> I'm interested in doing a release of hadoop.
> >>> The version we need an RM is 3.1.3 right? What's the target date for
> >>> that?
> >>>
> >>> Thanks,
> >>> Gabor
> >>>
> >>> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka 
> >>> wrote:
> >>>
> >>> > Thank you Wangda.
> >>> >
> >>> > Now it's 2020. Let's release Hadoop 3.3.0.
> >>> > I created a wiki page for tracking blocker/critical issues for 3.3.0
> >>> and
> >>> > I'll check the issues in the list.
> >>> >
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
> >>> > If you find blocker/critical issues in trunk, please set the target
> >>> version
> >>> > to 3.3.0 for tracking.
> >>> >
> >>> > > We still need RM for 3.3.0 and 3.1.3.
> >>> > I can work as a release manager for 3.3.0. Is there anyone who wants
> >>> to be
> >>> > a RM?
> >>> >
> >>> > Thanks and regards,
> >>> > Akira
> >>> >
> >>> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang 
> >>> > wrote:
> >>> >
> >>> > > Thanks Wangda for bring this up!
> >>> > >
> >>> > > I ran the submarine 0.2.0 release before with a lot of help from
> >>> folks
> >>> > > especially Sunil. :D
> >>> > > And this time I would like to help to release the 3.1.4. Thanks!
> >>> > >
> >>> > > BR,
> >>> > > Zhankun
> >>> > >
> >>> > > Hui Fei 于2019年8月16日 周五下午7:19写道：
> >>> > >
> >>> > > > Hi Wangda,
> >>> > > > Thanks for bringing this up!
> >>> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade
> >>> is a
> >>> > > > problem.
> >>> > > > Hope commiters watch and review these issues, Thanks
> >>> > > > https://issues.apache.org/jira/browse/HDFS-13596
> >>> > > > https://issues.apache.org/jira/browse/HDFS-14396
> >>> > > >
> >>> > > > Wangda Tan  于2019年8月10日周六 上午10:59写道：
> >>> > > >
> >>> > > > > Hi all,
> >>> > > > >
> >>> > > > > Hope this email finds you well
> >>> > > > >
> >>> > > > > I want to hear your thoughts about what should be the release
> >>> plan
> >>> > for
> >>> > > > > 2019.
> >>> > > > >
> >>> > > > > In 2018, we released:
> >>> > > > > - 1 maintenance release of 2.6
> >>> > > > > - 3 maintenance releases of 2.7
> >>> > > > > - 3 maintenance releases of 2.8
> >>> > > > > - 3 releases of 2.9
> >>> > > > > - 4 releases of 3.0
> >>> > > > > - 2 releases of 3.1
> >>> > > > >
> >>> > > > > Total 16 releases in 2018.
> >>> > > > >
> >>> > > > > In 2019, by far we only have two releases:
> >>> > > > > - 1 maintenance release of 3.1
> >>> > > > > - 1 minor release of 3.2.
> >>> > > > >
> >>> > > > > However, the community put a lot of efforts to stabilize
> >>> features of
> >>> > > > > various release branches.
> >>> > > > > There're:
> >>> > > > > - 217 fixed patches in 3.1.3 [1]
> >>> > > > > - 388 fixed patches in 3.2.1 [2]
> >>> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> >>> > > > >
> >>> > > > > I think it is the time to do maintenance releases of 3.1/3.2
> and
> >>> do a
> >>> > > > minor
> >>> > > > > release for 3.3.0.
> >>> > > > >
> >>> > > > > In addition, I saw community discussion to do a 2.8.6 release
> for
> >>> > > > security
> >>> > > > > fixes.
> >>

Re: Hadoop 3.3 Release Plan Proposal

[Steve Loughran]

>
> 2. Features close to finish:
>
>
>   *HADOOP-15620: Über-jira: S3A phase VI: Hadoop 3.3 features. ( owner
> :  Steve Loughran)
>   *HADOOP-15763: Über-JIRA: abfs phase II: Hadoop 3.3 features &
> fixes. ( owner :  Steve Loughran)
>   *HADOOP-15619:Über-JIRA: S3Guard Phase IV: Hadoop 3.3 features. (
> owner :  Steve Loughran)
>
> I'll move out anything that isn't needed.

FWIW, most of these are in CDP 1.x, so there's been reasonable testing and
I've got some provisional tuning to do. That is -if things didn't work in
the test/production deployments, I'd know about the regressions (e.g.
HADOOP-16751).

This is S3A and ABFS code -no idea about the rest, and inevitably the big
JAR changes will have surprises. We need to fix the shaded protobuf in
Token issue to even get spark to compile.

-Steve

>
>

Re: [DISCUSS] Hadoop 2019 Release Planning

[Steve Loughran]

Well volunteered! I will help with the testing

On Mon, Jan 6, 2020 at 10:08 AM Gabor Bota 
wrote:

> I'm interested in doing a release of hadoop.
> The version we need an RM is 3.1.3 right? What's the target date for that?
>
> Thanks,
> Gabor
>
> On Mon, Jan 6, 2020 at 8:31 AM Akira Ajisaka  wrote:
>
> > Thank you Wangda.
> >
> > Now it's 2020. Let's release Hadoop 3.3.0.
> > I created a wiki page for tracking blocker/critical issues for 3.3.0 and
> > I'll check the issues in the list.
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.3+Release
> > If you find blocker/critical issues in trunk, please set the target
> version
> > to 3.3.0 for tracking.
> >
> > > We still need RM for 3.3.0 and 3.1.3.
> > I can work as a release manager for 3.3.0. Is there anyone who wants to
> be
> > a RM?
> >
> > Thanks and regards,
> > Akira
> >
> > On Fri, Aug 16, 2019 at 9:28 PM zhankun tang 
> > wrote:
> >
> > > Thanks Wangda for bring this up!
> > >
> > > I ran the submarine 0.2.0 release before with a lot of help from folks
> > > especially Sunil. :D
> > > And this time I would like to help to release the 3.1.4. Thanks!
> > >
> > > BR,
> > > Zhankun
> > >
> > > Hui Fei 于2019年8月16日 周五下午7:19写道：
> > >
> > > > Hi Wangda,
> > > > Thanks for bringing this up!
> > > > Looking forward to see HDFS 3.x is widely used，but RollingUpgrade is
> a
> > > > problem.
> > > > Hope commiters watch and review these issues, Thanks
> > > > https://issues.apache.org/jira/browse/HDFS-13596
> > > > https://issues.apache.org/jira/browse/HDFS-14396
> > > >
> > > > Wangda Tan  于2019年8月10日周六 上午10:59写道：
> > > >
> > > > > Hi all,
> > > > >
> > > > > Hope this email finds you well
> > > > >
> > > > > I want to hear your thoughts about what should be the release plan
> > for
> > > > > 2019.
> > > > >
> > > > > In 2018, we released:
> > > > > - 1 maintenance release of 2.6
> > > > > - 3 maintenance releases of 2.7
> > > > > - 3 maintenance releases of 2.8
> > > > > - 3 releases of 2.9
> > > > > - 4 releases of 3.0
> > > > > - 2 releases of 3.1
> > > > >
> > > > > Total 16 releases in 2018.
> > > > >
> > > > > In 2019, by far we only have two releases:
> > > > > - 1 maintenance release of 3.1
> > > > > - 1 minor release of 3.2.
> > > > >
> > > > > However, the community put a lot of efforts to stabilize features
> of
> > > > > various release branches.
> > > > > There're:
> > > > > - 217 fixed patches in 3.1.3 [1]
> > > > > - 388 fixed patches in 3.2.1 [2]
> > > > > - 1172 fixed patches in 3.3.0 [3] (OMG!)
> > > > >
> > > > > I think it is the time to do maintenance releases of 3.1/3.2 and
> do a
> > > > minor
> > > > > release for 3.3.0.
> > > > >
> > > > > In addition, I saw community discussion to do a 2.8.6 release for
> > > > security
> > > > > fixes.
> > > > >
> > > > > Any other releases? I think there're release plans for Ozone as
> well.
> > > And
> > > > > please add your thoughts.
> > > > >
> > > > > Volunteers welcome! If you have interests to run a release as
> Release
> > > > > Manager (or co-Resource Manager), please respond to this email
> thread
> > > so
> > > > we
> > > > > can coordinate.
> > > > >
> > > > > Thanks,
> > > > > Wangda Tan
> > > > >
> > > > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.1.3
> > > > > [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.2.1
> > > > > [3] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND resolution =
> Fixed
> > > AND
> > > > > fixVersion = 3.3.0
> > > > >
> > > >
> > >
> >
>

HDFS-13616 : batch listing of multiple directories

[Steve Loughran]

https://issues.apache.org/jira/browse/HDFS-13616

I don't want to be territorial here -but as I keep reminding this list
whenever it happens,  -I do not want any changes to go into the core
FileSystem class without

* raising a HADOOP- JIRA
* involving those of us who work on object stores. We have different
problems (latencies, failure modes) and want to move to move
async/completable APIs, ideally with builder APIs for future flexibility
and per-FS options.
* specify semantics formally enough that people implementing and using know
what they get.
* a specification in the filesystem.md
* contract tests to match the spec and which object stores can implement,
as well as HDFS

The change has ~no javadocs and doesn't even state
* whether it's recursive or not.
* whether it includes directories or not

batchedListStatusIterator is exactly the kind of feature this should apply
to -it is where we get a chance to fix those limitations of the previous
calls (blocking sync, no expectation of right to cancel listings), ...

I'd like to be able to
* provide a hint on batch sizes.
* get an async response so the fact the LIST can can take time is more
visible.
* and let us cancel that query if it is taking too long

I also like to be able to close an iterator too; that is something we
can/should retrofit, or require all implementations to add


Completable> listing =
  batchList(Path)
   .recursive(true)
   .opt("fs.option.batchlist.size", 100)
   .build()

RemoteIterator it = listing.get()

FileStatus largeFile = null;

try {
  while(it.hasNext()) {
FileStatus st = it.next();
if (st.length()> 1_000_000) {
  largeFile = st;
  break;
}
  } finally {
if (it instanceof Closeable) {
  IOUtils.closeQuietly((Closeable)it);
}
  }

  if (largeFile != null) {
processLargeFile(largeFile);
  }
}

See: something for slower IO, controllable batch sizes and a way to cancel
the scan -so let us recycle the HTTP connection even when breaking out
early.

This is a recurrent problem and I am getting as bored as a sending these
emails out as people probably are at receiving them.

Please please at least talk to me. Yes I'm going to add more homework but
the goal is to make it something well documented well testable and
straightforward to implement by other implementations without us having to
reverse engineer HDFS's behaviour and consider that a normative

What I do here?
1. Do I overreact and revert the change until my needs are met? Because I
know that if I volunteered to do this work myself it's going to get
neglected.
2. Is someone going to put their hand up to help this?

At the very least, I'm going to tag the APIs as unstable and potentially
likely to break so that anyone who uses it in hadoop-3.3.0 isn't going to
be upset when it is moved to a builder API. And it will have to  for the
objects stores.

sorry

steve

Re: HDFS-13616 : batch listing of multiple directories

[Steve Loughran]

On Sat, 29 Feb 2020 at 00:23, Wei-Chiu Chuang 
wrote:

> Steve,
>
> You made a great point and I'm sorry this API was implemented without
> consideration of other FS implementation.
> Thank you for your direct feedback.
>
> async -- yes
> builder -- yes
> cancellable -- totally agree
>
> There are good use cases for this API though -- Impala and Presto both
> require lots of file system metadata operation, and this API would make
> them much more efficient.
>

Well, I absolutely do not want an API we will have to maintain for decades
yet lacks a rigorous specification other than "look at the HDFS
implementation" and has seemingly not taken the needs of cloud storage into
account.

It is the long-term obligation to maintain this API which I am most worried
about. Please: don't add new operations there unless you think it ready for
broad use and that, in the absence of any builder API, is "the perfect
operation"

Proposed:

* the new API is pulled into a new interface marked unstable; FileSystems
subclasses get to implement if they support it.
* new classes (PartialListing) also tagged unstable. Side issue, please,
always mark new stuff as Evolving.
* and that interface extends PathCapabilities, so you can't implement it
without declaring whether paths support the feature.
* And we will define a new path capability.

Applications can cast to the new interface, and use PathCapabilities to
verify it is actually available under a given path, even through filter
filesystems


Then, after 3.3.0 is out, someone gets to do the FileSystem implementation,
with specification, tests etc. Not me -you are the HDFS team -of course you
can do this. But it does need to be done taking into account the fact that
alternate stores and far systems will be wanting to implement this and all
others will be fielding support calls related to it for a long time.


> On top of that, I would also like to have a batched delete API. HBase could
> benefit a lot from that.
>
>
Another interesting bit of work, especially since Gabor and I have just
been dealing with S3 delete throttling issues.

I will gladly give advice there. at the very least, it must implement
Progressable, so when deletes are very slow processes/threads can still
send heartbeats back. It also sets expectations up as to how long some of
these things can take.

-Steve


>
>
> On Fri, Feb 28, 2020 at 5:48 AM Steve Loughran  >
> wrote:
>
> > https://issues.apache.org/jira/browse/HDFS-13616
> >
> > I don't want to be territorial here -but as I keep reminding this list
> > whenever it happens,  -I do not want any changes to go into the core
> > FileSystem class without
> >
> > * raising a HADOOP- JIRA
> > * involving those of us who work on object stores. We have different
> > problems (latencies, failure modes) and want to move to move
> > async/completable APIs, ideally with builder APIs for future flexibility
> > and per-FS options.
> > * specify semantics formally enough that people implementing and using
> know
> > what they get.
> > * a specification in the filesystem.md
> > * contract tests to match the spec and which object stores can implement,
> > as well as HDFS
> >
> > The change has ~no javadocs and doesn't even state
> > * whether it's recursive or not.
> > * whether it includes directories or not
> >
> > batchedListStatusIterator is exactly the kind of feature this should
> apply
> > to -it is where we get a chance to fix those limitations of the previous
> > calls (blocking sync, no expectation of right to cancel listings), ...
> >
> > I'd like to be able to
> > * provide a hint on batch sizes.
> > * get an async response so the fact the LIST can can take time is more
> > visible.
> > * and let us cancel that query if it is taking too long
> >
> > I also like to be able to close an iterator too; that is something we
> > can/should retrofit, or require all implementations to add
> >
> >
> > Completable> listing
> =
> >   batchList(Path)
> >.recursive(true)
> >.opt("fs.option.batchlist.size", 100)
> >.build()
> >
> > RemoteIterator it = listing.get()
> >
> > FileStatus largeFile = null;
> >
> > try {
> >   while(it.hasNext()) {
> > FileStatus st = it.next();
> > if (st.length()> 1_000_000) {
> >   largeFile = st;
> >   break;
> > }
> >   } finally {
> > if (it instanceof Closeable) {
> >   IOUtils.closeQuietly((Closeable)it);
> > }
> >   }
> >
> >   if (largeFile != null) {
> > processLargeFile(largeFile);
> 

asf git repo sync

[Steve Loughran]

FYI, looks like the ASF git repo @
https://gitbox.apache.org/repos/asf/hadoop.git and the github repo at
https://github.com/apache/hadoop.git have lost sync;

I've merged a couple of changes in via the github UI and they aren't
surfacing in the asf repo, which is still @ commit 60fa15366e8,  YARN-10240.

The asf internal one is the store of record, and its where changes should
go until things are syncing again; we'll have to deal with the merge
problems when they occur. There's a risk the resync may be lossy.

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Steve Loughran]

Okay.

I am not going to be a purist and say "what were they doing -using our
private APIs?" because as we all know, with things like UGI tagged @private
there's been no way to get something is done without getting into the
private stuff.

But why did we do the protobuf changes? So that we could update our private
copy of protobuf with out breaking every single downstream application. The
great protobuf upgrade to 2.5 is not something we wanted to repeat. When
was that? before hadoop-2.2 shipped? I certainly remember a couple of weeks
were absolutely nothing would build whatsoever, not until every downstream
project had upgraded to the same version of the library.

If you ever want to see an upgrade which makes a guava update seem a minor
detail, protobuf upgrades are it. Hence the shading

HBase
=

it looks like HBase has been using deep internal stuff. That is,
"unfortunate". I think in that world we have to look and say is there
something specific we can do here to help HBase in a way we could also
backport. They shouldn't need those IPC internals.

Tez & Tokens


I didn't know Tez was using those protobuf APIs internally. That is,
"unfortunate".

What is key is this: without us moving those methods things like Spark
wouldn't work. And they weren't even using the methods, just trying to work
with Token for job submission.

All Tez should need is a byte array serialization of a token. Given Token
is also Writable, that could be done via WritableUtils in a way which will
also work with older releases.

Ozone
=

When these were part of/in-sync with the hadoop build there wouldn't have
been problems. Now there are. Again, they're going in deep, but here
clearly to simulate some behaviour. Any way to do that differently?

Ratis
=

No idea.

On Wed, 29 Apr 2020 at 07:12, Wei-Chiu Chuang 
wrote:

> Most of the problems are downstream applications using Hadoop's private
> APIs.
>
> Tez:
>
> 17:08:38 2020/04/16 00:08:38 INFO: [ERROR] COMPILATION ERROR :
> 17:08:38 2020/04/16 00:08:38 INFO: [INFO]
> -
> 17:08:38 2020/04/16 00:08:38 INFO: [ERROR]
>
> /grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/tez/tez-plugins/tez-aux-services/src/main/java/org/apache/tez/auxservices/ShuffleHandler.java:[757,45]
> incompatible types: com.google.protobuf.ByteString cannot be converted
> to org.apache.hadoop.thirdparty.protobuf.ByteString
> 17:08:38 2020/04/16 00:08:38 INFO: [INFO] 1 error
>
>
> Tez keeps track of job tokens internally.
> The change would look like this:
>
> private void recordJobShuffleInfo(JobID jobId, String user,
> Token jobToken) throws IOException {
>   if (stateDb != null) {
> TokenProto tokenProto = ProtobufHelper.protoFromToken(jobToken);
> /*TokenProto tokenProto = TokenProto.newBuilder()
> .setIdentifier(ByteString.copyFrom(jobToken.getIdentifier()))
> .setPassword(ByteString.copyFrom(jobToken.getPassword()))
> .setKind(jobToken.getKind().toString())
> .setService(jobToken.getService().toString())
> .build();*/
> JobShuffleInfoProto proto = JobShuffleInfoProto.newBuilder()
> .setUser(user).setJobToken(tokenProto).build();
> try {
>   stateDb.put(bytes(jobId.toString()), proto.toByteArray());
> } catch (DBException e) {
>   throw new IOException("Error storing " + jobId, e);
> }
>   }
>   addJobToken(jobId, user, jobToken);
> }
>
>
> HBase:
>
>1. HBASE-23833 
> (this
>is recently fixed in the master branch)
>2.
>
>   [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> (default-compile) on project hbase-server: Compilation failure:
> Compilation failure:
>   [ERROR]
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[361,44]
> cannot access org.apache.hadoop.thirdparty.protobuf.MessageOrBuilder
>   [ERROR]   class file for
> org.apache.hadoop.thirdparty.protobuf.MessageOrBuilder not found
>   [ERROR]
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[362,14]
> cannot access org.apache.hadoop.thirdparty.protobuf.GeneratedMessageV3
>   [ERROR]   class file for
> org.apache.hadoop.thirdparty.protobuf.GeneratedMessageV3 not found
>   [ERROR]
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[366,16]
> cannot access org.apache.hadoop.thirdparty.protobuf.ByteString
>   [ERROR]   class file for
> org.apache.hadoop.thirdparty.protobuf.ByteString not found
>   [ERROR]
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutputSaslHelper.java:[375,12]
> cann

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC2)

[Steve Loughran]

+1, with the instruction "warn everyone about the guava update possibly
breaking things at run time"

With the key issues being
* code compiled with the new guava release will not link against the older
releases, even without any changes in the source files.
* this includes hadoop-common

Applications which exclude the guava dependency published by hadoop-
artifacts to use their own, must set guava.version=27.0-jre or
guava.version=27.0 to be consistent with that of this release.


My tests were all with using the artifacts downstream via maven; I trust
others to look at the big tarball release.


*Project 1: cloudstore*


This is my extra diagnostics and cloud utils module.
https://github.com/steveloughran/cloudstore


All compiled fine, but the tests failed on guava linkage

testNoOverwriteDest(org.apache.hadoop.tools.cloudup.TestLocalCloudup)  Time
elapsed: 0.012 sec  <<< ERROR! java.lang.NoSuchMethodError: 'void
com.google.common.base.Preconditions.checkArgument(boolean,
java.lang.String, java.lang.Object, java.lang.Object)'
at org.apache.hadoop.fs.tools.cloudup.Cloudup.run(Cloudup.java:177)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hadoop.tools.store.StoreTestUtils.exec(StoreTestUtils.java:4


Note: that app is designed to run against hadoop branch-2 and other
branches, so I ended up reimplementing the checkArgument and checkState
calls so that I can have a binary which links everywhere. My code, nothing
serious.

*Project 2: Spark*


apache spark main branch built with maven (not tried the SBT build).


mvn -T 1  -Phadoop-3.2 -Dhadoop.version=3.1.4 -Psnapshots-and-staging
-Phadoop-cloud,yarn,kinesis-asl -DskipTests clean package

All good. Then I ran the committer unit test suite

mvn -T 1  -Phadoop-3.2 -Dhadoop.version=3.1.4
-Phadoop-cloud,yarn,kinesis-as  -Psnapshots-and-staging --pl hadoop-cloud
test

CommitterBindingSuite:
*** RUN ABORTED ***
  java.lang.NoSuchMethodError: 'void
com.google.common.base.Preconditions.checkArgument(boolean,
java.lang.String, java.lang.Object)'
  at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
  at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
  at
org.apache.spark.internal.io.cloud.CommitterBindingSuite.newJob(CommitterBindingSuite.scala:89)
  at
org.apache.spark.internal.io.cloud.CommitterBindingSuite.$anonfun$new$1(CommitterBindingSuite.scala:55)
  at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
  at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
  at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104)
  at org.scalatest.Transformer.apply(Transformer.scala:22)
  at org.scalatest.Transformer.apply(Transformer.scala:20)
  at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:186)
  ...

Fix: again, tell the build this is a later version of Guava:


mvn -T 1  -Phadoop-3.2 -Dhadoop.version=3.1.4
-Phadoop-cloud,yarn,kinesis-asl  -Psnapshots-and-staging --pl hadoop-cloud
-Dguava.version=27.0-jre test


the mismatch doesn't break spark internally, they shade their stuff anyway,
the guava.version here is actually the one which hadoop is to be linked
with.

outcome: tests work

[INFO] --- scalatest-maven-plugin:2.0.0:test (test) @
spark-hadoop-cloud_2.12 ---
Discovery starting.
Discovery completed in 438 milliseconds.
Run starting. Expected test count is: 4
CommitterBindingSuite:
- BindingParquetOutputCommitter binds to the inner committer
- committer protocol can be serialized and deserialized
- local filesystem instantiation
- reject dynamic partitioning
Run completed in 1 second, 411 milliseconds.
Total number of tests run: 4
Suites: completed 2, aborted 0
Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0

This is a real PITA, and its invariably those checkArgument calls, because
the later guava versions added some overloaded methods. Compile existing
source with a later guava version and the .class no longer binds to the
older guava version, even though no new guava APIs have been adopted.

I am really tempted to go through src/**/*.java and replace all Guava
checkArgument/checkState with our own implementation in hadoop.common, at
least for any which uses the vararg variant. But: it'd be a big change and
there may be related issues elsewhere. At least now things fail fast.

*Project 3: spark cloud integration  *

https://github.com/hortonworks-spark/cloud-integration

This is where the functional tests for the s3a committer through spark live

-Dhadoop.version=3.1.2 -Dspark.version=3.1.0-SNAPSHOT
-Psnapshots-and-staging

and a full test run

mvn test -Dcloud.test.configuration.file=../test-configs/s3a.xml --pl
cloud-examples -Dhadoop.version=3.1.2 -Dspark.version=3.1.0-SNAPSHOT
-Psnapshots-and-staging

All good. A couple of test failures, but that was because one of my test
datasets is not on any bucket I have...will have to fix that.


To conclude: the artefacts are all there, existing code c

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC2)

[Steve Loughran]

hmm

YARN-9341 went through all of the yarn lock code -it's in 3.3 but not in
3.1. And we do not want to attempt to backport 175KB of lock
acquire/release code, do we?

anyone in yarn-dev got any thoughts here?

On Sun, 5 Jul 2020 at 22:14, Masatake Iwasaki 
wrote:

> Thanks for putting this up, Gabor Bota.
>
> I'm testing the RC2 on 3 node docker cluster with NN-HA and RM-HA enabled.
> ResourceManager reproducibly blocks on submitApplication while launching
> example MR jobs.
> Does anyone run into the same issue?
>
> The same configuration worked for 3.1.3.
> I got no issue if RM-HA is disabled.
>
>
> "IPC Server handler 1 on default port 8032" #167 daemon prio=5 os_prio=0
> tid=0x7fe91821ec50 nid=0x3b9 waiting on condition [0x7fe901bac000]
> java.lang.Thread.State: WAITING (parking)
>  at sun.misc.Unsafe.park(Native Method)
>  - parking to wait for  <0x85d37a40> (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
>  at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
>  at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
>  at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
>  at
>
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
>  at
>
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
>  at
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.checkAndGetApplicationPriority(CapacityScheduler.java:2521)
>  at
>
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.createAndPopulateNewRMApp(RMAppManager.java:417)
>  at
>
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager.submitApplication(RMAppManager.java:342)
>  at
>
> org.apache.hadoop.yarn.server.resourcemanager.ClientRMService.submitApplication(ClientRMService.java:678)
>  at
>
> org.apache.hadoop.yarn.api.impl.pb.service.ApplicationClientProtocolPBServiceImpl.submitApplication(ApplicationClientProtocolPBServiceImpl.java:277)
>  at
>
> org.apache.hadoop.yarn.proto.ApplicationClientProtocol$ApplicationClientProtocolService$2.callBlockingMethod(ApplicationClientProtocol.java:563)
>  at
>
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:527)
>  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1036)
>  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1015)
>  at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:943)
>  at java.security.AccessController.doPrivileged(Native Method)
>  at javax.security.auth.Subject.doAs(Subject.java:422)
>  at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1729)
>  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2943)
>
>
> Masatake Iwasaki
>
> On 2020/06/26 22:51, Gabor Bota wrote:
> > Hi folks,
> >
> > I have put together a release candidate (RC2) for Hadoop 3.1.4.
> >
> > The RC is available at:
> http://people.apache.org/~gabota/hadoop-3.1.4-RC2/
> > The RC tag in git is here:
> > https://github.com/apache/hadoop/releases/tag/release-3.1.4-RC2
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1269/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > and http://keys.gnupg.net/pks/lookup?op=get&search=0xB86249D83539B38C
> >
> > Please try the release and vote. The vote will run for 5 weekdays,
> > until July 6. 2020. 23:00 CET.
> >
> > The release includes the revert of HDFS-14941, as it caused
> > HDFS-15421. IBR leak causes standby NN to be stuck in safe mode.
> > (https://issues.apache.org/jira/browse/HDFS-15421)
> > The release includes HDFS-15323, as requested.
> > (https://issues.apache.org/jira/browse/HDFS-15323)
> >
> > Thanks,
> > Gabor
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>

Re: [Reposting] How to set md5 checksum

[Steve Loughran]

There's a broader issue here which is "S3A support for buckets with Object
Lock"

I suspect that there will be more to this than just setting an MD5 checksum
on file uploads.

AFAIK, nobody has done this/is working on the feature; there's so many
other issues than nobody is going to do it unless it is critical for them:
https://issues.apache.org/jira/browse/HADOOP-16829
Given that you are the one with this problem, I'm afraid that leaves you.

We'll take a patch with docs and tests; submission process is:
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html

steve

On Fri, 17 Jul 2020 at 21:10, nikita Balakrishnan 
wrote:

> Hey team,
>
> I’m developing a system where we are trying to sink to an immutable s3
> bucket as part of a flink job. This bucket has server side encryption set
> to KMS. The DataStream sink works perfectly fine when I don’t use the
> immutable bucket but when I use an immutable bucket, I get exceptions
> regarding multipart upload failures. It says we need to enable md5 hashing
> for the put object to work.
>
> According to was s3 documentation for immutable buckets (with object locks)
> they say it’s mandatory to have a content-md5 header -
> https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html
> “The Content-MD5 header is required for any request to upload an object
> with a retention period configured using Amazon S3 Object Lock. For more
> information about Amazon S3 Object Lock, see Amazon S3 Object Lock Overview
> 
> in
> the *Amazon Simple Storage Service Developer Guide*."
>
> My question to the link team was -  "How do I set this HTTP header while
> sinking? I checked most of the documentation and tried going through the
> source code too but couldn’t really find a provision where we could set the
> headers for a request that goes in as a sink." And they got back asking me
> to set fs.s3a.etag.checksum.enabled: true. But that didn’t work either. And
> then they redirected me to the Hadoop team. Can you please help me out
> here?
>
> The one thing I’m unclear about is how does the system know that we’re
> using md5 hashing when we enable checksum? Is there some way to specify
> that? Feels like I’m missing that.
>
> Here’s the stack trace:
>
> org.apache.flink.streaming.runtime.tasks.AsynchronousException: Caught
> exception while processing timer.
> at
>
> org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1090)
> at
>
> org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1058)
> at
>
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1520)
> at
>
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$10(StreamTask.java:1509)
> at
>
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.run(StreamTaskActionExecutor.java:87)
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
> at
>
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:261)
> at
>
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:186)
> at
>
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:487)
> at
>
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: org.apache.flink.streaming.runtime.tasks.TimerException:
> java.io.IOException: Uploading parts failed
> ... 11 common frames omitted
> Caused by: java.io.IOException: Uploading parts failed
> at
>
> org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUploadImpl.awaitPendingPartUploadToComplete(RecoverableMultiPartUploadImpl.java:231)
> at
>
> org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUploadImpl.awaitPendingPartsUpload(RecoverableMultiPartUploadImpl.java:215)
> at
>
> org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUploadImpl.snapshotAndGetRecoverable(RecoverableMultiPartUploadImpl.java:151)
> at
>
> org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUploadImpl.snapshotAndGetCommitter(RecoverableMultiPartUploadImpl.java:123)
> at
>
> org.apache.flink.fs.s3.common.writer.RecoverableMultiPartUploadImpl.snapshotAndGetCommitter(RecoverableMultiPartUploadImpl.java:56)
> at
>
> org.apache.flink.fs.s3.common.writer.S3RecoverableFsDataOutputStream.closeForCommit(S3RecoverableFsDataOutputStream.java:167)
> at
>
> org.apache.flink.streaming.api.functions.sink.filesystem.PartFileWriter.closeForCommit(PartFileWriter.java:71)
> at
>
> org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.closePartFile(Bucket.java:239)

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC4)

[Steve Loughran]

 +1

did a cloudstore clean build and test
did as well as I could with a spark build.

For anyone having maven problems building hadoop on a mac, homebrew now
forces its version of maven to use a homebrew specific openjdk 11
(one /usr/libexec/java_home doesn't locate); bits of the hadoop build don't
work if maven is running on java 11. Removing the homebrew maven fixes up
that but now the bits of the spark maven build which call out to the SBT
build tool are going out of memory. This is one of those times when I think
"I need a linux box'

Anyway: maven builds are happy, spark compiles with the branch once you
change its guava version. Well, that's progress

-Steve

Re: [VOTE] Release Apache Hadoop 3.1.4 (RC4)

[Steve Loughran]

 I should add, that's a binding +1

On Mon, 27 Jul 2020 at 19:55, Steve Loughran  wrote:

>  +1
>
> did a cloudstore clean build and test
> did as well as I could with a spark build.
>
> For anyone having maven problems building hadoop on a mac, homebrew now
> forces its version of maven to use a homebrew specific openjdk 11
> (one /usr/libexec/java_home doesn't locate); bits of the hadoop build don't
> work if maven is running on java 11. Removing the homebrew maven fixes up
> that but now the bits of the spark maven build which call out to the SBT
> build tool are going out of memory. This is one of those times when I think
> "I need a linux box'
>
> Anyway: maven builds are happy, spark compiles with the branch once you
> change its guava version. Well, that's progress
>
> -Steve
>

Re: [DISCUSS] fate of branch-2.9

[Steve Loughran]

+1

are there any Hadoop branch-2 releases planned, ever? If so I'll need to
backport my s3a directory compatibility patch to whatever is still live.


On Thu, 27 Aug 2020 at 06:55, Wei-Chiu Chuang  wrote:

> Bump up this thread after 6 months.
>
> Is anyone still interested in the 2.9 release line? Or are we good to start
> the EOL process? The 2.9.2 was released in Nov 2018.
>
> I'd really like to see the community to converge to fewer release lines and
> make more frequent releases in each line.
>
> Thanks,
> Weichiu
>
>
> On Fri, Mar 6, 2020 at 5:47 PM Wei-Chiu Chuang  wrote:
>
> > I think that's a great suggestion.
> > Currently, we make 1 minor release per year, and within each minor
> release
> > we bring up 1 thousand to 2 thousand commits in it compared with the
> > previous one.
> > I can totally understand it is a big bite for users to swallow. Having a
> > more frequent release cycle, plus LTS and non-LTS releases should help
> with
> > this. (Of course we will need to make the release preparation much
> easier,
> > which is currently a pain)
> >
> > I am happy to discuss the release model further in the dev ML. LTS v.s.
> > non-LTS is one suggestion.
> >
> > Another similar issue: In the past Hadoop strived to
> > maintain compatibility. However, this is no longer sustainable as more
> CVEs
> > coming from our dependencies: netty, jetty, jackson ... etc.
> > In many cases, updating the dependencies brings breaking changes. More
> > recently, especially in Hadoop 3.x, I started to make the effort to
> update
> > dependencies much more frequently. How do users feel about this change?
> >
> > On Thu, Mar 5, 2020 at 7:58 AM Igor Dvorzhak 
> > wrote:
> >
> >> Maybe Hadoop will benefit from adopting a similar release and support
> >> strategy as Java? I.e. designate some releases as LTS and support them
> for
> >> 2 (?) years (it seems that 2.7.x branch was de-facto LTS), other non-LTS
> >> releases will be supported for 6 months (or until next release). This
> >> should allow to reduce maintenance cost of non-LTS release and provide
> >> conservative users desired stability by allowing them to wait for new
> LTS
> >> release and upgrading to it.
> >>
> >> On Thu, Mar 5, 2020 at 1:26 AM Rupert Mazzucco <
> rupert.mazzu...@gmail.com>
> >> wrote:
> >>
> >>> After recently jumping from 2.7.7 to 2.10 without issue myself, I vote
> >>> for keeping only the 2.10 line.
> >>> It would seem all other 2.x branches can upgrade to a 2.10.x easily if
> >>> they feel like upgrading at all,
> >>> unlike a jump to 3.x, which may require more planning.
> >>>
> >>> I also vote for having only one main 3.x branch. Why are there 3.1.x
> and
> >>> 3.2.x seemingly competing,
> >>> and now 3.3.x? For a community that does not have the resources to
> >>> manage multiple release lines,
> >>> you guys sure like to multiply release lines a lot.
> >>>
> >>> Cheers
> >>> Rupert
> >>>
> >>> Am Mi., 4. März 2020 um 19:40 Uhr schrieb Wei-Chiu Chuang
> >>> :
> >>>
>  Forwarding the discussion thread from the dev mailing lists to the
> user
>  mailing lists.
> 
>  I'd like to get an idea of how many users are still on Hadoop 2.9.
>  Please share your thoughts.
> 
>  On Mon, Mar 2, 2020 at 6:30 PM Sree Vaddi
>   wrote:
> 
> > +1
> >
> > Sent from Yahoo Mail on Android
> >
> >   On Mon, Mar 2, 2020 at 5:12 PM, Wei-Chiu Chuang >
> > wrote:   Hi,
> >
> > Following the discussion to end branch-2.8, I want to start a
> > discussion
> > around what's next with branch-2.9. I am hesitant to use the word
> "end
> > of
> > life" but consider these facts:
> >
> > * 2.9.0 was released Dec 17, 2017.
> > * 2.9.2, the last 2.9.x release, went out Nov 19 2018, which is more
> > than
> > 15 months ago.
> > * no one seems to be interested in being the release manager for
> 2.9.3.
> > * Most if not all of the active Hadoop contributors are using Hadoop
> > 2.10
> > or Hadoop 3.x.
> > * We as a community do not have the cycle to manage multiple release
> > line,
> > especially since Hadoop 3.3.0 is coming out soon.
> >
> > It is perhaps the time to gradually reduce our footprint in Hadoop
> > 2.x, and
> > encourage people to upgrade to Hadoop 3.x
> >
> > Thoughts?
> >
> >
>

Re: [VOTE] End of Life Hadoop 2.9

[Steve Loughran]

+1
 binding

On Mon, 31 Aug 2020 at 20:09, Wei-Chiu Chuang  wrote:

> Dear fellow Hadoop developers,
>
> Given the overwhelming feedback from the discussion thread
> https://s.apache.org/hadoop2.9eold, I'd like to start an official vote
> thread for the community to vote and start the 2.9 EOL process.
>
> What this entails:
>
> (1) an official announcement that no further regular Hadoop 2.9.x releases
> will be made after 2.9.2 (which was GA on 11/19/2019)
> (2) resolve JIRAs that specifically target 2.9.3 as won't fix.
>
>
> This vote will run for 7 days and will conclude by September 7th, 12:00pm
> pacific time.
> Committers are eligible to cast binding votes. Non-committers are welcomed
> to cast non-binding votes.
>
> Here is my vote, +1
>

Re: [VOTE] Moving Ozone to a separated Apache project

[Steve Loughran]

+1, binding

On Fri, 25 Sep 2020 at 06:59, Elek, Marton  wrote:

> Hi all,
>
> Thank you for all the feedback and requests,
>
> As we discussed in the previous thread(s) [1], Ozone is proposed to be a
> separated Apache Top Level Project (TLP)
>
> The proposal with all the details, motivation and history is here:
>
>
> https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Hadoop+subproject+to+Apache+TLP+proposal
>
> This voting runs for 7 days and will be concluded at 2nd of October, 6AM
> GMT.
>
> Thanks,
> Marton Elek
>
> [1]:
>
> https://lists.apache.org/thread.html/rc6c79463330b3e993e24a564c6817aca1d290f186a1206c43ff0436a%40%3Chdfs-dev.hadoop.apache.org%3E
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

reviewers needed: HADOOP-16830 Add public IOStatistics API

[Steve Loughran]

Hi,

Can I get some reviews of this PR
https://github.com/apache/hadoop/pull/2323

It adds a new API, IOStatisticsSource, for any class to act as a source of
a static or dynamic IOStatistics set of counters/gauges/min/max/mean stats

The intent is to allow applications to collect statistics on streams,
iterators, and other classes they use to interact with filesystems/remote
stores, so get detailed statistics on the #of operations, latencies etc.
There's help to log these results, as well as aggregate them


Here's the API specifications

https://github.com/steveloughran/hadoop/blob/s3/HADOOP-16830-iostatistics-common/hadoop-common-project/hadoop-common/src/site/markdown/filesystem/iostatistics.md

The FSDataStreams do passthrough of this, and there's a set of remote
iterators which also do passthrough, making it easy to chain/wrap
iteration code.
https://github.com/steveloughran/hadoop/blob/s3/HADOOP-16830-iostatistics-common/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/functional/RemoteIterators.java

It also includes a statistics snapshot which can be serialized as JSON and
java objects, and aggregate results
https://github.com/steveloughran/hadoop/blob/s3/HADOOP-16830-iostatistics-common/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/IOStatisticsSnapshot.java

This is how applications can aggregate results, and then propagate it back
to the AM/job driver/query engine

We already have PRs using this for S3A and ABFS on input streams, and in
S3A we also count LIST performance, which clients can pick up provided they
use the listStatusIterator, listFiles etc calls which return RemoteIterator.

I know it's a lot of code, but it's split into interface and
implementation, the public interface is for applications, the
implementation is what we are using internally, and which we will tune as
we adopt it more.

I have been working on this on and off for months, and yes it has grown.
But now that we are supporting more complex storage systems, the existing
tracking of long/short reads isn't informative enough. I want to know how
many GET requests failed and had to be retried, how often the DELETE calls
were throttled, and what the real latency of list operations are over
long-haul connections.

Please, take a look. As a new API it's unlikely to cause any regressions
-the main things to worry about are "is that API the one applications can
use" and "hi Steve got something fundamentally wrong in his implementation
code?"

-Steve

Re: Wire compatibility between Hadoop 3.x client and 2.x server

[Steve Loughran]

I have this belief that webhdfs:// is better for cross-version
compatibility. That right?

On Fri, 16 Oct 2020 at 19:59, Chao Sun  wrote:

> Thanks for the replies all!
>
> > But in the opposite case, it might have problem, because 2.x server may
> not understand new client calls added in 3.x
>
> Yes not expecting this to work. I'm thinking about the case where one
> upgrades existing 2.x clients to 3.x and expects it to still work against
> 2.x server, which should not involve those new APIs.
>
> > Another backcompat issue was HDFS-15191, in 3.2.1.
>
> Thanks for pointing this out! In fact this looks like a serious bug in
> 3.2.1. Glad to see it is fixed in 3.2.2.
>
> > But aside from that, we've been using 3x client libraries against both 2x
> and 3x clusters without issue.
>
> Great! thanks.
>
> IMO it will be great if the community maintains official compatibility doc
> w.r.t 2.x/3.x, which can help the migration easier.
>
> Chao
>
> On Tue, Oct 13, 2020 at 11:57 PM Wu,Jianliang(vip.com) <
> jianliang...@vipshop.com> wrote:
>
> > Ok, I will file a HDFS jira to report this issue.
> >
> > > 2020年10月13日 20:43，Wei-Chiu Chuang  写道：
> > >
> > > Thanks Jialiang for reporting the issue.
> > > That sounds bad and should've not happened. Could you file a HDFS jira
> > and
> > > fill in more details?
> > >
> > > On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
> > > jianliang...@vipshop.com> wrote:
> > >
> > >> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
> > >> 2.6,  we found hive to call getContentSummary method , the client and
> > >> server was not compatible  because of hadoop3 added new PROVIDED
> storage
> > >> type.
> > >>
> > >> 2020年10月13日 06:41，Chao Sun  > sunc...@apache.org>>
> > >> 写道：
> > >>
> > >>
> > >>
> > >>
> >
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> > >> This communication is intended only for the addressee(s) and may
> contain
> > >> information that is privileged and confidential. You are hereby
> notified
> > >> that, if you are not an intended recipient listed above, or an
> > authorized
> > >> employee or agent of an addressee of this communication responsible
> for
> > >> delivering e-mail messages to an intended recipient, any
> dissemination,
> > >> distribution or reproduction of this communication (including any
> > >> attachments hereto) is strictly prohibited. If you have received this
> > >> communication in error, please notify us immediately by a reply e-mail
> > >> addressed to the sender and permanently delete the original e-mail
> > >> communication and any attachments from all storage devices without
> > making
> > >> or otherwise retaining a copy.
> > >>
> >
> >
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> > This communication is intended only for the addressee(s) and may contain
> > information that is privileged and confidential. You are hereby notified
> > that, if you are not an intended recipient listed above, or an authorized
> > employee or agent of an addressee of this communication responsible for
> > delivering e-mail messages to an intended recipient, any dissemination,
> > distribution or reproduction of this communication (including any
> > attachments hereto) is strictly prohibited. If you have received this
> > communication in error, please notify us immediately by a reply e-mail
> > addressed to the sender and permanently delete the original e-mail
> > communication and any attachments from all storage devices without making
> > or otherwise retaining a copy.
> >
>

For review: HADOOP-17313. FileSystem.get to support slow-to-instantiate FS clients

[Steve Loughran]

Broad review of this would be good as FileSystem.get() and
Path.getFileSystem are so common
https://github.com/apache/hadoop/pull/2396

-Steve

PS: I'm talking about the IOStatistics API at 11:00 PST today (Oct 28)
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Storage+Online+Meetup

Re: Intervention. Stabilizing Yetus (Attn. Azure)

[Steve Loughran]

I'll look at these tomorrow -I'd rather fix whatever problem there is
rather than roll back a JAR change which is inevitably going to come back.

As extra motivation: no commits -> hadoop-azure until this is fixed.

On Mon, 9 Nov 2020 at 21:19, Ayush Saxena  wrote:

> The failing Azure tests are being tracked at HADOOP-17325
>
> https://issues.apache.org/jira/browse/HADOOP-17325
>
> On Mon, 9 Nov 2020 at 23:02, Ahmed Hussein  wrote:
>
> > I created new Jiras for HDFS failures. Please consider doing the same for
> > Yarn and Azure.
> > For convenience, the list of failures in the qbt report is as follows:
> >
> > Test Result
> > <
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/
> > >
> > (50
> > failures / -7)
> >
> >-
> >
> >
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testGetCachedDatanodeReport
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testGetCachedDatanodeReport/
> > >
> >-
> >
> >
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testNamenodeMetrics
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testNamenodeMetrics/
> > >
> >-
> >
> >
> org.apache.hadoop.hdfs.server.federation.router.TestRouterRpcMultiDestination.testErasureCoding
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.federation.router/TestRouterRpcMultiDestination/testErasureCoding/
> > >
> >-
> >
> >
> org.apache.hadoop.hdfs.server.datanode.TestBPOfferService.testMissBlocksWhenReregister
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.hdfs.server.datanode/TestBPOfferService/testMissBlocksWhenReregister/
> > >
> >-
> >
> org.apache.hadoop.yarn.sls.TestReservationSystemInvariants.testSimulatorRunning[Testing
> >with: SYNTH,
> >
> >
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler,
> >(nodeFile null)]
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls/TestReservationSystemInvariants/testSimulatorRunning_Testing_with__SYNTH__org_apache_hadoop_yarn_server_resourcemanager_scheduler_fair_FairScheduler___nodeFile_null__/
> > >
> >-
> > org.apache.hadoop.yarn.sls.appmaster.TestAMSimulator.testAMSimulator[1]
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.sls.appmaster/TestAMSimulator/testAMSimulator_1_/
> > >
> >-
> >
> >
> org.apache.hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer.testTokenThreadTimeout
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.server.resourcemanager.security/TestDelegationTokenRenewer/testTokenThreadTimeout/
> > >
> >-
> >
> >
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithOpportunisticContainers
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithOpportunisticContainers/
> > >
> >-
> >
> >
> org.apache.hadoop.yarn.applications.distributedshell.TestDistributedShell.testDSShellWithEnforceExecutionType
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.yarn.applications.distributedshell/TestDistributedShell/testDSShellWithEnforceExecutionType/
> > >
> >- org.apache.hadoop.fs.azure.TestBlobMetadata.testFolderMetadata
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFolderMetadata/
> > >
> >-
> >
> >
> org.apache.hadoop.fs.azure.TestBlobMetadata.testFirstContainerVersionMetadata
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testFirstContainerVersionMetadata/
> > >
> >- org.apache.hadoop.fs.azure.TestBlobMetadata.testPermissionMetadata
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testPermissionMetadata/
> > >
> >-
> org.apache.hadoop.fs.azure.TestBlobMetadata.testOldPermissionMetadata
> ><
> >
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/319/testReport/junit/org.apache.hadoop.fs.azure/TestBlobMetadata/testOldPermissionMetadata/
> > >
> >-
> >
> >
> org.apa

Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Steve Loughran]

Gosh, has it really been only since february since I last asked the HDFS
dev list to stop adding anything to FileSystem/FileContext APIs without

* mentioning this on the hadoop-common list.
* specifying what it does in filesystem.md
* with a contract test
* a new hasPathCapabilities probe. Throwing UnsupportedOperationException
only lets people work out if it is unsupported through invocation. Being
able to probe for it is better.
* ViewFS support.
* And, for any new API, one which works well for high-latency object
stores: returning Future and  Future
when > 1 result is returned

This needs to hold even for pulling something up from HDFS. Because if
another FS wants to implement it, they need to know what it does, and have
tests to verify this. I say this as someone who has tried to document HDFS
rename() semantics and gave up.

It's really frustrating that every time someone does an FS API change like
this in the past (most recently HDFS-13616) I am the one who has to keep
sending the reminders out, and then having to try and clean up/.

So what now?

Options
1. I roll HDFS-15567 back "please be follow process"
2. Someone does a followup patch with specification and contract test, view
FS. Add even more to the java
3. We do as per HADOOP-16898 into an MSyncable interface and then
FileSystem & HDFS can implement. ViewFS and filterFS still need to pass
through.

*If nobody is going to volunteer for the specification/test changes, I'm
happy for the rollback. It'll remind people about process, *

Pre-emptive Warning: No matter what we do for this patch, I will roll back
the next change which adds a new API if it's not accompanied by
specification and tests.

Unhappily yours,

Steve

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Steve Loughran]

Silence from the  HDFS team


Hadoop 3.2.2 is in an RC; it has the new FS API call. I really don't want
to veto the release just because someone pulled up a method without doing
the due diligence.

Is anyone in the HDFS going to do that due diligence or should we include
something in the release notes "msync()" must be considered unstable.

Then we can do a proper msync().

If it goes into FS/FC, what does it do for a viewfs with >1 mounted HDFS?
Should it take path, msync(path) so that viewFS knows where to forward it?

Alternatively: go with an MSync interface which those few FS which
implement it (hdfs) can do that, and the fact that it doesn't have doc or
tests won't be a blocker any more?

-steve




On Thu, 10 Dec 2020 at 12:41, Steve Loughran  wrote:

>
> Gosh, has it really been only since february since I last asked the HDFS
> dev list to stop adding anything to FileSystem/FileContext APIs without
>
> * mentioning this on the hadoop-common list.
> * specifying what it does in filesystem.md
> * with a contract test
> * a new hasPathCapabilities probe. Throwing UnsupportedOperationException
> only lets people work out if it is unsupported through invocation. Being
> able to probe for it is better.
> * ViewFS support.
> * And, for any new API, one which works well for high-latency object
> stores: returning Future and  Future
> when > 1 result is returned
>
> This needs to hold even for pulling something up from HDFS. Because if
> another FS wants to implement it, they need to know what it does, and have
> tests to verify this. I say this as someone who has tried to document HDFS
> rename() semantics and gave up.
>
> It's really frustrating that every time someone does an FS API change like
> this in the past (most recently HDFS-13616) I am the one who has to keep
> sending the reminders out, and then having to try and clean up/.
>
> So what now?
>
> Options
> 1. I roll HDFS-15567 back "please be follow process"
> 2. Someone does a followup patch with specification and contract test,
> view FS. Add even more to the java
> 3. We do as per HADOOP-16898 into an MSyncable interface and then
> FileSystem & HDFS can implement. ViewFS and filterFS still need to pass
> through.
>
> *If nobody is going to volunteer for the specification/test changes, I'm
> happy for the rollback. It'll remind people about process, *
>
> Pre-emptive Warning: No matter what we do for this patch, I will roll back
> the next change which adds a new API if it's not accompanied by
> specification and tests.
>
> Unhappily yours,
>
> Steve
>

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Steve Loughran]

Maybe it's not in the release; it's certainly in the 3.2 branch. Will check
further. If it's in the release I was thinking of adding a warning in the
notes "unstable API"; stable if invoked from DFSClient

On Fri, 11 Dec 2020 at 18:21, Chao Sun  wrote:

> I'm just curious why this is included in the 3.2.2 release? HDFS-15567 is
> tagged with 3.2.3 and the corresponding HDFS-14272 on server side is tagged
> with 3.3.0.
>
> > If it goes into FS/FC, what does it do for a viewfs with >1 mounted HDFS?
> Should it take path, msync(path) so that viewFS knows where to forward it?
>
> The API shouldn't take any path - for viewFS I think it should call this on
> all the child file systems. It might also need to handle the case where
> some downstream clusters support this capability while others don't.
>

That's an extra bit of work for ViewFS then. It should probe for capability
and invoke as/when supported.




>
> > Options
> 1. I roll HDFS-15567 back "please be follow process"
> 2. Someone does a followup patch with specification and contract test, view
> FS. Add even more to the java
> 3. We do as per HADOOP-16898 into an MSyncable interface and then
> FileSystem & HDFS can implement. ViewFS and filterFS still need to pass
> through.
>
> I'm slightly in favor of the hasPathCapabilities approach and make this a
> mixin where FS impls can optionally support. Happy to hear what others
> think.
>

Mixins are great when FC and FS can both implement; makes it easier to code
against either. All the filtering/aggregating FS's will have to implement
it, which means that presence of the interface doesn't guarantee support.

This is an API where it'd be ok to have a no-op if not implemented,
correct? Or is there an requirement like Syncable that specific guarantees
are met?


>
> Chao
>
>
>
> On Fri, Dec 11, 2020 at 9:00 AM Steve Loughran  >
> wrote:
>
> > Silence from the  HDFS team
> >
> >
> > Hadoop 3.2.2 is in an RC; it has the new FS API call. I really don't want
> > to veto the release just because someone pulled up a method without doing
> > the due diligence.
> >
> > Is anyone in the HDFS going to do that due diligence or should we include
> > something in the release notes "msync()" must be considered unstable.
> >
> > Then we can do a proper msync().
> >
> > If it goes into FS/FC, what does it do for a viewfs with >1 mounted HDFS?
> > Should it take path, msync(path) so that viewFS knows where to forward
> it?
> >
> > Alternatively: go with an MSync interface which those few FS which
> > implement it (hdfs) can do that, and the fact that it doesn't have doc or
> > tests won't be a blocker any more?
> >
> > -steve
> >
> >
> >
> >
> > On Thu, 10 Dec 2020 at 12:41, Steve Loughran 
> wrote:
> >
> > >
> > > Gosh, has it really been only since february since I last asked the
> HDFS
> > > dev list to stop adding anything to FileSystem/FileContext APIs without
> > >
> > > * mentioning this on the hadoop-common list.
> > > * specifying what it does in filesystem.md
> > > * with a contract test
> > > * a new hasPathCapabilities probe. Throwing
> UnsupportedOperationException
> > > only lets people work out if it is unsupported through invocation.
> Being
> > > able to probe for it is better.
> > > * ViewFS support.
> > > * And, for any new API, one which works well for high-latency object
> > > stores: returning Future and
> Future
> > > when > 1 result is returned
> > >
> > > This needs to hold even for pulling something up from HDFS. Because if
> > > another FS wants to implement it, they need to know what it does, and
> > have
> > > tests to verify this. I say this as someone who has tried to document
> > HDFS
> > > rename() semantics and gave up.
> > >
> > > It's really frustrating that every time someone does an FS API change
> > like
> > > this in the past (most recently HDFS-13616) I am the one who has to
> keep
> > > sending the reminders out, and then having to try and clean up/.
> > >
> > > So what now?
> > >
> > > Options
> > > 1. I roll HDFS-15567 back "please be follow process"
> > > 2. Someone does a followup patch with specification and contract test,
> > > view FS. Add even more to the java
> > > 3. We do as per HADOOP-16898 into an MSyncable interface and then
> > > FileSystem & HDFS can implement. ViewFS and filterFS still need to pass
> > > through.
> > >
> > > *If nobody is going to volunteer for the specification/test changes,
> I'm
> > > happy for the rollback. It'll remind people about process, *
> > >
> > > Pre-emptive Warning: No matter what we do for this patch, I will roll
> > back
> > > the next change which adds a new API if it's not accompanied by
> > > specification and tests.
> > >
> > > Unhappily yours,
> > >
> > > Steve
> > >
> >
>

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Steve Loughran]

This isn't worth holding up the RC. We'd just add something to the
release notes "use with caution". And if we can get what the API does
defined in a way which works, it shouldn't need changing.

(which reminds me, I do need to check that RC out, don't I?)

On Sun, 13 Dec 2020 at 09:00, Xiaoqiao He  wrote:

> Thanks Steve very much for your discussion here.
>
> Leave some comments inline. Will focus on this thread to wait for the final
> conclusion to decide if we should prepare another release candidate of
> 3.2.2.
> Thanks Steve and Chao again for your warm discussions.
>
> On Sat, Dec 12, 2020 at 7:18 PM Steve Loughran  >
> wrote:
>
> > Maybe it's not in the release; it's certainly in the 3.2 branch. Will
> check
> > further. If it's in the release I was thinking of adding a warning in the
> > notes "unstable API"; stable if invoked from DFSClient
>
> On Fri, 11 Dec 2020 at 18:21, Chao Sun  wrote:
> >
> > > I'm just curious why this is included in the 3.2.2 release? HDFS-15567
> is
> > > tagged with 3.2.3 and the corresponding HDFS-14272 on server side is
> > tagged
> > > with 3.3.0.
> >
>
> Just checked that HDFS-15567 has been involved in Hadoop-3.2.2 RC4. IIRC, I
> have cut branch-3.2.2 in early October, at that time branch-3.2.3 has
> created but source code not freeze completely because several blocked
> issues reported and code freeze has done about mid October. Some issues
> which are tagged with 3.2.3 has also been involved in 3.2.2 during
> that period, include HDFS-15567. I will check them later, and make sure
> that we have mark the correct tags.
>
>
> > >
> > > > If it goes into FS/FC, what does it do for a viewfs with >1 mounted
> > HDFS?
> > > Should it take path, msync(path) so that viewFS knows where to forward
> > it?
> > >
> > > The API shouldn't take any path - for viewFS I think it should call
> this
> > on
> > > all the child file systems. It might also need to handle the case where
> > > some downstream clusters support this capability while others don't.
> > >
> >
> > That's an extra bit of work for ViewFS then. It should probe for
> capability
> > and invoke as/when supported.
> >
> > >
> > > > Options
> > > 1. I roll HDFS-15567 back "please be follow process"
> > > 2. Someone does a followup patch with specification and contract test,
> > view
> > > FS. Add even more to the java
> > > 3. We do as per HADOOP-16898 into an MSyncable interface and then
> > > FileSystem & HDFS can implement. ViewFS and filterFS still need to pass
> > > through.
> > >
> > > I'm slightly in favor of the hasPathCapabilities approach and make
> this a
> > > mixin where FS impls can optionally support. Happy to hear what others
> > > think.
> > >
> >
> > Mixins are great when FC and FS can both implement; makes it easier to
> code
> > against either. All the filtering/aggregating FS's will have to implement
> > it, which means that presence of the interface doesn't guarantee support.
> >
> > This is an API where it'd be ok to have a no-op if not implemented,
> > correct? Or is there an requirement like Syncable that specific
> guarantees
> > are met?
> >
> > >
> > > Chao
> > >
> > >
> > > On Fri, Dec 11, 2020 at 9:00 AM Steve Loughran
> >  > > >
> > > wrote:
> > >
> > > > Silence from the  HDFS team
> > > >
> > > >
> > > > Hadoop 3.2.2 is in an RC; it has the new FS API call. I really don't
> > want
> > > > to veto the release just because someone pulled up a method without
> > doing
> > > > the due diligence.
> >
>
> Thanks Steve started this discussion here. I agree to roll back HDFS-15567
> if there are still some incompatible issues not resolved completely. And
> release will not be the blocked things here, I would like to prepare
> another RC if we would reach common agreement. To be honest, I think it is
> better to involve Shvachko here.
>
>
> > > > Is anyone in the HDFS going to do that due diligence or should we
> > include
> > > > something in the release notes "msync()" must be considered unstable.
> > > >
> > > > Then we can do a proper msync().
> > > >
> > > > If it goes into FS/FC, what does it do for a viewfs with >1 mounted
> > HDFS?
>

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Steve Loughran]

On Sun, 13 Dec 2020 at 21:08, Konstantin Shvachko 
wrote:

> Hi Steve,
>
> I am not sure I fully understand what is broken here. It is not an
> incompatible change, right?
>

The issue is that the FileSystem/FileContext APIs are something we have to
maintain ~forever, so every API change needs to be

- something we are happy with being there for the life of the class
- defined strictly enough that people implementing other filesystems can
re-implement without having to reverse-engineer HDFS and then conclude
"that is what they meant to do". That's with a bit of
- and with an AbstractFileSystemContractTest for implementors.

That's it: define, specify, add a contract test rather than just something
for HDFS.



> Could you please explain what you think the process is.
> Would be best if you could share a link to a document describing it.
> I would be glad to follow up with tests and documentation that are needed.
>
>
ideally,
hadoop-common-project/hadoop-common/src/site/markdown/filesystem/extending.md

Pulling up something from hdfs is different from saying "hey, new rename
API!", but it's still time to actually define what it does so that not only
can other people like me reimplement it, but to actually define it well
enough that we can all see when there's a regression.

Equally important: is there a way to test that it works?

We've been using hasPathCapabilities() to probe for an FS having a given
feature down a path; the idea is to let code check upfront for a feature
before having to call it and catching an exception,

We can add that for an API even if it has shipped already. For example,
here is a PR to do exactly that for Syncable
https://github.com/apache/hadoop/pull/2102



> As you can see I proposed multiple solutions to the problem in the jira.
> Seemed nobody was objecting, so I chose one and explained why.
> I believe we call it lazy consensus.
>

I'm happy with lazy consensus, but can you involve more people? In
particular. i was filed in an HDFS JIRA so it didn't surface in
hadoop-common.

If you'd done a HADOOP- JIRA "pull up msync into FileSystem API" or even
just a note to hadoop-common saying "we need to do this" that would have
been enough to start a discussion.

 As it is I only noticed after some rebasing with a "hang on a minute.
here's a new method who's behaviour doesn't seem to have defined other than
'whatever hdfs does'". Which, if you've ever tried to work out what
rename() does, you'll recognise as danger.

Anyway, to finish off, have a look at the extending.md doc and just add a
new method definition in the filesystem.md saying what it is meant to do.

Now: what about viewfs? maybe: For all mounted fileystems which declare
their support, call msync()? Or just "call it and swallow all exceptions?"


> Stay safe,
>


yeah: )

> --Konstantin
>

>>

Re: Regarding HDFS-15567. HDFS should expose msync() API to allow downstream applications call it explicitly

[Steve Loughran]

On Fri, 18 Dec 2020 at 23:29, Konstantin Shvachko 
wrote:

> Hey Steve,
>
> Thanks for the references. I was reading but still need to understand how
> exactly this applies to msync.
>

mainly: pull it up and it becomes part of the broader API, so needs to be
specified in a way which can be understood by users and for implementors of
others stores: to give their own stores the same semantics.

What does the HDFS one do?



> Will come up with a plan and post it on a new jira.
> Will make sure to create it under HADOOP and ping Hadoop Common list for
> visibility.
>
>
thanks


> You are right about ViewFS. The impl should make sure it calls msync() on
> all mount points that enabled observer reads.
>
>
That's the kind of issue this process aims to resolve. Another is to
identify where we have HDFS-layer "quirks" and at least document them (e.g.
how hdfs streams are thread safe, rename isn't Posix, ...) and list what we
know breaks if you don't re-implement

[DISCUSS] HADOOP-17464. Create hadoop-compression module

[Steve Loughran]

Following on from discussions in a previous PR, Liang-Chi Hsieh has created
a PR to add a new hadoop-compression module, with the goal being "new
compression codecs and their dependencies can go in here rather than
hadoop-common"

https://github.com/apache/hadoop/pull/2611

I think this is the right thing to do as it keeps the stuff out of
hadoop-common dependencies, and isolates the changes.

But as you can see from the test results on that PR, there's stuff in
hadoop-hdfs and hadoop-mapreduce which expects those codes to always been
on the classpath.

What to do?

1. We add the new pom as a dependency of hadoop hdfs server and its tests
(but not hdfs client), and for MR tests
2. we leave the old codecs in hadoop-common, and its only the recent stuff
which we add to the new module.

Suggestions?

hdfs tests failing in yarn?

[Steve Loughran]

I've got a PR which goes near HDFS output stream (declares it supports
Syncable in hasCapability())

The PR is failing on four tests -two timeouts, two assertions

https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2587/2/testReport/

are these things I should expect to see & not worry, or have I broken
something?

-steve

Re: [DISCUSS] Hadoop 3.3.1 release

[Steve Loughran]

yeah, it's time.I've been trying to get things lined up for this
object-store wise, a few things tagged as blockers where I really want them
in

Wei Chiu -if you want to be the release manager I'll share the workload,

Regarding blockers &c: how about we have a little hackathon where we try
and get things in. This means a promise of review time from the people with
commit rights and other people who understand the code (Stack?)

-steve

On Thu, 28 Jan 2021 at 06:48, Ayush Saxena  wrote:

> +1
> Just to mention we would need to release hadoop-thirdparty too before.
> Presently we are using the snapshot version of it.
>
> -Ayush
>
> > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang  wrote:
> >
> > Hi all,
> >
> > Hadoop 3.3.0 was released half a year ago, and as of now we've
> accumulated
> > more than 400 changes in the branch-3.3. A number of downstreamers are
> > eagerly waiting for 3.3.1 which addresses the guava version conflict
> issue.
> >
> >
> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
> >
> > We should start the release work for 3.3.1 before the diff becomes even
> > larger.
> >
> > I believe there are  currently only two real blockers for a 3.3.1 (using
> > this filter
> >
> https://issues.apache.org/jira/issues/?filter=-1&jql=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
> > )
> >
> >
> >   1. HDFS-15566 
> >   2.
> >  1. HADOOP-17112  >
> > 2.
> >
> >
> >
> > Is there anyone who would volunteer to be the 3.3.1 RM?
> >
> > Also, the HowToRelease wiki does not describe the ARM build process.
> That's
> > going to be important for future releases.
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [ANNOUNCE] Mukund Thakur is a new Apache Hadoop Committer

[Steve Loughran]

Many congratulations Mukund!
I look forward to you vetoing my PRs on the basis of quality and test
coverage.

-steve

On Wed, 3 Feb 2021 at 11:39, hemanth boyina 
wrote:

> Congratulations Mukund !!
>
>
>
> Thanks
> HemanthBoyina
>
> On Wed, 3 Feb 2021, 11:11 Mingliang Liu,  wrote:
>
> > Hi all,
> >
> > It's my pleasure to announce that Mukund Thakur has been elected as a
> > committer on the Apache Hadoop project recognizing his continued
> > contributions to the project. He has accepted the invitation.
> >
> >
> > His involvement in the hadoop object store modules, hadoop common and
> > distcp tool is a great addition to the Hadoop project. His reviews are
> > comprehensive with high standards to "approve a pull request". The
> granting
> > of committership to him will enable better productivity.
> >
> >
> > Please join me in congratulating him. Welcome aboard, Mukund!
> >
> >
> >
> > Mingliang Liu
> >
> > (on behalf of the Apache Hadoop PMC)
> >
>

Re: Seeing YETUS issue in jenkins runs

[Steve Loughran]

Akira,

I see you've started fixing upstream OSS projects.

take care: first it's spotbugs, next it'll be maven, before long you'll be
doing linux device drivers...



On Fri, 19 Feb 2021 at 05:53, Akira Ajisaka  wrote:

> Probably this issue is due to
> https://issues.apache.org/jira/browse/HADOOP-16870. Reverted.
>
> -Akira
>
> On Fri, Feb 19, 2021 at 11:27 AM Sunil Govindan  wrote:
> >
> > Hi All,
> >
> > We are getting the below error in recent runs.
> >
> > -1.  yetus.   0m 8s.   Unprocessed flag(s): --findbugs-strict-precheck
> >
> > https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/636/console
> >
> > Are there any knows issues or some workarounds?
> > Please advise.
> >
> > Thanks
> > Sunil
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Hadoop 3.3.1 release

[Steve Loughran]

I'm going to argue its too late to do IPv6 support close to a release, as
it's best if its on developer machines for some time to let all quirks
surface. It's not so much IPv6 itself, but do we cause any regressions on
IPv4?

But: it can/should go into trunk and stabilize there

On Thu, 4 Mar 2021 at 03:52, Muralikrishna Dmmkr <
muralikrishna.dm...@gmail.com> wrote:

> Hi Brahma,
>
> I have missed out mentioning about the IPV6 feature in the last mail,
> Support for IPV6 has been in development since 2015 and We have done a good
> amount of testing at our organisation, the feature is stable and used by
> our customers extensively in the last one year. I think it is a good time
> to add the IPV6 support to 3.3.1.
>
> https://issues.apache.org/jira/browse/HADOOP-11890
>
> Thanks
> D M Murali Krishna Reddy
>
> On Wed, Feb 24, 2021 at 9:13 AM Muralikrishna Dmmkr <
> muralikrishna.dm...@gmail.com> wrote:
>
> > Hi Brahma,
> >
> > Can we have this new feature "YARN Registry based AM discovery with retry
> > and in-flight task persistent via JHS" in the upcoming 3.1.1 release. I
> > have also attached a test-report in the below jira.
> >
> > https://issues.apache.org/jira/browse/MAPREDUCE-6726
> >
> >
> > Thanks,
> > D M Murali Krishna Reddy
> >
> > On Tue, Feb 23, 2021 at 10:11 AM Brahma Reddy Battula  >
> > wrote:
> >
> >> Hi Bilwa,
> >>
> >> I have commented on the jira's you mentioned. Based on the stability we
> >> can
> >> plan this.But needs to be merged ASAP.
> >>
> >>
> >>
> >> On Fri, Feb 19, 2021 at 5:20 PM bilwa st  wrote:
> >>
> >> > Hi Brahma,
> >> >
> >> > Can we have below features in 3.3.1 release? We have been using these
> >> > features for a long time. They are stable and tested in bigger
> clusters.
> >> >
> >> > 1. Container reuse -
> >> https://issues.apache.org/jira/browse/MAPREDUCE-6749
> >> > 2. Speculative attempts should not run on the same node -
> >> > https://issues.apache.org/jira/browse/MAPREDUCE-7169
> >> >
> >> > Thanks,
> >> > Bilwa
> >> >
> >> > On Thu, Feb 18, 2021, 1:49 PM Brahma Reddy Battula  >
> >> > wrote:
> >> >
> >> >> Sorry for the late reply..
> >> >>
> >> >> I will come up with a plan.. Please let me know if anybody has some
> >> >> features/improvements/bugs that need to be included.
> >> >>
> >> >> On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan 
> >> wrote:
> >> >>
> >> >> > Hi Wei-Chiu,
> >> >> >
> >> >> > What will be the next steps here for 3.3.1 planning?
> >> >> >
> >> >> > Thanks
> >> >> > Sunil
> >> >> >
> >> >> > On Mon, Feb 8, 2021 at 11:56 PM Stack  wrote:
> >> >> >
> >> >> > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran
> >> >> >  >> >> > > >
> >> >> > > wrote:
> >> >> > >
> >> >> > > >
> >> >> > > > Regarding blockers &c: how about we have a little hackathon
> >> where we
> >> >> > try
> >> >> > > > and get things in. This means a promise of review time from the
> >> >> people
> >> >> > > with
> >> >> > > > commit rights and other people who understand the code (Stack?)
> >> >> > > >
> >> >> > > >
> >> >> > >
> >> >> > > I'm up for helping get 3.3.1 out (reviewing, hackathon, testing).
> >> >> > > Thanks,
> >> >> > > S
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > >
> >> >> > > > -steve
> >> >> > > >
> >> >> > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena  >
> >> >> wrote:
> >> >> > > >
> >> >> > > > > +1
> >> >> > > > > Just to mention we would need to release hadoop-thirdparty
> too
> >> >> > before.
> >> >> > > > > Presently we are using the snapshot version of it.
> >> >> > > > &

Re: RPC logging in ipc Server

[Steve Loughran]

HADOOP-17450 added an API for IOStatistics -anything can be an IOStatistics
source, which can be collected, aggregated, reported (There's a
serializable form). This goes beyond simple counts -we include mins, maxes
and means too

S3A and ABFS support this in their streams, s3a does it for its remote
iterators and FS; ABFS adding for the FS. MR's LocatedFileStatusFetcher
also collects/aggregates any stats from the filesystems it iterates over.

Independent of any sampling based collection of stats on the server (I'd
point to open telemetry) it'd be great if DFSClient collected IOStats for
the FS and for the input/output streams which can then be included in
reports.

Example: stats of a single test case. Really interesting to see those mean
costs of operations -but note also the min/max values.


counters=((action_http_head_request=2)
(audit_access_check_failure=1)
(audit_request_execution=7)
(audit_span_start=5)
(directories_created=1)
(object_list_request=4)
(object_metadata_request=2)
(object_put_request=1)
(object_put_request_completed=1)
(op_access=1)
(op_access.failures=1)
(op_mkdirs=2)
(store_io_request=7));

gauges=();

minimums=((action_http_head_request.min=31)
(object_list_request.min=41)
(object_put_request.min=226)
(op_access.failures.min=73)
(op_mkdirs.min=385));

maximums=((action_http_head_request.max=45)
(object_list_request.max=1039)
(object_put_request.max=226)
(op_access.failures.max=73)
(op_mkdirs.max=1062));

means=((action_http_head_request.mean=(samples=2, sum=76, mean=38.))
(object_list_request.mean=(samples=4, sum=1171, mean=292.7500))
(object_put_request.mean=(samples=1, sum=226, mean=226.))
(op_access.failures.mean=(samples=1, sum=73, mean=73.))
(op_mkdirs.mean=(samples=2, sum=1447, mean=723.5000)));


On Mon, 12 Apr 2021 at 17:30, Stephen O'Donnell
 wrote:

> I have not tried to do this, but as we (Cloudera) deal with more and more
> performance related problems, I feel something like this is needed.
>
> It is a tricky problem due to the number of requests the NN handles and how
> performance sensitive it is.
>
> At the IPC Server level, we should be able to know the request queue time,
> processing time, response queue time and the type of request.
>
> If we sampled X% of requests and then emitted one log line per interval (eg
> per minute), we could perhaps build a histogram of queue size, queue times,
> processing times per request type.
>
> From JMX, we can get the request counts and queue length, but I am not sure
> if we can get something like percentiles or queue time and processing time
> over the previous minute for example?
>
> Even given the above details, if we see a long queue length, it may still
> remain a mystery about what was causing that queue. Often it is due to a
> long running request (eg contentSummay, snapshotdiff etc) holding the NN
> lock in write mode for too long.
>
> What would be very useful, is a way to see the percentage of time the NN
> lock is held in Exclusive mode (write), shared mode (read) or not held at
> all (rare on a busy cluster). Even better if we can somehow bubble up the
> top requests holding the lock in exclusive mode.
>
> Perhaps sampling the time spent waiting to acquire the lock could be useful
> too.
>
> I also think it would be useful to expose response times from the client
> perspective.
>
> https://issues.apache.org/jira/browse/HDFS-14084 seemed interesting and
> could be worth finishing.
>
> I also found https://issues.apache.org/jira/browse/HDFS-12861 some time
> back to get the client to log data read speeds.
>
> Have you made any attempts in this area so far, and did you have any
> success?
>
> Thanks,
>
> Stephen.
>
> On Thu, Mar 18, 2021 at 5:41 AM Fengnan Li  wrote:
>
> > Hi community,
> >
> >
> >
> > Has someone ever tried to implement sampling logging for ipc Server? We
> > would like to gain more observability for all of the traffics to the
> > Namenode.
> >
> >
> >
> > Thanks,
> > Fengnan
> >
> >
>

Re: Java 8 Lambdas

[Steve Loughran]

LambdaTestUtils added l-expression based intercept() in HADOOP-13716 in
October 2016.
Five Years Ago. That was still java-7...we added it knowing what java 8
would bring.

There is no way we could go back on not using intercept() in tests.

Since then some other big l-expression stuff I've been involved in include
the org.apache.hadoop.fs.s3a.Invoker which lets you executre a remote
operation with conversion of AWS SDK exceptions into IOEs, and a retry
policy based off those IOEs.

final String region = invoker.retry("getBucketLocation()", bucketName,
true,
() -> s3.getBucketLocation(bucketName));

That was HADOOP-13786, S3A committers: 2017.  Four years ago. Which was
after branch-3 was java 8 only.

More recently, if you look closely, the whole
org.apache.hadoop.util.functional package is designed to give us basic
Functional Programming around IOE-raising code, including our remote
iterators, giving us a minimal *and tested* set of transformations we can
do with our code.

  public RemoteIterator
createLocatedFileStatusIterator(
  RemoteIterator statusIterator) {
return RemoteIterators.mappingRemoteIterator(
statusIterator,
listingOperationCallbacks::toLocatedFileStatus);
  }

This ties in nicely with the duration tracking/IOStatistics code it came in
(HADOOP-17450), so I can evaluate an operation and collect min/mean/max
durations of operations, not just log but serialize into the task/job
summary files and so get some details on where the bottlenecks are in
talking to cloud services.

final RemoteIterator listing =
trackDuration(iostatistics, OP_DIRECTORY_SCAN, () ->
operations.listStatusIterator(srcDir));


So I'm afraid that I will be carrying on using L-expressions, such as in
HADOOP-17511. But I don't expect any of the code there to be backportable
to Java 7(*)

At the same time, I'd like to know what the performance impact of us using
l-expressions is in terms of cost of allocations of closures, evaluation
etc. There's also the *little* detail that stack trace data doesn't get
preserved that well. Together that argues against gratuitous use of java
streams.


To summarise my PoV then

Java 8 lambda expressions are an incredible tool which can be used in
interesting and innovative ways. Adding retryability, stats gathering and
auditing of remote IO being the key ones I've been using it for, in the
Hadoop codebase, for 4-5 years.

I'm happy to let someone lay out a style guide on good/bad uses, a "no
gratuitous move to streams()" policy, and may be a designated "No Lambda's
here" bit of code. (UGI?)

But a discussion about whether to have them in the code at all? Not only
too late, I don't see how that can be justified.

-Steve

(*) . Having recently been backporting some ABFS code to a branch-3.1 fork,
Mockito version downgrading is enough of a blocker on test cases there
that the language version is a detail...you won't get that far.

Re: Do I need a +1 to merge a backport PR?

[Steve Loughran]

I think it comes down to "do you think somebody else needs to review it?".

I do like to test before a cherrypick -at least of all the tests which
the patch changed, and for object store stuff a full test run is good due
diligence, but I think at least for me

cherrypick no merge issues: local compile, maybe test, push up
cherrypick import merge issues: the same
merge problems and need to pull in previous work -that's when it gets
complicated.

The last bit of work of mine with major backport was the list-side changes
of HADOOP-13230, where all versions of Hadoop needed to be aware that an
empty directory marker in s3 didn't mean that the path was empty.

That I did as a new JIRA with it's own PRs for each branch, even though it
was just a small subset of the main patch + some of the PathCapabilities
API (HADOOP-17210) for probes (
https://issues.apache.org/jira/browse/HADOOP-17199).

I did all that though the normal build process but once I was happy I did
the merge myself. I could make the case here that (a) the changes were
isolated and (b) they were needed. And I was doing the changes through
gerrit if anybody wanted to see what I was doing.

-Steve



On Wed, 2 Jun 2021 at 06:18, Masatake Iwasaki 
wrote:

> I think we usually merge backport without explicit +1 if the conflict is
> trivial,
> same as done on JIRA-only workflow
> (on which we attached patch like HADOOP-xxx-branch-3.3.001.patch for
> precommit check then commit it without explicit +1).
>
> If committers want review since the change is not trivial,
> backporting PR might tend to be ignored without explicit request.
>
> There seems to be no clear statement about approving backports on
> HowToCommit wiki.
> https://cwiki.apache.org/confluence/display/HADOOP2/HowToCommit
>
> On 2021/06/02 13:43, Wei-Chiu Chuang wrote:
> > I'm curious about the GitHub PR conventions we use today... say I want to
> > backport a commit from trunk to branch-3.3, and there's a small code
> > conflict so I push a PR against branch-3.3 using GitHub to go through the
> > precommit check.
> >
> > Do I need explicit approval from another committer to merge the backport
> > PR? (As.a committer, I know I can merge at any time) or can I merge when
> > the precommit comes back okay?
> >
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Steve Loughran]

Extend for a bit from the last RC, as it takes time to qualify.

I'm busy testing, doing

- packaging &c
- CLI working with abfs and s3, both fs and cloudstore library calls
- building downstream projects (so validating maven artifacts). cloudstore
and spark there
- building downstream of downstream projects, i.e. my spark cloud
IO/committer test module. Moving to spark 3 cost me the afternoon, not
through any incompatible changes there but because the upgraded scalatest
"moved" their foundational FunTest class to a different package and name.
Not happy with Team Scalatest there.
- reviewing the docs in the -aws and azure modules to see they link
together OK.

So far so good.

One troublespot (which isn't any reason to hold up the release), is that
the table in the directory_markers markdown file doesn't render right.
 Created https://issues.apache.org/jira/browse/HADOOP-17746.

This is *not a blocker*

I can prepare a fix and we can have it in so that if any other changes come
in the page will look OK.




On Thu, 3 Jun 2021 at 17:30, Wei-Chiu Chuang  wrote:

> Hello,
> do we want to extend the release vote? I understand a big release like this
> takes time to validate.
>
> I am aware a number of people are testing it: Attila tested Ozone on Hadoop
> 3.3.1 RC3, Stack is testing HBase, Chao tested Spark.
> I also learned that anecdotally Spark on S3 on Hadoop 3.3 is faster by 20%
> over Hadoop 3.2 library.
>

ooh. That'll be from Mukund's listing improvements translating into query
planning speedups,

Nice

If someone benchmarking this stuff were to enable directory marker retention
fs.s3a.directory.marker.retention=keep , I'd be interested to know how much
speedup that
delivers on versioned and unversioned buckets.

Unversioned: reduces risk of IO throttling on writes
Versioned: that and should stop subsequent LIST operations from getting
slowed down from all the tombstones


>
> Looks like we may need some more time to test. How about extending it by a
> week?
>


That would be good. This week included some holidays for people in the
US/UK which is why I'm a bit behind on my testing.


>
>

Re: [VOTE] Hadoop 3.1.x EOL

[Steve Loughran]

On Thu, 3 Jun 2021 at 07:14, Akira Ajisaka  wrote:

> Dear Hadoop developers,
>
> Given the feedback from the discussion thread [1], I'd like to start
> an official vote
> thread for the community to vote and start the 3.1 EOL process.
>
> What this entails:
>
> (1) an official announcement that no further regular Hadoop 3.1.x releases
> will be made after 3.1.4.
> (2) resolve JIRAs that specifically target 3.1.5 as won't fix.
>
> This vote will run for 7 days and conclude by June 10th, 16:00 JST [2].
>
> Committers are eligible to cast binding votes. Non-committers are welcomed
> to cast non-binding votes.
>
> Here is my vote, +1
>


+1 (binding)

>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Steve Loughran]

+1, binding.

Awesome piece of work!

I've done three forms of qualification, all related to s3 and azure storage

   1. tarball validate, CLI use
   2. build/test of downstream modules off maven artifacts; mine and some
   other ASF ones. I  (and it its very much me) have broken some downstream
   modules tests, as I will discuss below. PRs submitted to the relevant
   projects
   3. local rerun of the hadoop-aws and hadoop-azure test suites


*Regarding issues which surfaced*

Wei-Chiu: can you register your private GPG key with the public keystores?
The gpg client apps let you do this? Then we can coordinate signing each
other's keys

Filed PRs for the test regressions:
https://github.com/apache/hbase-filesystem/pull/23
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

*Artifact validation*

SHA checksum good:


shasum -a 512 hadoop-3.3.1-RC3.tar.gz
b80e0a8785b0f3d75d9db54340123872e39bad72cc60de5d263ae22024720e6e824e022090f01e248bf105e03b0f06163729adbe15b5b0978bae0447571e22eb
 hadoop-3.3.1-RC3.tar.gz


GPG: trickier, because Wei-Chiu wasn't trusted

> gpg --verify hadoop-3.3.1-RC3.tar.gz.asc

gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: requesting key 0xB362E1C021854B9D from hkps server
hkps.pool.sks-keyservers.net
gpg: Can't check signature: No public key


*Wei-Chiu: can you add your public keys to the GPG key servers*

To validate the keys I went to the directory where I have our site under
svn (https://dist.apache.org/repos/dist/release/hadoop/common) , and, after
reinstalling svn (where did it go? when did it go?) did an svn update to
get the keys

Did a gpg import of the KEYS file, added

gpg: key 0x386D80EF81E7469A: public key "Brahma Reddy Battula (CODE SIGNING
KEY) " imported
gpg: key 0xFC8D04357BB49FF0: public key "Sammi Chen (CODE SIGNING KEY) <
sammic...@apache.org>" imported
gpg: key 0x36243EECE206BB0D: public key "Masatake Iwasaki (CODE SIGNING
KEY) " imported
*gpg: key 0xB362E1C021854B9D: public key "Wei-Chiu Chuang
>" imported*

This time an import did work, but Wei-Chiu isn't trusted by anyone yet

gpg --verify hadoop-3.3.1-RC3.tar.gz.asc
gpg: assuming signed data in 'hadoop-3.3.1-RC3.tar.gz'
gpg: Signature made Tue Jun  1 11:00:41 2021 BST
gpg:using RSA key CD32D773FF41C3F9E74BDB7FB362E1C021854B9D
gpg: Good signature from "Wei-Chiu Chuang " [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:  There is no indication that the signature belongs to the
owner.
Primary key fingerprint: CD32 D773 FF41 C3F9 E74B  DB7F B362 E1C0 2185 4B9D

(Wei-Chiu, let's coordinate signing each other's public keys via a slack
channel; you need to be in the apache web of trust)


> time gunzip hadoop-3.3.1-RC3.tar.gz

(5 seconds)

cd into the hadoop dir;
cp my confs in: cp ~/(somewhere)/hadoop-conf/*  etc/hadoop/
cp the hadoop-azure dependencies from share/hadoop/tools/lib/ to
share/hadoop/common/lib (products built targeting Azure put things there)

run: all the s3a "qualifying an AWS SDK update" commands
https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/testing.html#Qualifying_an_AWS_SDK_Update

run: basic abfs:// FS operations; again no problems.
FWIW I think we should consider having the hadoop-aws module and
dependencies, and the aws ones in hadoop-common/lib. I can get them there
through env vars and the s3guard shell sets things up, but azure is fiddly.

*Build and test cloudstore JAR; invoke from CLI*

This is my cloud-storage extension library
https://github.com/steveloughran/cloudstore

I've always intended to put it into hadoop, but as it is where a lot of
diagnostics and quick way to put together fixes "here's a faster du ("dux"")

https://github.com/steveloughran/cloudstore.git

modify the hadoop-3.3 profile to use 3.3.1 artifacts, then build with
snapshots enabled. Because I'd not (yet) built any 3.3.1 artifacts locally,
this fetched them from maven staging

mvn package -Phadoop-3.3 -Pextra -Psnapshots-and-staging


Set up env var $CLOUDSTORE to point to JAR; $BUCKET to s3a bucket, run
various commands (storediag, cloudup, ...). As an example, here's the "dux"
command, which is "hadoop fs -du" with parallel scan underneath the dir for
better scaling


bin/hadoop jar $CLOUDSTORE dux  -threads 64 -limit 1000 -verbose
s3a://stevel-london/

output is in
https://gist.github.com/steveloughran/664d30cef20f605f3164ad01f92a458a

*Build and (unit test) google GCS: *


Two test failures, one of which was classpath related and the other just a
new rename contract test needing a new setting in gs.xml to declare what
rename of file over file does.

Everything is covered in:
https://github.com/GoogleCloudDataproc/hadoop-connectors/pull/569

Classpath: assertJ not coming through hadoop-common-test JAR dependencies.

[ERROR]
com.google.cloud.hadoop.fs.gcs.contract.TestInMemoryGoogleContractR

Re: [DISCUSS] Change project style guidelines to allow line length 100

[Steve Loughran]

+1

if you look closely the hadoop-azure module went to 100 lines a while back
and all is good

On Wed, 19 May 2021 at 22:13, Sean Busbey  wrote:

> Hello!
>
> What do folks think about changing our line length guidelines to allow for
> 100 character width?
>
> Currently, we tell folks to follow the sun style guide with some exception
> unrelated to line length. That guide says width of 80 is the standard and
> our current check style rules act as enforcement.
>
> Looking at the current trunk codebase our nightly build shows a total of
> ~15k line length violations; it’s about 18% of identified checkstyle issues.
>
> The vast majority of those line length violations are <= 100 characters
> long. 100 characters happens to be the length for the Google Java Style
> Guide, another commonly adopted style guide for java projects, so I suspect
> these longer lines leaking past the checkstyle precommit warning might be a
> reflection of committers working across multiple java codebases.
>
> I don’t feel strongly about lines being longer, but I would like to move
> towards more consistent style enforcement as a project. Updating our
> project guidance to allow for 100 character lines would reduce the
> likelihood that folks bringing in new contributions need a precommit test
> cycle to get the formatting correct.
>
> Does anyone feel strongly about keeping the line length limit at 80
> characters?
>
> Does anyone feel strongly about contributions coming in that clear up line
> length violations?
>
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Tips for improving productivity, workflow in the Hadoop project?

[Steve Loughran]

not sure about stale PR closing; when you've a patch which is still pending
review it's not that fun to have it closed.

maybe better to have review sessions. I recall many, many years ago
attempts to try and catch up with all outstanding patch reviews.




On Wed, 14 Jul 2021 at 03:00, Akira Ajisaka  wrote:

> Thank you Wei-Chiu for starting the discussion,
>
> > 3. JIRA security
> I'm +1 to use private JIRA issues to handle vulnerabilities.
>
> > 5. Doc update
> +1, I build the document daily and it helps me fixing documents:
> https://aajisaka.github.io/hadoop-document/ It's great if the latest
> document is built and published by the Apache Hadoop community.
>
> My idea related to GitHub PR:
> 1. Disable the precommit jobs for JIRA, always use GitHub PR. It saves
> costs to configure and debug the precommit jobs.
> https://issues.apache.org/jira/browse/HADOOP-17798
> 2. Improve the pull request template for the contributors
> https://issues.apache.org/jira/browse/HADOOP-17799
>
> Regards,
> Akira
>
> On Tue, Jul 13, 2021 at 12:35 PM Wei-Chiu Chuang 
> wrote:
> >
> > I work on multiple projects and learned a bunch from those projects.There
> > are nice add-ons that help with productivity. There are things we can do
> to
> > help us manage the project better.
> >
> > 1. Add new issue types.
> > We can add "Epic" jira type to organize a set of related jiras. This
> could
> > be easier to manage than using a regular JIRA and call it "umbrella".
> >
> > 2. GitHub Actions
> > I am seeing more projects moving to GitHub Actions for precommits. We
> don't
> > necessarily need to migrate off Jenkins, but there are nice add-ons that
> > can perform static analysis, catching potential issues. For example,
> Ozone
> > adds SonarQube to post-commit, and exports the report to SonarCloud.
> Other
> > add-ons are available to scan for docker images, vulnerabilities scans.
> >
> > 3. JIRA security
> > It is possible to set up security level (public/private) in JIRA. This
> can
> > be used to track vulnerability issues and be made only visible to
> > committers. Example: INFRA-15258
> > 
> >
> > 4. New JIRA fields
> > It's possible to add new fields. For example, we can add a "Reviewer"
> > field, which could help improve the attention to issues.
> >
> > 5. Doc update
> > It is possible to set up automation such that the doc on the Hadoop
> website
> > is refreshed for every commit, providing the latest doc to the public.
> >
> > 6. Webhook
> > It's possible to set up webhook such that every commit in GitHub sends a
> > notification to the ASF slack. It can be used for other kinds of
> > automation. Sky's the limit.
> >
> > Thoughts? What else can do we?
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Hadoop 3.2.3 release

[Steve Loughran]

I'm testing how well a backport of the latest AWS SDK goes; if all is
docile there then I'll merge that. Due diligence more than anything
elseone of the shaded netty JARs (never used by our code) has a CVE in
older versions

On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka  wrote:

> Hi Steven,
>
> Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I
> opened a PR to backport YARN-8990:
> https://github.com/apache/hadoop/pull/3254
>
> Thanks,
> Akira
>
> On Thu, Jul 29, 2021 at 10:36 AM Steven Rand 
> wrote:
> >
> > I think it would be helpful if we could include YARN-8990 and YARN-8992
> in the 3.2.3 release. Both are important fixes which were included in
> 3.2.0, but never made their way to branch-3.2, so were omitted from both
> 3.2.1 and 3.2.2.
> >
> > Best,
> > Steve
> >
> > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He  wrote:
> >>
> >> cc @dev mail-list.
> >>
> >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He 
> wrote:
> >>
> >> > Hi Brahma,
> >> >
> >> > I just created version 3.2.4, and changed all unresolved issues
> (target
> >> > version/s: 3.2.3) to 3.2.4 after checking both of them are not blocker
> >> > issues. Dashboard[1] is clean now.
> >> >
> >> > Regards,
> >> > - He Xiaoqiao
> >> >
> >> > [1]
> >> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
> >> >
> >> > On Sun, Jul 25, 2021 at 7:45 PM Brahma Reddy Battula <
> bra...@apache.org>
> >> > wrote:
> >> >
> >> >> Hi Xiaoqiao,
> >> >>
> >> >> Thanks for creating the Dashboard, we need to change the filters and
> >> >> target versions in the jira.
> >> >>
> >> >> On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He 
> wrote:
> >> >>
> >> >>> Thanks Brahma for volunteering and driving this release plan. I just
> >> >>> created a dashboard for 3.2.3 release[1].
> >> >>> I would like to support for this release line if need. (cc Brahma)
> >> >>>
> >> >>> Thanks. Regards,
> >> >>> - He Xiaoqiao
> >> >>>
> >> >>> [1]
> >> >>>
> >> >>>
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
> >> >>>
> >> >>>
> >> >>> On Sat, Jul 24, 2021 at 1:16 AM Akira Ajisaka 
> >> >>> wrote:
> >> >>>
> >> >>> > Hi Brahma,
> >> >>> >
> >> >>> > Thank you for volunteering!
> >> >>> >
> >> >>> > -Akira
> >> >>> >
> >> >>> > On Fri, Jul 23, 2021 at 5:57 PM Brahma Reddy Battula <
> >> >>> bra...@apache.org>
> >> >>> > wrote:
> >> >>> > >
> >> >>> > > Hi Akira,
> >> >>> > >
> >> >>> > > Thanks for bringing this..
> >> >>> > >
> >> >>> > > I want to drive this if nobody already plan to do this..
> >> >>> > >
> >> >>> > >
> >> >>> > > On Thu, 22 Jul 2021 at 8:48 AM, Akira Ajisaka <
> aajis...@apache.org>
> >> >>> > wrote:
> >> >>> > >
> >> >>> > > > Hi all,
> >> >>> > > >
> >> >>> > > > Hadoop 3.2.2 was released half a year ago, and now, we have
> >> >>> > > > accumulated more than 230 commits [1]. Therefore I want to
> start
> >> >>> the
> >> >>> > > > release work for 3.2.3.
> >> >>> > > >
> >> >>> > > > There is one blocker for 3.2.3 [2].
> >> >>> > > > - https://issues.apache.org/jira/browse/HDFS-12920
> >> >>> > > >
> >> >>> > > > Is there anyone who would volunteer to be the 3.2.3 release
> >> >>> manager?
> >> >>> > > > Are there any other blockers? If any, please file an issue,
> raise
> >> >>> the
> >> >>> > > > blocker, and add the target version.
> >> >>> > > >
> >> >>> > > > [1]
> >> >>> > > >
> >> >>> >
> >> >>>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%203.2.3
> >> >>> > > > [2]
> >> >>> > > >
> >> >>> >
> >> >>>
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20cf%5B12310320%5D%20%3D%203.2.3
> >> >>> > > >
> >> >>> > > > Regards,
> >> >>> > > > Akira
> >> >>> > > >
> >> >>> > > >
> >> >>>
> -
> >> >>> > > > To unsubscribe, e-mail:
> >> >>> mapreduce-dev-unsubscr...@hadoop.apache.org
> >> >>> > > > For additional commands, e-mail:
> >> >>> mapreduce-dev-h...@hadoop.apache.org
> >> >>> > > >
> >> >>> > > > --
> >> >>> > >
> >> >>> > >
> >> >>> > >
> >> >>> > > --Brahma Reddy Battula
> >> >>> >
> >> >>> >
> -
> >> >>> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> >>> > For additional commands, e-mail:
> common-dev-h...@hadoop.apache.org
> >> >>> >
> >> >>> >
> >> >>>
> >> >>
> >> >>
> >> >> --
> >> >>
> >> >>
> >> >>
> >> >> --Brahma Reddy Battula
> >> >>
> >> >
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Hadoop 3.2.3 release

[Steve Loughran]

it didn't take, so don't worry about it. Test failures

java.lang.IllegalArgumentException: Input is expected to be encoded in
multiple of 4 bytes but found: 75
at com.amazonaws.util.Base64Codec.decode(Base64Codec.java:198)
at com.amazonaws.util.Base64.decode(Base64.java:116)
at
com.amazonaws.services.s3.AmazonS3Client.populateSSE_C(AmazonS3Client.java:4620)
at
com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1806)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.putObjectDirect(S3AFileSystem.java:1603)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$createEmptyObject$13(S3AFileSystem.java:2828)
at org.apache.hadoop.fs.s3a.Invoker.once(Invoker.java:109)
at org.apache.hadoop.fs.s3a.Invoker.lambda$retry$3(Invoker.java:265)
at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:322)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:261)
at org.apache.hadoop.fs.s3a.Invoker.retry(Invoker.java:236)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.createEmptyObject(S3AFileSystem.java:2826)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.createFakeDirectory(S3AFileSystem.java:2801)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerMkdirs(S3AFileSystem.java:2146)
at org.apache.hadoop.fs.s3a.S3AFileSystem.mkdirs(S3AFileSystem.java:2079)
at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2304)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.mkdirs(AbstractFSContractTestBase.java:338)
at
org.apache.hadoop.fs.contract.AbstractFSContractTestBase.setup(AbstractFSContractTestBase.java:193)

On Tue, 3 Aug 2021 at 19:41, Brahma Reddy Battula  wrote:

> Hi Steve,
>
> Is there any jira for this ??
>
>
>
> On Tue, 3 Aug 2021 at 4:05 PM, Steve Loughran  >
> wrote:
>
> > I'm testing how well a backport of the latest AWS SDK goes; if all is
> > docile there then I'll merge that. Due diligence more than anything
> > elseone of the shaded netty JARs (never used by our code) has a CVE
> in
> > older versions
> >
> > On Mon, 2 Aug 2021 at 08:07, Akira Ajisaka  wrote:
> >
> > > Hi Steven,
> > >
> > > Marked YARN-8990 and YARN-8992 as release-blocker. In addition, I
> > > opened a PR to backport YARN-8990:
> > > https://github.com/apache/hadoop/pull/3254
> > >
> > > Thanks,
> > > Akira
> > >
> > > On Thu, Jul 29, 2021 at 10:36 AM Steven Rand 
> > > wrote:
> > > >
> > > > I think it would be helpful if we could include YARN-8990 and
> YARN-8992
> > > in the 3.2.3 release. Both are important fixes which were included in
> > > 3.2.0, but never made their way to branch-3.2, so were omitted from
> both
> > > 3.2.1 and 3.2.2.
> > > >
> > > > Best,
> > > > Steve
> > > >
> > > > On Wed, Jul 28, 2021 at 5:14 AM Xiaoqiao He 
> > wrote:
> > > >>
> > > >> cc @dev mail-list.
> > > >>
> > > >> On Wed, Jul 28, 2021 at 5:11 PM Xiaoqiao He 
> > > wrote:
> > > >>
> > > >> > Hi Brahma,
> > > >> >
> > > >> > I just created version 3.2.4, and changed all unresolved issues
> > > (target
> > > >> > version/s: 3.2.3) to 3.2.4 after checking both of them are not
> > blocker
> > > >> > issues. Dashboard[1] is clean now.
> > > >> >
> > > >> > Regards,
> > > >> > - He Xiaoqiao
> > > >> >
> > > >> > [1]
> > > >> >
> > >
> >
> https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12336167
> > > >> >
> > > >> > On Sun, Jul 25, 2021 at 7:45 PM Brahma Reddy Battula <
> > > bra...@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> >> Hi Xiaoqiao,
> > > >> >>
> > > >> >> Thanks for creating the Dashboard, we need to change the filters
> > and
> > > >> >> target versions in the jira.
> > > >> >>
> > > >> >> On Sun, Jul 25, 2021 at 2:05 PM Xiaoqiao He  >
> > > wrote:
> > > >> >>
> > > >> >>> Thanks Brahma for volunteering and driving this release plan. I
> > just
> > > >> >>> created a dashboard for 3.2.3 release[1].
> > > >> >>> I would like to support for this release line if need. (cc
> Brahma)
> > > >> >>>
> > > >> >>> Thanks. Regards,
> > > >> >>> - He Xiaoqiao
> > > >> >>&g

Re: How to access Azure blob storage using Hadoop SDK 3.3.1 and Azure MSI authentication

[Steve Loughran]

On Tue, 2 Nov 2021 at 21:16, Sergey Shabalov <
sergey.shaba...@metaintegration.com> wrote:

> I try to use a simple code
>
>
>1.  conf.set("fs.defaultFS","wasbs://
> sshcon...@sshblobhierarchyoff.blob.core.windows.net");
>2.  conf.set("fs.azure.ssl.channel.mode","Default_JSSE");
>3.
> conf.set("fs.azure.account.oauth.provider.type","org.apache.hadoop.fs.azurebfs.oauth2.MsiTokenProvider");
>

that's an abfs class


>4.  conf.set("fs.azure.account.auth.type","OAuth");
>5.  conf.set("fs.azure.account.oauth2.msi.tenant","My Tenant
> ID");
>6.  conf.set("fs.azure.account.oauth2.client.id","My Client
> ID");
>7.  FileSystem fs = org.apache.hadoop.fs.FileSystem.get(conf);
>
> i think you;re trying to use abfs config options with a wasb url.

the abfs connector can connect to wasb stores -it just doesn't give you
"real" directories


>
> but I have got:
>
> org.apache.hadoop.fs.azure.AzureException:
> org.apache.hadoop.fs.azure.AzureException: No credentials found for account
> sshblobhierarchyoff.blob.core.windows.net in the configuration, and its
> container sshcont01 is not accessible using anonymous credentials. Please
> check if the container exists first. If it is not publicly available, you
> have to provide account credentials.
> Error [18:01:18] at
>
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1098)
> Error [18:01:18] at
>
> org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.initialize(AzureNativeFileSystemStore.java:547)
> Error [18:01:18] at
>
> org.apache.hadoop.fs.azure.NativeAzureFilSystem.initialize(NativeAzureFileSystem.java:1379)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3469)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
> Error [18:01:18] at
> org.apache.hadoop.fs.FileSystem.get(FileSystem.java:288)
>
> But if I set "access level: "container" for this container it passes well.
> We suppose the MSI should provide access to the container without making it
> public.
>
> BTW: MSI access to Azure Data Lake works properly
>
>
> Thanks,
>
> Sergiy
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC0

[Steve Loughran]

I'll do my best to test this; I'm a bit broken right now.

I think we should mention in a release notes that is the version of a log4j
included in this and all previous releases is not vulnerable. But provide a
list plus links to any that have been fixed

On Fri, 10 Dec 2021 at 02:09, Chao Sun  wrote:

> Hi all,
>
> Sorry for the long delay. I've prepared RC0 for Hadoop 3.3.2 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC0/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC0
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1330/
>
> You can find my public key at: https://people.apache.org/~sunchao/KEYS
>
> Please evaluate the RC and vote.
>
> Thanks,
> Chao
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC0

[Steve Loughran]

On Tue, 14 Dec 2021 at 22:56, Chao Sun  wrote:

> Thanks all for taking a look! looks like we need another RC addressing the
> following issues.
>
> > 1. the overview page of the doc is for the Hadoop 3.0 release. It would
> be best to base the doc on top of Hadoop 3.3.0 overview page. (it's a miss
> on my part... The overview page of 3.3.1 wasn't updated)
>
> For this, I just need to update
> the hadoop-project/src/site/markdown/index.md.vm and incorporate notable
> changes made in 3.3.1/3.3.2, is that correct? looks like the file hasn't
> been touched for a while.
>
> > 2. ARM binaries is not included. For the 3.3.1 release, I had to run the
> create release script on an ARM machine separately to create the binary
> tarball.
>
> Hmm this might be challenging for me. Could you share the steps of how you
> did it? especially where did you get an ARM machine.
>
> > 3. the jdiff version
> https://github.com/apache/hadoop/blob/branch-3.3.2/hadoop-project-dist/pom.xml#L137
>
> I just need to backport this commit:
> https://github.com/apache/hadoop/commit/a77bf7cf07189911da99e305e3b80c589edbbfb5
> to branch-3.3.2 (and potentially branch-3.3)?
>
> > The 3.3.1 binary tarball is 577mb. The 3.3.2 RC0 is 608mb. I'm curious
> what are added.
>
> The difference is mostly in aws-java-sdk-bundle jar: 3.3.1 uses
> https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.901
> while 3.3.2 uses
> https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-bundle/1.11.1026.
> The difference is ~32.5mb.
>
> Chao
>
>>
>>> Wow, that's a big change. But the bundling of a shaded jar avoids so
many problems we've had in the past. Come January I will look at moving to
the 1.12 SDK; I don't know how long that will take to stabilize.

Fwd: [jira] [Created] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5 detected in hdfs-client

[Steve Loughran]

okhttp was last updated in 2017

why use this over httpclient? its only used in a couple of places and
removing it entirely would make this problem go away forever

-- Forwarded message -
From: Eugene Shinn (Truveta) (Jira) 
Date: Wed, 5 Jan 2022 at 19:48
Subject: [jira] [Created] (HADOOP-18069) CVE-2021-0341 in okhttp@2.7.5
detected in hdfs-client
To: 


Eugene Shinn (Truveta) created HADOOP-18069:
---

 Summary: CVE-2021-0341 in okhttp@2.7.5 detected in
hdfs-client
 Key: HADOOP-18069
 URL: https://issues.apache.org/jira/browse/HADOOP-18069
 Project: Hadoop Common
  Issue Type: Bug
  Components: hdfs-client
Affects Versions: 3.3.1
Reporter: Eugene Shinn (Truveta)


Our static vulnerability scanner (Fortify On Demand) detected [NVD -
CVE-2021-0341 (nist.gov)|
https://nvd.nist.gov/vuln/detail/CVE-2021-0341#VulnChangeHistorySection] in
our application. We traced the vulnerability to a transitive dependency
coming from hadoop-hdfs-client, which depends on okhttp@2.7.5
([hadoop/pom.xml at trunk · apache/hadoop (github.com)|
https://github.com/apache/hadoop/blob/trunk/hadoop-project/pom.xml#L137]).
To resolve this issue, okhttp should be upgraded to 4.9.2+ (ref:
[CVE-2021-0341 · Issue #6724 · square/okhttp (github.com)|
https://github.com/square/okhttp/issues/6724]).



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: ElasticByteBufferPool is ever growing thus can cause memory leak.

[Steve Loughran]

DirectBufferPool does the weak refs effectively, but it only recycles
buffers of the same size; the elastic one will return the first largest
buffer.

I think we may want to have a variant of the ElasticBufferPool which uses
weak refs..we can adopt that and leave the (old) one alone. It is marked as
public and stable after all

I would also like a new method ByteBufferPool.release() which, when
implementations support it, does a cleanup -essentially
resetting the maps. interface "default" methods make it safer to add these.

Handling buffet recycling really matters for the new vectored IO work which
is very ready to go in.
I want to create a stabilisation branch in the ASF repo with the code as
is, then mukund can add a new PR with the buffer pooling
improvements...that will make it easier for people to cherry pick if they
want.

On Wed, 12 Jan 2022 at 18:23, Chris Nauroth  wrote:

> Thanks for discussing this, Mukund.
>
> Another difference between the 2 classes is that ElasticByteBufferPool
> supports a caller's preference for either on-heap or off-heap buffers and
> internally calls either allocate or allocateDirect. Do you intend to
> preserve that functionality too in a single merged class?
>
> Chris Nauroth
>
>
> On Wed, Jan 12, 2022 at 1:25 AM Mukund Madhav Thakur
>  wrote:
>
> > Hello Everyone,
> > I was just browsing through the code while doing my Vectored IO stuff. It
> > seems like ElasticByteBufferPool is an ever growing pool and memory is
> not
> > getting released as there is no WeakReference being maintained in the
> pool.
> > This can cause memory leaks in the production environment.  This is
> widely
> > used in places like StripedReconstructor,
> > DFSStripedInputStream, BlockBlobAppendStream etc.
> >
> > I would suggest we use DirectBufferPool class for direct buffer pooling
> as
> > it already is keeping WeakReference for the buffers. Although, we will
> have
> > to make this implement the ByteBufferPool interface and implement the
> > corresponding methods. Happy to make the API changes once finalized.
> >
> > Thanks,
> > Mukund
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

[Steve Loughran]

thanks, i'm on it..will run the aws and azure tests and then play with the
artifacts

On Wed, 19 Jan 2022 at 17:50, Chao Sun  wrote:

> Hi all,
>
> I've put together Hadoop 3.3.2 RC2 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1332
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> I've done the following tests and they look good:
> - Ran all the unit tests
> - Started a single node HDFS cluster and tested a few simple commands
> - Ran all the tests in Spark using the RC2 artifacts
>
> Please evaluate the RC and vote, thanks!
>
> Best,
> Chao
>

Re: [DISCUSS] Migrate hadoop from log4j1 to log4j2

[Steve Loughran]

On Thu, 20 Jan 2022 at 17:15, Andrew Purtell 
wrote:

> Just to clarify: I think you want to upgrade to Log4J2 (or switch to
> LogBack) as a strategy for new releases, but you have the option in
> maintenance releases to use Reload4J to maintain Appender API and
> operational compatibility, and users who want to minimize risks in
> production while mitigating the security issues will prefer that.


i like this



>
>
> > On Jan 20, 2022, at 8:59 AM, Andrew Purtell 
> wrote:
> >
> > Reload4J has fixed all of those CVEs without requiring an upgrade.
> >
> >> On Jan 20, 2022, at 5:56 AM, Duo Zhang  wrote:
> >>
> >> There are 3 new CVEs for log4j1 reported recently[1][2][3]. So I think
> it
> >> is time to speed up the migration to log4j2 work[4] now.
> >>
> >> You can see the discussion on the jira issue[4], our goal is to fully
> >> migrate to log4j2 and the current most blocking issue is lack of the
> >> "log4j.rootLogger=INFO,Console" grammer support for log4j2. I've already
> >> started a discussion thread on the log4j dev mailing list[5] and the
> result
> >> is optimistic and I've filed an issue for log4j2[6], but I do not think
> it
> >> could be addressed and released soon. If we want to fully migrate to
> >> log4j2, then either we introduce new environment variables or split the
> old
> >> HADOOP_ROOT_LOGGER variable in the startup scripts. And considering the
> >> complexity of our current startup scripts, the work is not easy and it
> will
> >> also break lots of other hadoop deployment systems if they do not use
> our
> >> startup scripts...
> >>
> >> So after reconsidering the current situation, I prefer we use the
> log4j1.2
> >> bridge to remove the log4j1 dependency first, and once LOG4J2-3341 is
> >> addressed and released, we start to fully migrate to log4j2. Of course
> we
> >> have other problems for log4j1.2 bridge too, as we have TaskLogAppender,
> >> ContainerLogAppender and ContainerRollingLogAppender which inherit
> >> FileAppender and RollingFileAppender in log4j1, which are not part of
> the
> >> log4j1.2 bridge. But anyway, at least we could just copy the source
> code to
> >> hadoop as we have WriteAppender in log4j1.2 bridge, and these two
> classes
> >> do not have related CVEs.
> >>
> >> Thoughts? For me I would like us to make a new 3.4.x release line to
> remove
> >> the log4j1 dependencies ASAP.
> >>
> >> Thanks.
> >>
> >> 1. https://nvd.nist.gov/vuln/detail/CVE-2022-23302
> >> 2. https://nvd.nist.gov/vuln/detail/CVE-2022-23305
> >> 3. https://nvd.nist.gov/vuln/detail/CVE-2022-23307
> >> 4. https://issues.apache.org/jira/browse/HADOOP-16206
> >> 5. https://lists.apache.org/thread/gvfb3jkg6t11cyds4jmpo7lrswmx28w3
> >> 6. https://issues.apache.org/jira/browse/LOG4J2-3341
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

[Steve Loughran]

*+1 binding.*

reviewed binaries, source, artifacts in the staging maven repository in
downstream builds. all good.

*## test run*

checked out the asf github repo at commit 6da346a358c into a location
already set up with aws and azure test credentials

ran the hadoop-aws tests with -Dparallel-tests -DtestsThreadCount=6
 -Dmarkers=delete -Dscale
and hadoop-azure against azure cardiff with -Dparallel-tests=abfs
-DtestsThreadCount=6

all happy



*## binary*
downloaded KEYS and imported, so adding your key to my list (also signed
this and updated the key servers)

downloaded rc tar and verified
```
> gpg2 --verify hadoop-3.3.2.tar.gz.asc hadoop-3.3.2.tar.gz
gpg: Signature made Sat Jan 15 23:41:10 2022 GMT
gpg:using RSA key DE7FA241EB298D027C97B2A1D8F1A97BE51ECA98
gpg: Good signature from "Chao Sun (CODE SIGNING KEY) "
[full]


> cat hadoop-3.3.2.tar.gz.sha512
SHA512 (hadoop-3.3.2.tar.gz) =
cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d

> shasum -a 512 hadoop-3.3.2.tar.gz
cdd3d9298ba7d6e63ed63f93c159729ea14d2b7d5e3a0640b1761c86c7714a721f88bdfa8cb1d8d3da316f616e4f0ceaace4f32845ee4441e6aaa7a12b8c647d
 hadoop-3.3.2.tar.gz
```


*# cloudstore against staged artifacts*
```
cd ~/.m2/repository/org/apache/hadoop
find . -name \*3.3.2\* -print | xargs rm -r
```
ensures no local builds have tainted the repo.

in cloudstore mvn build without tests
```
mci -Pextra -Phadoop-3.3.2 -Psnapshots-and-staging
```
this fetches all from asf staging

```
Downloading from ASF Staging:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom
Downloaded from ASF Staging:
https://repository.apache.org/content/groups/staging/org/apache/hadoop/hadoop-client/3.3.2/hadoop-client-3.3.2.pom
(11 kB at 20 kB/s)
```
there's no tests there, but it did audit the download process. FWIW, that
project has switched to logback, so I now have all hadoop imports excluding
slf4j and log4j. it takes too much effort right now.

build works.

tested abfs and s3a storediags, all happy




*### google GCS against staged artifacts*

gcs is now java 11 only, so I had to switch JVMs here.

had to add a snapshots and staging profile, after which I could build and
test.

```
 -Dhadoop.three.version=3.3.2 -Psnapshots-and-staging
```
two test failures were related to auth failures where the tests were trying
to raise exceptions but things failed differently
```
[ERROR] Failures:
[ERROR]
GoogleHadoopFileSystemTest.eagerInitialization_fails_withInvalidCredentialsConfiguration:122
unexpected exception type thrown; expected:
but was:
[ERROR]
GoogleHadoopFileSystemTest.lazyInitialization_deleteCall_fails_withInvalidCredentialsConfiguration:100
value of: throwable.getMessage()
expected: Failed to create GCS FS
but was : A JSON key file may not be specified at the same time as
credentials via configuration.

```

I'm not worried here.

ran cloudstore's diagnostics against gcs.

Nice to see they are now collecting IOStatistics on their input streams. we
really need to get this collected through the parquet/orc libs and then
through the query engines.

```
> bin/hadoop jar $CLOUDSTORE storediag gs://stevel-london/

...
2022-01-20 17:52:47,447 [main] INFO  diag.StoreDiag
(StoreDurationInfo.java:(56)) - Starting: Reading a file
gs://stevel-london/dir-9cbfc774-76ff-49c0-b216-d7800369c3e1/file
input stream summary: org.apache.hadoop.fs.FSDataInputStream@6cfd9a54:
com.google.cloud.hadoop.fs.gcs.GoogleHadoopFSInputStream@78c1372d{counters=((stream_read_close_operations=1)
(stream_read_seek_backward_operations=0) (stream_read_total_bytes=7)
(stream_read_bytes=7) (stream_read_exceptions=0)
(stream_read_seek_operations=0) (stream_read_seek_bytes_skipped=0)
(stream_read_operations=3) (stream_read_bytes_backwards_on_seek=0)
(stream_read_seek_forward_operations=0)
(stream_read_operations_incomplete=1));
gauges=();
minimums=();
maximums=();
means=();
}
...
```

*### source*

once I'd done builds and tests which fetched from staging, I did a local
build and test

repeated download/validate of source tarball, unzip/untar

build with java11.

I've not done the test run there, because that directory tree doesn't have
the credentials, and this mornings run was good.

altogether then: very happy. tests good, downstream libraries building and
linking.

On Wed, 19 Jan 2022 at 17:50, Chao Sun  wrote:

> Hi all,
>
> I've put together Hadoop 3.3.2 RC2 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1332
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> I've done the following tests and they look good:
> - Ran all the unit tests
> - Started a single node HDFS cluster and tested

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

[Steve Loughran]

`now some bad news
https://issues.apache.org/jira/browse/HADOOP-18091
S3A auditing leaks memory through ThreadLocal references

surfaces in processes with long lived threads creating and destroying many
s3a FS instances.

working on a fix right now

On Fri, 21 Jan 2022 at 21:02, Eric Payne 
wrote:

> +1 (binding)
>
> - Built from source
>
> - Brought up a non-secure virtual cluster w/ NN, 1 DN, RM, AHS, JHS, and 3
> NMs
>
> - Validated inter- and intra-queue preemption
>
> - Validated exclusive node labels
>
> Thanks a lot Chao for your diligence and hard work on this release.
>
> Eric
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wednesday, January 19, 2022, 11:50:34 AM CST, Chao Sun <
> sunc...@apache.org> wrote:
>
>
>
>
>
> Hi all,
>
> I've put together Hadoop 3.3.2 RC2 below:
>
> The RC is available at:
> http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/
> The RC tag is at:
> https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2
> The Maven artifacts are staged at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1332
>
> You can find my public key at:
> https://downloads.apache.org/hadoop/common/KEYS
>
> I've done the following tests and they look good:
> - Ran all the unit tests
> - Started a single node HDFS cluster and tested a few simple commands
> - Ran all the tests in Spark using the RC2 artifacts
>
> Please evaluate the RC and vote, thanks!
>
> Best,
> Chao
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

[Steve Loughran]

fix is in t disable auditing, which is now the default
https://issues.apache.org/jira/browse/HADOOP-18094

everything is OK for apps which retain the same fs instances for the life
of the app, but not for Hive...

will do a better fix ASAP where in exchange for loss of auditing after a GC
event, only weak refs are held in maps private to the auditor.

i will put that in hadoop common as i would want to use the same code in
thread-levek IOStatistics tracking.
there we;d demand create an IOStatistics snapshot per thread,  short lived
worker threads for stream io would still update the stats of the thread the
stream was created in. this will let lus collect stats on store io through
the orc/paquet readers for each thread doing work for a job, and include
them in job stats.

and how would that be useful? well. look at this coimparison of job/task
commit performance with the manifest committer
https://gist.github.com/steveloughran/7dc1e68220db67327b781b345b42c0b8

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC2

[Steve Loughran]

that error
org.restlet.jee:org.restlet:pom:2.3.0 from/to maven-default-http-blocker (
http://0.0.0.0/): Blocked mirror for repositories: [maven-restlet

implies maven is not downloading http artifacts, and it had decided that
the reslet artifacts were coming off an http repo, even though its in maven
central

which means look at your global maven settings



On Tue, 25 Jan 2022 at 07:27, Mukund Madhav Thakur
 wrote:

> Hi Chao,
> I was using the command "mvn package -Pdist -DskipTests -Dtar
> -Dmaven.javadoc.skip=true" on commit id *6da346a358c. *
> It is working for me today. So maybe it was an intermittent issue in my
> local last time when I was trying this. So we can ignore this. Thanks
>
>
>
> On Tue, Jan 25, 2022 at 6:21 AM Stack  wrote:
>
> > +1 (binding)
> >
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (1.8.0_191): ok
> >  - mvn clean apache-rat:check
> > * Built from source (1.8.0_191): ok
> >  - mvn clean install  -DskipTests
> >
> > Poking around in the binary, it looks good. Unpacked site. Looks right.
> > Checked a few links work.
> >
> > Deployed over ten node cluster. Ran HBase ITBLL over it for a few hours
> w/
> > chaos. Worked like 3.3.1...
> >
> > I tried to build with 3.8.1 maven and got the below.
> >
> > [ERROR] Failed to execute goal on project
> > hadoop-yarn-applications-catalog-webapp: Could not resolve dependencies
> for
> > project
> > org.apache.hadoop:hadoop-yarn-applications-catalog-webapp:war:3.3.2:
> Failed
> > to collect dependencies at org.apache.solr:solr-core:jar:7.7.0 ->
> > org.restlet.jee:org.restlet:jar:2.3.0: Failed to read artifact descriptor
> > for org.restlet.
> > jee:org.restlet:jar:2.3.0: Could not transfer artifact
> > org.restlet.jee:org.restlet:pom:2.3.0 from/to maven-default-http-blocker
> (
> > http://0.0.0.0/): Blocked mirror for repositories: [maven-restlet (
> > http://maven.restlet.org, default, releases+snapshots),
> apache.snapshots (
> > http://repository.apache.org/snapshots, default, disabled)] -> [Help 1]
> >
> > I used 3.6.3 mvn instead (looks like a simple fix).
> >
> > Thanks for packaging up this fat point release Chao Sun.
> >
> > S
> >
> > On Wed, Jan 19, 2022 at 9:50 AM Chao Sun  wrote:
> >
> > > Hi all,
> > >
> > > I've put together Hadoop 3.3.2 RC2 below:
> > >
> > > The RC is available at:
> > > http://people.apache.org/~sunchao/hadoop-3.3.2-RC2/
> > > The RC tag is at:
> > > https://github.com/apache/hadoop/releases/tag/release-3.3.2-RC2
> > > The Maven artifacts are staged at:
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1332
> > >
> > > You can find my public key at:
> > > https://downloads.apache.org/hadoop/common/KEYS
> > >
> > > I've done the following tests and they look good:
> > > - Ran all the unit tests
> > > - Started a single node HDFS cluster and tested a few simple commands
> > > - Ran all the tests in Spark using the RC2 artifacts
> > >
> > > Please evaluate the RC and vote, thanks!
> > >
> > > Best,
> > > Chao
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.2 - RC3

[Steve Loughran]

maybe even before moving the catalog away, we could make it optional

On Fri, 28 Jan 2022 at 08:24, Akira Ajisaka  wrote:

> Thank you Masatake and Chao!
>
> On Fri, Jan 28, 2022 at 5:11 PM Chao Sun  wrote:
>
> > Thanks Masatake and Akira for discovering the issue. I used
> > `dev-support/bin/create-release` which runs `mvn deploy -DskipTests
> > -Pnative -Pdist ...` in a separate container and somehow it didn't hit
> this
> > issue.
> >
> > Let me cherry-pick https://issues.apache.org/jira/browse/YARN-10561 to
> > branch-3.3.2 and start another RC then.
> >
> > Thanks,
> > Chao
> >
> > On Fri, Jan 28, 2022 at 12:01 AM Masatake Iwasaki <
> > iwasak...@oss.nttdata.co.jp> wrote:
> >
> >> Thanks, Akira.
> >>
> >> I confirmed that the issue is fixed in current branch-3.3 containing
> >> YARN-10561.
> >>
> >> On 2022/01/28 14:25, Akira Ajisaka wrote:
> >> > Hi Masatake,
> >> >
> >> > I faced the same error in a clean environment and
> >> https://issues.apache.org/jira/browse/YARN-10561 <
> >> https://issues.apache.org/jira/browse/YARN-10561> should fix this
> issue.
> >> I'll rebase the patch shortly.
> >> >
> >> > By the way, I'm afraid there is no active maintainer in
> >> hadoop-yarn-applications-catalog module. The module is for a sample
> >> application catalog, so I think we can move the module to a separate
> >> repository. Of course, it should be discussed separately.
> >> >
> >> > Thanks and regards,
> >> > Akira
> >> >
> >> > On Fri, Jan 28, 2022 at 1:39 PM Masatake Iwasaki <
> >> iwasak...@oss.nttdata.co.jp >
> wrote:
> >> >
> >> > Thanks for putting this up, Chao Sun.
> >> >
> >> > I got following error on building the RC3 source tarball.
> >> > It is reproducible even in the container launched by
> >> `./start-build-env.sh`.
> >> > There seems to be no relevant diff between release-3.3.2-RC0 and
> >> release-3.3.2-RC3 (and trunk)
> >> > under hadoop-yarn-applications-catalog-webapp.
> >> >
> >> > I guess developers having caches of related artifacts under ~/.m2
> >> did not see this?
> >> >
> >> > ```
> >> > $ mvn clean install -DskipTests -Pnative -Pdist
> >> > ...
> >> > [INFO] Installing node version v8.11.3
> >> > [INFO] Downloading
> >> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz <
> >> https://nodejs.org/dist/v8.11.3/node-v8.11.3-linux-x64.tar.gz> to
> >>
> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
> >> > [INFO] No proxies configured
> >> > [INFO] No proxy was configured, downloading directly
> >> > [INFO] Unpacking
> >>
> /home/centos/.m2/repository/com/github/eirslett/node/8.11.3/node-8.11.3-linux-x64.tar.gz
> >> into
> >>
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp
> >> > [INFO] Copying node binary from
> >>
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/tmp/node-v8.11.3-linux-x64/bin/node
> >> to
> >>
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/node
> >> > [INFO] Installed node locally.
> >> > [INFO] Installing Yarn version v1.7.0
> >> > [INFO] Downloading
> >>
> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz
> >> <
> >>
> https://github.com/yarnpkg/yarn/releases/download/v1.7.0/yarn-v1.7.0.tar.gz
> >
> >> to
> >>
> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
> >> > [INFO] No proxies configured
> >> > [INFO] No proxy was configured, downloading directly
> >> > [INFO] Unpacking
> >>
> /home/centos/.m2/repository/com/github/eirslett/yarn/1.7.0/yarn-1.7.0.tar.gz
> >> into
> >>
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target/node/yarn
> >> > [INFO] Installed Yarn locally.
> >> > [INFO]
> >> > [INFO] --- frontend-maven-plugin:1.11.2:yarn (yarn install) @
> >> hadoop-yarn-applications-catalog-webapp ---
> >> > [INFO] testFailureIgnore property is ignored in non test phases
> >> > [INFO] Running 'yarn ' in
> >>
> /home/centos/srcs/hadoop-3.3.2-src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-catalog/hadoop-yarn-applications-catalog-webapp/target
> >> > [INFO] yarn install v1.7.0
> >> > [INFO] info No lockfile found.
> >> > [INFO] [1/4] Resolving packages...
> >> > [INFO] [2/4] Fetching packages...
> >> > [INFO] error safe-stable-stringify@2.3.1: The engine "node" is
> >> incompatible with this module. Expected version ">=10".
> >> > [INFO] error safe-s

HADOOP-18198. Release Hadoop 3.3.3: hadoop-3.3.2 with CVE fixes

[Steve Loughran]

I've just created a new JIRA and assigned to myself: HADOOP-18198. Release
Hadoop 3.3.3: hadoop-3.3.2 with CVE fixes


https://issues.apache.org/jira/browse/HADOOP-18198

--

Hadoop 3.3.3 is a minor followup release to Hadoop 3.3.2 with

* CVE fixes in Hadoop source
* CVE fixes in dependencies
* replacement of log4j 1.2.17 to reload4j
* some changes which shipped in hadoop 3.2.3 for consistency

--


This is not a release off branch-3.3, it is a fork of 3.3.2 with the
changes.

The next release of branch-3.3 will be numbered hadoop-3.3.4; updating
maven versions and JIRA fix versions is part of this release process.

To get these fixes out fast and avoid any regressions, *I'm not putting
anything else in other than the fixes which shipped in 3.2.4*

For all non-CVE related fixes, consult this process:
https://scarfolk.blogspot.com/2015/08/no-1973-1975.html

I will try and do some ARM binaries too, but I'm not going to make a
commitment. My laptop is now an ARM CPU, so in fact cutting this release
involves me actually building it on a different machine; my previous
laptop, or, if that doesn't work out, some remote server.

as usual, any help testing would be wonderful.

After this, I would like to start planning that 3.3.4 feature release. I
think I will nominate myself as the release engineer there, with help from
colleagues, especially Mehakmeet and Mukund.

-Steve

Re: HADOOP-18198. Release Hadoop 3.3.3: hadoop-3.3.2 with CVE fixes

[Steve Loughran]

I should add that the CVEs in question are minor, unless you are running
Hadoop on windows. given you have to compile the native binaries yourself
for that, that is not something we know anyone actually does in production.

The reload4j fix means that we can get out of the classpath the log4j
vulnerabilities which were never reached in the Hadoop code, but which
audit tools would flag up.

I'd also like to update our shaded protobuf library too



On Mon, 11 Apr 2022 at 14:54, Steve Loughran  wrote:

>
> I've just created a new JIRA and assigned to myself: HADOOP-18198. Release
> Hadoop 3.3.3: hadoop-3.3.2 with CVE fixes
>
>
> https://issues.apache.org/jira/browse/HADOOP-18198
>
> --
>
> Hadoop 3.3.3 is a minor followup release to Hadoop 3.3.2 with
>
> * CVE fixes in Hadoop source
> * CVE fixes in dependencies
> * replacement of log4j 1.2.17 to reload4j
> * some changes which shipped in hadoop 3.2.3 for consistency
>
> --
>
>
> This is not a release off branch-3.3, it is a fork of 3.3.2 with the
> changes.
>
> The next release of branch-3.3 will be numbered hadoop-3.3.4; updating
> maven versions and JIRA fix versions is part of this release process.
>
> To get these fixes out fast and avoid any regressions, *I'm not putting
> anything else in other than the fixes which shipped in 3.2.4*
>
> For all non-CVE related fixes, consult this process:
> https://scarfolk.blogspot.com/2015/08/no-1973-1975.html
>
> I will try and do some ARM binaries too, but I'm not going to make a
> commitment. My laptop is now an ARM CPU, so in fact cutting this release
> involves me actually building it on a different machine; my previous
> laptop, or, if that doesn't work out, some remote server.
>
> as usual, any help testing would be wonderful.
>
> After this, I would like to start planning that 3.3.4 feature release. I
> think I will nominate myself as the release engineer there, with help from
> colleagues, especially Mehakmeet and Mukund.
>
> -Steve
>

[ANNOUNCE] branch-3.3.3 created off 3.3.2; branch-3.3 hadoop.version is now 3.3.4-SNAPSHOT

[Steve Loughran]

There is now a branch-3.3.3 in the repo; this is forked off 3.3.2, *not*
branch-3.3

branch-3.3 has its hadoop version now set to 3.3.4

I've renamed the release version 3.3.3 in JIRA to 3.3.4, so all
fixed/target/affects references have been automatically updated. text
references in JIRAs and PRs will not. If you have JIRAs with 3.3. in the
title, now is the time to change them.

There is now a new 3.3.3 version in JIRA for all fixes only targeted at
this release, and for future bug reports.

The list of JIRAs to go in is under
https://issues.apache.org/jira/browse/HADOOP-18198 ; I've re-opened them
all to track the merge

-Steve

[VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

I have put together a release candidate (rc0) for Hadoop 3.3.3

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/

The git tag is release-3.3.3-RC0, commit d37586cbda3

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1348/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.


   - The critical fixes which shipped in the 3.2.3 release.
   -  CVEs in our code and dependencies
   - Shaded client packaging issues.
   - A switch from log4j to reload4j


reload4j is an active fork of the log4j 1.17 library with the classes which
contain CVEs removed. Even though hadoop never used those classes, they
regularly raised alerts on security scans and concen from users. Switching
to the forked project allows us to ship a secure logging framework. It will
complicate the builds of downstream maven/ivy/gradle projects which exclude
our log4j artifacts, as they need to cut the new dependency instead/as well.

See the release notes for details.

This is my first release through the new docker build process, do please
validate artifact signing &c to make sure it is good. I'll be trying builds
of downstream projects.

We know there are some outstanding issues with at least one library we are
shipping (okhttp), but I don't want to hold this release up for it. If the
docker based release process works smoothly enough we can do a followup
security release in a few weeks.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Steve Loughran]

I'm not enthusiastic here as it not only makes the builds slower, it
reduces the #of builds we can through a day

one thing I am wondering is could we remove java8 support on some branches?

make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and this
summers branch-3.3 release (which I'd rebadge 3.4) would ship as java 11
only.
that would cut buld and test time for those trunk PRs in half...after which
the preospect of building on more than one platform becomes more viable.

On Thu, 5 May 2022 at 15:34, Gautham Banasandra  wrote:

> Hi Hadoop devs,
>
> Last week, there was a Hadoop build failure on Debian 10 caused by
> https://github.com/apache/hadoop/pull/3988. In dev-support/jenkins.sh,
> there's the capability to build and test Hadoop across the supported
> platforms. Currently, we're limiting this only for those PRs having only
> C/C++ changes[1], since C/C++ changes are more likely to cause
> cross-platform build issues and bypassing the full platform build for non
> C/C++ PRs would save a great deal of CI time. However, the build failure
> caused by PR #3988 motivates me to enable the capability to build and
> test Hadoop for all the supported platforms for ALL the PRs.
>
> While this may cause longer CI run duration for each PR, it would
> immensely minimize the risk of breaking Hadoop across platforms and
> saves us a lot of debugging time. Kindly post your opinion regarding this
> and I'll move to enable this capability for all PRs if the response is
> sufficiently positive.
>
> [1] =
>
> https://github.com/apache/hadoop/blob/bccf2f3ef4c8f09f010656f9061a4e323daf132b/dev-support/jenkins.sh#L97-L103
>
>
> Thanks,
> --Gautham
>

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

api-3.3.3.jar
> > >>> META-INF/
> > >>> META-INF/MANIFEST.MF
> > >>> META-INF/NOTICE.txt
> > >>> META-INF/LICENSE.txt
> > >>> META-INF/maven/
> > >>> META-INF/maven/org.apache.hadoop/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> > >>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> > >>>
> > >>> On Fri, May 6, 2022 at 4:24 PM Stack  wrote:
> > >>>>
> > >>>> +1 (binding)
> > >>>>
> > >>>>  * Signature: ok
> > >>>>  * Checksum : passed
> > >>>>  * Rat check (1.8.0_191): passed
> > >>>>   - mvn clean apache-rat:check
> > >>>>  * Built from source (1.8.0_191): failed
> > >>>>   - mvn clean install  -DskipTests
> > >>>>   - mvn -fae --no-transfer-progress -DskipTests
> > -Dmaven.javadoc.skip=true
> > >>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> > >>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> > >>>>  * Unit tests pass (1.8.0_191):
> > >>>>- HDFS Tests passed (Didn't run more than this).
> > >>>>
> > >>>> Deployed a ten node ha hdfs cluster with three namenodes and five
> > >>>> journalnodes. Ran a ten node hbase (older version of 2.5 branch
> built
> > >>>> against 3.3.2) against it. Tried a small verification job. Good.
> Ran a
> > >>>> bigger job with mild chaos. All seems to be working properly
> > (recoveries,
> > >>>> logs look fine). Killed a namenode. Failover worked promptly. UIs
> look
> > >>>> good. Poked at the hdfs cli. Seems good.
> > >>>>
> > >>>> S
> > >>>>
> > >>>> On Tue, May 3, 2022 at 4:24 AM Steve Loughran
> > 
> > >>>> wrote:
> > >>>>
> > >>>>> I have put together a release candidate (rc0) for Hadoop 3.3.3
> > >>>>>
> > >>>>> The RC is available at:
> > >>>>> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/
> > >>>>>
> > >>>>> The git tag is release-3.3.3-RC0, commit d37586cbda3
> > >>>>>
> > >>>>> The maven artifacts are staged at
> > >>>>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > >>>>>
> > >>>>> You can find my public key at:
> > >>>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >>>>>
> > >>>>> Change log
> > >>>>>
> https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/CHANGELOG.md
> > >>>>>
> > >>>>> Release notes
> > >>>>>
> > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC0/RELEASENOTES.md
> > >>>>>
> > >>>>> There's a very small number of changes, primarily critical
> > code/packaging
> > >>>>> issues and security fixes.
> > >>>>>
> > >>>>>
> > >>>>>   - The critical fixes which shipped in the 3.2.3 release.
> > >>>>>   -  CVEs in our code and dependencies
> > >>>>>   - Shaded client packaging issues.
> > >>>>>   - A switch from log4j to reload4j
> > >>>>>
> > >>>>>
> > >>>>> reload4j is an active fork of the log4j 1.17 library with the
> > classes which
> > >>>>> contain CVEs removed. Even though hadoop never used those classes,
> > they
> > >>>>> regularly raised alerts on security scans and concen from users.
> > Switching
> > >>>>> to the forked project allows us to ship a secure logging framework.
> > It will
> > >>>>> complicate the builds of downstream maven/ivy/gradle projects which
> > exclude
> > >>>>> our log4j artifacts, as they need to cut the new dependency
> > instead/as
> > >>>>> well.
> > >>>>>
> > >>>>> See the release notes for details.
> > >>>>>
> > >>>>> This is my first release through the new docker build process, do
> > please
> > >>>>> validate artifact signing &c to make sure it is good. I'll be
> trying
> > builds
> > >>>>> of downstream projects.
> > >>>>>
> > >>>>> We know there are some outstanding issues with at least one library
> > we are
> > >>>>> shipping (okhttp), but I don't want to hold this release up for it.
> > If the
> > >>>>> docker based release process works smoothly enough we can do a
> > followup
> > >>>>> security release in a few weeks.
> > >>>>>
> > >>>>> Please try the release and vote. The vote will run for 5 days.
> > >>>>>
> > >>>>> -Steve
> > >>>>>
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >>>
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
> >
>

Re: [DISCUSS] Enabling all platform builds in CI for all Hadoop PRs

[Steve Loughran]

how about for trunk we -wait for it- declare it java11 only?

do that and we cut out a lot of the CI build.

we would have to mandate a test run through branch-3.3 before any
cherrypick there

On Sat, 7 May 2022 at 11:19, Ayush Saxena  wrote:

> Three for trunk:
> Java-8
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/
>
> Java-11
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/
>
> ARM
>
> https://ci-hadoop.apache.org/view/Hadoop/job/hadoop-qbt-linux-ARM-trunk/
>
> -Ayush
>
> >
> > On 07-May-2022, at 11:49 AM, Gautham Banasandra 
> wrote:
> >
> > Hi all,
> >
> > Although validating cross platform builds at pre-commit
> > would be the most effective approach, I understand the
> > huge disadvantage caused by the slowdown. The best
> > way to tackle this would be to enable parallel builds for
> > the different platforms. I had given it a shot about a year
> > ago[1], it didn't go well and ran into all sorts of random
> > errors. I think we should make the parallel builds run on
> > different agents as opposed to starting the builds parallelly
> > on the same agent (which is what I had done earlier).
> >
> > So, I'll settle down to integrating the full suite of platform
> > builds into the nightly builds. Could anyone please point
> > me to the Jenkins job for this?
> >
> > [1] = https://github.com/apache/hadoop/pull/3166
> >
> > Thanks,
> > --Gautham
> >
> >> On Fri, 6 May 2022 at 21:04, Ayush Saxena  wrote:
> >>
> >> From functional point of view it does makes sense to validate all the
> >> platforms as part of the builds, but the Pre commits builds taking time
> is
> >> now no longer a small things, In past one year or may be two, we have
> >> already increased it more than twice as compared to what it was before,
> If
> >> someone has a change in HDFS, which includes both hdfs-client &
> >> hadoop-hdfs, it takes more than 5 hours, which long back was around 2
> hours.
> >> With the current state, I don't think we should just go and add these
> >> extra overheads. Having them as part of the nightly builds does makes
> sense
> >> for now.
> >>
> >> In future if we feel there is a strong need for this and we start to see
> >> very frequent failures in some other platforms and we are left with no
> >> other option but to integrate it in our pre-commit jobs, we can explore
> >> having these build phases running in parallel, along with trying other
> >> phases also to run in parallel like compilation/javadoc builds of JDK-8
> &
> >> JDK-11 can run in parallel and may be explore other opportunities as
> well
> >> to compensate for this time.
> >>
> >> For now lets just integrate it our nightly builds only and circle back
> >> again here in future if the need be.
> >>
> >> -Ayush
> >>
> >>> On Fri, 6 May 2022 at 20:44, Wei-Chiu Chuang 
> wrote:
> >>>
> >>> Running builds for all platforms for each and every PR seems too
> >>> excessive.
> >>>
> >>> How about doing all platform builds in the nightly jobs?
> >>>
> >>> On Fri, May 6, 2022 at 8:02 AM Steve Loughran
>  >>>>
> >>> wrote:
> >>>
> >>>> I'm not enthusiastic here as it not only makes the builds slower, it
> >>>> reduces the #of builds we can through a day
> >>>>
> >>>> one thing I am wondering is could we remove java8 support on some
> >>> branches?
> >>>>
> >>>> make branch 3.3.2.x (i.e the 3.3.3 release) the last java 8 build, and
> >>> this
> >>>> summers branch-3.3 release (which I'd rebadge 3.4) would ship as java
> 11
> >>>> only.
> >>>> that would cut buld and test time for those trunk PRs in half...after
> >>> which
> >>>> the preospect of building on more than one platform becomes more
> viable.
> >>>>
> >>>> On Thu, 5 May 2022 at 15:34, Gautham Banasandra 
> >>>> wrote:
> >>>>
> >>>>> Hi Hadoop devs,
> >>>>>
> >>>>> Last week, there was a Hadoop build failure on Debian 10 caused by
> >>>>> https://github.com/apache/hadoop/pull/3988. In
> >>> dev-support/jenkins.sh,
> >>>>> there's the capability to build and test Hadoop across the supported
>

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

I didn't do that as the docker image was doing it itself...I discussed this
with Akira and Ayush & they agreed. so whatever went wrong. it was
something else.

I have been building a list of things I'd like to change there; cutting
that line was one of them. but I need to work out the correct workflow.

trying again, and creating a stub module to verify the client is in staging

On Mon, 9 May 2022 at 15:19, Masatake Iwasaki 
wrote:

> It seems to be caused by obsolete instruction in HowToRelease Wiki?
>
> After HADOOP-15058, `mvn deploy` is kicked by
> `dev-support/bin/create-release --asfrelease`.
> https://issues.apache.org/jira/browse/HADOOP-15058
>
> Step #10 in "Creating the release candidate (X.Y.Z-RC)" section
> of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>
> 2 sets of artifact are deployed after creating RC based on the instruction.
> The latest one contains empty shaded jars.
>
> hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> looks having same issue...
>
> Masatake Iwasaki
>
> On 2022/05/08 6:45, Akira Ajisaka wrote:
> > Hi Chao,
> >
> > How about using
> > https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > instead of https://repository.apache.org/content/repositories/staging/ ?
> >
> > Akira
> >
> > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena  wrote:
> >
> >> Hmm, I see the artifacts ideally should have got overwritten by the new
> >> RC, but they didn’t. The reason seems like the staging path shared
> doesn’t
> >> have any jars…
> >> That is why it was picking the old jars. I think Steve needs to run mvn
> >> deploy again…
> >>
> >> Sent from my iPhone
> >>
> >>> On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
> >>>
> >>> 
> >>>>
> >>>> Chao can you use the one that Steve mentioned in the mail?
> >>>
> >>> Hmm how do I do that? Typically after closing the RC in nexus the
> >>> release bits will show up in
> >>>
> >>
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> >>> and Spark build will be able to pick them up for testing. However in
> >>> this case I don't see any 3.3.3 jars in the URL.
> >>>
> >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena 
> wrote:
> >>>>
> >>>> There were two 3.3.3 staged. The earlier one was with skipShade, the
> >> date was also april 22, I archived that. Chao can you use the one that
> >> Steve mentioned in the mail?
> >>>>
> >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun  wrote:
> >>>>>
> >>>>> Seems there are some issues with the shaded client as I was not able
> >>>>> to compile Apache Spark with the RC
> >>>>> (https://github.com/apache/spark/pull/36474). Looks like it's
> compiled
> >>>>> with the `-DskipShade` option and the hadoop-client-api JAR doesn't
> >>>>> contain any class:
> >>>>>
> >>>>> ➜  hadoop-client-api jar tf 3.3.3/hadoop-client-api-3.3.3.jar
> >>>>> META-INF/
> >>>>> META-INF/MANIFEST.MF
> >>>>> META-INF/NOTICE.txt
> >>>>> META-INF/LICENSE.txt
> >>>>> META-INF/maven/
> >>>>> META-INF/maven/org.apache.hadoop/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.xml
> >>>>> META-INF/maven/org.apache.hadoop/hadoop-client-api/pom.properties
> >>>>>
> >>>>> On Fri, May 6, 2022 at 4:24 PM Stack  wrote:
> >>>>>>
> >>>>>> +1 (binding)
> >>>>>>
> >>>>>>   * Signature: ok
> >>>>>>   * Checksum : passed
> >>>>>>   * Rat check (1.8.0_191): passed
> >>>>>>- mvn clean apache-rat:check
> >>>>>>   * Built from source (1.8.0_191): failed
> >>>>>>- mvn clean install  -DskipTests
> >>>>>>- mvn -fae --no-transfer-progress -DskipTests
> >> -Dmaven.javadoc.skip=true
> >>>>>> -Pnative -Drequire.openssl -Drequire.snappy -Drequire.valgrind
> >>>>>> -Drequire.zstd -Drequire.test.libhadoop clean install
> >>>>>>   * Unit tests pass (1.8.0_191):
> >>>>>> - HDFS Tests pass

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

I've done another docker build and the client jars appear to be there. I'll
test tomorrow before putting up another vote. it will be exactly the same
commit as before, just recompiled/republished

On Mon, 9 May 2022 at 17:45, Chao Sun  wrote:

> Agreed, that step #10 is out-dated and should be removed (I skipped
> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>
> > How about using
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>
> Akira, I tried this too but it didn't work. I think we'd need the
> artifacts to be properly pushed to the staging repository.
>
> > Could you please let me know how I can consume the Hadoop 3 jars in
> maven?
>
> Gautham (if you are following this thread), you'll need to add the
> following:
>
> 
>staged
>staged-releases
>https://repository.apache.org/content/repositories/staging/
> 
>
>  true
>
>
>      true
>    
>  
>
> to the `` section in your Maven pom.xml file.
>
> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>  wrote:
> >
> > I didn't do that as the docker image was doing it itself...I discussed
> this
> > with Akira and Ayush & they agreed. so whatever went wrong. it was
> > something else.
> >
> > I have been building a list of things I'd like to change there; cutting
> > that line was one of them. but I need to work out the correct workflow.
> >
> > trying again, and creating a stub module to verify the client is in
> staging
> >
> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
> iwasak...@oss.nttdata.co.jp>
> > wrote:
> >
> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
> > >
> > > After HADOOP-15058, `mvn deploy` is kicked by
> > > `dev-support/bin/create-release --asfrelease`.
> > > https://issues.apache.org/jira/browse/HADOOP-15058
> > >
> > > Step #10 in "Creating the release candidate (X.Y.Z-RC)" section
> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
> > >
> > > 2 sets of artifact are deployed after creating RC based on the
> instruction.
> > > The latest one contains empty shaded jars.
> > >
> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
> > > looks having same issue...
> > >
> > > Masatake Iwasaki
> > >
> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
> > > > Hi Chao,
> > > >
> > > > How about using
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
> > > > instead of
> https://repository.apache.org/content/repositories/staging/ ?
> > > >
> > > > Akira
> > > >
> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena 
> wrote:
> > > >
> > > >> Hmm, I see the artifacts ideally should have got overwritten by the
> new
> > > >> RC, but they didn’t. The reason seems like the staging path shared
> > > doesn’t
> > > >> have any jars…
> > > >> That is why it was picking the old jars. I think Steve needs to run
> mvn
> > > >> deploy again…
> > > >>
> > > >> Sent from my iPhone
> > > >>
> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
> > > >>>
> > > >>> 
> > > >>>>
> > > >>>> Chao can you use the one that Steve mentioned in the mail?
> > > >>>
> > > >>> Hmm how do I do that? Typically after closing the RC in nexus the
> > > >>> release bits will show up in
> > > >>>
> > > >>
> > >
> https://repository.apache.org/content/repositories/staging/org/apache/hadoop
> > > >>> and Spark build will be able to pick them up for testing. However
> in
> > > >>> this case I don't see any 3.3.3 jars in the URL.
> > > >>>
> > > >>>> On Fri, May 6, 2022 at 6:24 PM Ayush Saxena 
> > > wrote:
> > > >>>>
> > > >>>> There were two 3.3.3 staged. The earlier one was with skipShade,
> the
> > > >> date was also april 22, I archived that. Chao can you use the one
> that
> > > >> Steve mentioned in the mail?
> > > >>>>
> > > >>>>> On Sat, 7 May 2022 at 06:18, Chao Sun 
> wrote:
> > > 

Re: [VOTE] Release Apache Hadoop 3.3.3

[Steve Loughran]

update: got a stub project whose test run requires the client api and
shaded artifacts, and whose clean target wiil delete all artifacts of a
specific version from your local repo, so as to ensure all it untainted

https://github.com/steveloughran/validate-hadoop-client-artifacts

I do now have a version in staging with the files; still don't know how the
previous delete lacked them, I will do more testing tomorrow before putting
up the next RC. which will be off the same git commit as before, just a
rebuild, repackage and republish.

-Steve

On Tue, 10 May 2022 at 00:18, Steve Loughran  wrote:

> I've done another docker build and the client jars appear to be there.
> I'll test tomorrow before putting up another vote. it will be exactly the
> same commit as before, just recompiled/republished
>
> On Mon, 9 May 2022 at 17:45, Chao Sun  wrote:
>
>> Agreed, that step #10 is out-dated and should be removed (I skipped
>> that when releasing Hadoop 3.3.2 but didn't update it, sorry).
>>
>> > How about using
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>>
>> Akira, I tried this too but it didn't work. I think we'd need the
>> artifacts to be properly pushed to the staging repository.
>>
>> > Could you please let me know how I can consume the Hadoop 3 jars in
>> maven?
>>
>> Gautham (if you are following this thread), you'll need to add the
>> following:
>>
>> 
>>staged
>>staged-releases
>>https://repository.apache.org/content/repositories/staging/
>> 
>>    
>>  true
>>
>>
>>  true
>>
>>  
>>
>> to the `` section in your Maven pom.xml file.
>>
>> On Mon, May 9, 2022 at 8:52 AM Steve Loughran
>>  wrote:
>> >
>> > I didn't do that as the docker image was doing it itself...I discussed
>> this
>> > with Akira and Ayush & they agreed. so whatever went wrong. it was
>> > something else.
>> >
>> > I have been building a list of things I'd like to change there; cutting
>> > that line was one of them. but I need to work out the correct workflow.
>> >
>> > trying again, and creating a stub module to verify the client is in
>> staging
>> >
>> > On Mon, 9 May 2022 at 15:19, Masatake Iwasaki <
>> iwasak...@oss.nttdata.co.jp>
>> > wrote:
>> >
>> > > It seems to be caused by obsolete instruction in HowToRelease Wiki?
>> > >
>> > > After HADOOP-15058, `mvn deploy` is kicked by
>> > > `dev-support/bin/create-release --asfrelease`.
>> > > https://issues.apache.org/jira/browse/HADOOP-15058
>> > >
>> > > Step #10 in "Creating the release candidate (X.Y.Z-RC)" section
>> > > of the Wiki still instructs to run `mvn deploy` with `-DskipShade`.
>> > >
>> > > 2 sets of artifact are deployed after creating RC based on the
>> instruction.
>> > > The latest one contains empty shaded jars.
>> > >
>> > > hadoop-client-api and hadoop-client-runtime of already released 3.2.3
>> > > looks having same issue...
>> > >
>> > > Masatake Iwasaki
>> > >
>> > > On 2022/05/08 6:45, Akira Ajisaka wrote:
>> > > > Hi Chao,
>> > > >
>> > > > How about using
>> > > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1348/
>> > > > instead of
>> https://repository.apache.org/content/repositories/staging/ ?
>> > > >
>> > > > Akira
>> > > >
>> > > > On Sat, May 7, 2022 at 10:52 AM Ayush Saxena 
>> wrote:
>> > > >
>> > > >> Hmm, I see the artifacts ideally should have got overwritten by
>> the new
>> > > >> RC, but they didn’t. The reason seems like the staging path shared
>> > > doesn’t
>> > > >> have any jars…
>> > > >> That is why it was picking the old jars. I think Steve needs to
>> run mvn
>> > > >> deploy again…
>> > > >>
>> > > >> Sent from my iPhone
>> > > >>
>> > > >>> On 07-May-2022, at 7:12 AM, Chao Sun  wrote:
>> > > >>>
>> > > >>> 
>> > > >>>>
>> > > >>>> Chao can you use the one that Steve mentioned in the mail?
>> > > &g

[VOTE] Release Apache Hadoop 3.3.3 (RC1)

[Steve Loughran]

I have put together a release candidate (RC1) for Hadoop 3.3.3

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/

The git tag is release-3.3.3-RC1, commit d37586cbda3

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1349/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.

* The critical fixes which shipped in the 3.2.3 release.
* CVEs in our code and dependencies
* Shaded client packaging issues.
* A switch from log4j to reload4j

reload4j is an active fork of the log4j 1.17 library with the classes
which contain CVEs removed. Even though hadoop never used those classes,
they regularly raised alerts on security scans and concen from users.
Switching to the forked project allows us to ship a secure logging
framework. It will complicate the builds of downstream
maven/ivy/gradle projects which exclude our log4j artifacts, as they
need to cut the new dependency instead/as well.

See the release notes for details.

This is the second release attempt. It is the same git commit as before, but
fully recompiled with another republish to maven staging, which has bee
verified by building spark, as well as a minimal test project.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.3 (RC1)

[Steve Loughran]

Here are the result of the vote.

Binding PMC members

+1 Stack
+1 Masatake Iwasaki
+1 Xiaoqiao He
+1 Chao Sun
+1 Chris Nauroth


Non binding votes

+1 Ayush Saxena
+1 Viraj Jasani
+1 Mukund Thakur


There were no negative votes, and maven is happy.
I'm pleased to announce therefore, that this release candidate has been
approved to become the final release.

I'm going to go through the next steps of the process: publishing the
artifacts, site docs, maven artifacts, announcements etc.

Thank you to all who reviewed this!

-Steve

PS: I'm going to start some discussion on correcting/tuning the release
process, once I've got through it.

On Mon, 16 May 2022 at 19:16, Chris Nauroth  wrote:

> +1 (binding)
>
> - Verified all checksums.
> - Verified all signatures.
> - Built from source, including native code on Linux.
> - Ran several examples successfully.
>
> Chris Nauroth
>
>
> On Mon, May 16, 2022 at 10:06 AM Chao Sun  wrote:
>
> > +1
> >
> > - Compiled from source
> > - Verified checksums & signatures
> > - Launched a pseudo HDFS cluster and ran some simple commands
> > - Ran full Spark tests with the RC
> >
> > Thanks Steve!
> >
> > Chao
> >
> > On Mon, May 16, 2022 at 2:19 AM Ayush Saxena  wrote:
> > >
> > > +1,
> > > * Built from source.
> > > * Successful native build on Ubuntu 18.04
> > > * Verified Checksums.
> > >
> >
> (CHANGELOG.md,RELEASENOTES.md,hadoop-3.3.3-rat.txt,hadoop-3.3.3-site.tar.gz,hadoop-3.3.3-src.tar.gz,hadoop-3.3.3.tar.gz)
> > > * Verified Signature.
> > > * Successful RAT check
> > > * Ran basic HDFS shell commands.
> > > * Ran basic YARN shell commands.
> > > * Verified version in hadoop version command and UI
> > > * Ran some MR example Jobs.
> > > * Browsed
> UI(Namenode/Datanode/ResourceManager/NodeManager/HistoryServer)
> > > * Browsed the contents of Maven Artifacts.
> > > * Browsed the contents of the website.
> > >
> > > Thanx Steve for driving the release, Good Luck!!!
> > >
> > > -Ayush
> > >
> > > On Mon, 16 May 2022 at 08:20, Xiaoqiao He 
> wrote:
> > >
> > > > +1(binding)
> > > >
> > > > * Verified signature and checksum of the source tarball.
> > > > * Built the source code on Ubuntu and OpenJDK 11 by `mvn clean
> package
> > > > -DskipTests -Pnative -Pdist -Dtar`.
> > > > * Setup pseudo cluster with HDFS and YARN.
> > > > * Run simple FsShell - mkdir/put/get/mv/rm and check the result.
> > > > * Run example mr applications and check the result - Pi & wordcount.
> > > > * Check the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> > etc.
> > > >
> > > > Thanks Steve for your work.
> > > >
> > > > - He Xiaoqiao
> > > >
> > > > On Mon, May 16, 2022 at 4:25 AM Viraj Jasani 
> > wrote:
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > * Signature: ok
> > > > > * Checksum : ok
> > > > > * Rat check (1.8.0_301): ok
> > > > >  - mvn clean apache-rat:check
> > > > > * Built from source (1.8.0_301): ok
> > > > >  - mvn clean install  -DskipTests
> > > > > * Built tar from source (1.8.0_301): ok
> > > > >  - mvn clean package  -Pdist -DskipTests -Dtar
> > -Dmaven.javadoc.skip=true
> > > > >
> > > > > HDFS, MapReduce and HBase (2.5) CRUD functional testing on
> > > > > pseudo-distributed mode looks good.
> > > > >
> > > > >
> > > > > On Wed, May 11, 2022 at 10:26 AM Steve Loughran
> > > > 
> > > > > wrote:
> > > > >
> > > > > > I have put together a release candidate (RC1) for Hadoop 3.3.3
> > > > > >
> > > > > > The RC is available at:
> > > > > > https://dist.apache.org/repos/dist/dev/hadoop/3.3.3-RC1/
> > > > > >
> > > > > > The git tag is release-3.3.3-RC1, commit d37586cbda3
> > > > > >
> > > > > > The maven artifacts are staged at
> > > > > >
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1349/
> > > > > >
> > > > > > You can find my public key at:
> > > > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > > > >
> >

[ANNOUNCE] Apache Hadoop 3.3.3 release

[Steve Loughran]

On behalf of the Apache Hadoop Project Management Committee, I'm pleased to
announce that Hadoop 3.3.3 has been
released:

https://hadoop.apache.org/release/3.3.3.html

This is the third stable release of the Apache Hadoop 3.3 line.

It contains 23 bug fixes, improvements and enhancements since 3.3.2.

This is primarily a security update; for this reason, upgrading is strongly
advised.

Users are encouraged to read the overview of major changes[1] since 3.3.2.
For details of bug fixes, improvements, and other enhancements since the
previous 3.3.2 release,
please check release notes[2] and changelog[3].

[1]: /docs/r3.3.3/index.html
[2]:
http://hadoop.apache.org/docs/r3.3.3/hadoop-project-dist/hadoop-common/release/3.3.3/RELEASENOTES.3.3.3.html
[3]:
http://hadoop.apache.org/docs/r3.3.3/hadoop-project-dist/hadoop-common/release/3.3.3/CHANGELOG.3.3.3.html


As the release notes highlight, this release contains HADOOP-18088 "Replace
log4j 1.x with reload4j"
https://issues.apache.org/jira/browse/HADOOP-18088

This ensures that the version of log4j shipped is free of known CVEs. the
standard log4j 1.2.17 has some known CVEs in classes which were never uses;
reload4j cuts them out. Audit scanning tools should stop highlighting
perceived risks here.

If you are using maven exclusions to manage logging libraries, or were
otherwise replacing the log4j artifacts in deployments, note the different
library/artifact names which need to be handled.

Many thanks to everyone who helped in this release by supplying patches,
reviewing them, helping get this release building and testing reviewing the
final artifacts.

Steve

Re: [VOTE] Release Apache Hadoop 2.10.2 - RC0

[Steve Loughran]

+1 binding

I've extended my validator project
https://github.com/steveloughran/validate-hadoop-client-artifacts

it now
* fetches KEYS
* fetches an RC from a remote url
* validates signing
* untars and tries to build source
* untars binary release and tries some native commands

the source build fails because I'm on a mac M1 and branch-2 doesn't have
the dynamic protoc switching. not sure if it can go in at all.

binary untar worked, basic commands ok except for bin/hadoop checknative
-again, no native libs for this system.

so: as far as all my newly automated process goes, and allowing for mac m1
native code issues, I'm happy.

On Wed, 25 May 2022 at 03:41, Masatake Iwasaki 
wrote:

> Hi all,
>
> Here's Hadoop 2.10.2 release candidate #0:
>
> The RC is available at:
>https://home.apache.org/~iwasakims/hadoop-2.10.2-RC0/
>
> The RC tag is at:
>https://github.com/apache/hadoop/releases/tag/release-2.10.2-RC0
>
> The Maven artifacts are staged at:
>https://repository.apache.org/content/repositories/orgapachehadoop-1350
>
> You can find my public key at:
>https://downloads.apache.org/hadoop/common/KEYS
>
> Please evaluate the RC and vote.
> The vote will be open for (at least) 5 days.
>
> Thanks,
> Masatake Iwasaki
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[DISCUSS] Forthcoming Hadoop releases

[Steve Loughran]

I want to start a quick discussion on a plan for hadoop releases this
summer. I am willing to do the release manager work. Mukund and Mehakmeet
have have already volunteered to help even if they don't know that yet.

I've got two goals

   1. minor followup to 3.3.3
   2. feature release of new stuff


*Followup to 3.3.3, working title "3.3.4"*

I've a PR up on github to add those change to the 3.3.2/3.3.3 line which
have shipped elsewhere and/or we consider critical.

https://github.com/apache/hadoop/pull/4345

This is for critical data integrity/service availability patches; things
like test values we will just triage.

I can start a new release of this at the end of the week, with an RC up
next week ready for review. With the wonderful docker based build and some
extra automation I've been adding for validating releases
(validate-hadoop-client-artifacts), getting that RC out is not that
problematic; issuing git commands is the heavy lifting.

What does take effort is the testing by everybody else; the smaller the set
of changes the more this is limited to validating the artifacts and the
maven publishing.

As it is a follow up to hadoop 3.3.3 then it needs the version number
3.3.4. This raises the question "what about branch-3.3", which brings me to
the next deliverable.

*branch-3.3 => branch-3.4, targeting hadoop 3.4.0 in 3Q22*

With the 3.3.x line being maintained for critical fixes only, make the
hadoop version in branch-3.3 "hadoop-3.4.0" and release later this year.

A release schedule which is probably doable despite people taking time off
over the summer could be

   - feature complete by July/August
   - RC(s) sept/oct with goal of shipping by October


I volunteer to be release manager, albeit with critical help from
colleagues. For people who haven't worked with me on a project release
before, know that I'm fairly ruthless about getting changes in once the
branch is locked down. So get those features in now.

hadoop trunk gets its version number incremented to 3.5.0-SNAPSHOT

It's probably time we think about what a release off trunk would mean -but
t I would like to get a branch-3.3 release out rather than later.

What do people think of this? And is there anyone else willing to get
involved with the release process?

-Steve

[DISCUSS] Filesystem API shim library to assist applications still targeting previous hadoop releases.

[Steve Loughran]

I've just created an initial project "fs-api-shim" to provide controlled
access to the hadoop 3.3.3+ filesystem API calls on hadoop 3.2.0+ releases
https://github.com/steveloughran/fs-api-shim

The goal here is to make it possible for core file format libraries
(Parquet, Avro, ORC, Arrow etc) and other apps (HBase, ...) to take
advantage of those APIs which we have updated and optimised for access to
cloud stores. Currently the applications do not and are under performance
on recent releases. I have the ability to change our internal forks but I
would like to let others gain from the changes and avoid having to diverge
i'll internal libraries too much.

Currently too many libraries seen frozen in time

Avro: still rejecting changes which don't compile on hadoop 2
https://github.com/apache/avro/pull/1431

Parquet: still using reflection to access non hadoop 1.x filesystem API
calls
https://github.com/apache/parquet-mr/pull/971

I'm not going to support hadoop 2.10 —but we can at least say "move up to
hadoop 3.2.x and we will let you use later APIs when available"

some calls, like openFile() will work everywhere; on versions with the open
file builder API they will take the final status and fake policy so let
libraries declare whether they are random/sequential is IO and skip those
HEAD requests on the object stores they do to verify that the file exists
and determine its length for the ranged GET call requests which will follow.

https://github.com/steveloughran/fs-api-shim/blob/main/fs-api-shim-library/src/main/java/org/apache/hadoop/fs/shim/FileSystemShim.java#L38

On Hadoop 3.2.x, or if openFile() fails for some reason, it will just
downgrade to the classic open() call.

Other API calls we can support dynamic binding to through reflection but
not actually fallback if they are unavailable. This will allow libraries to
use the API calls if present but force them to come up with alternative
solutions if not.

A key part of this is FSDataInputStream, where the ByteBufferReadable API
would be benefit to Parquet

https://github.com/steveloughran/fs-api-shim/blob/main/fs-api-shim-library/src/main/java/org/apache/hadoop/fs/shim/FSDataInputStreamShim.java

When we get the vectored IO feature branch in, we can offer similar
reflection-based access. It means applications can compile on hadoop 3.2.x
and 3.3.x but still take advantage of the APIs when they are on a version
without it.

I'm going to stay clear of more complicated APIs which don't offer tangible
performance gains and which are very hard to do (IOStatistics).

Testing is fun; I have a plan there which consists of FS contract tests in
the shim test source tree to verify the 3.2.0 functionality and an adjacent
module which will run those same tests against more recent versions. I need
test will have to beat targetable against objects doors as well as local
and mini HGFS for systems

This is all in github; however it is very much a hadoop extension library.
Is there a way we could release it as an ASF Library but on a different
timetable from normal Hadoop releases? There is always incubator, but this
is such a minor project it is closer to the org.apache.hadoop.thirdparty
library in that it is something all current committers okay should be able
to commit to and release, while releasing on a schedule independent of
hadoop releases themselves. Having it come from this project should give it
more legitimacy.

Steve

Hadoop 3.3.4 release process underway

[Steve Loughran]

I'm setting things up for a new release

https://issues.apache.org/jira/browse/HADOOP-18305

absolute minimum of fixes. as well as some related to ZK lockdown, i would
like to include

https://issues.apache.org/jira/browse/HADOOP-18303
remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime

and
https://issues.apache.org/jira/browse/HADOOP-18307
remove hadoop-cos as a dependency of hadoop-cloud-storage

the last one is a lstminute workaround for, a classpath ordering issue due
to too many libraries having unshaded references to mozilla/prefix-list.txt
https://issues.apache.org/jira/browse/HADOOP-18159

the proper fixes would be getting that updated library tested (who is set
up to test tencent cloud?) and ideally (aws, cos, gcs shaded libraries to
shade their references)

for 3.3.4, i am just going to cut the declaration of the module as a
dependency of hadoop-cloud-storage so out of downstream apps unless they
explicitly ask forit.

now, numbering


   1. I am calling this 3.3.4
   2. I am going to increase the version of branch 3.3. to 3.3.9. that
   leaves space for some more but doesn't confuse jira dropdown dialogs.


i do believe branch-3.3. should be renamed branch-3.4 and the release i
plan to do with mukund called 3.4.0, but that is another bit of project
organisation.

expect the first RC up soon, I am going to be away on vacation from june 28
to july 23 though, which complicates things

Re: Hadoop 3.3.4 release process underway

[Steve Loughran]

update


   1. branch 3.3 has a version of 3.3.9-SNAPSHOT
   2. and 3.3.9 should be the version for fixes against this, with 3.3.4
   for the new point release
   3. please don't use 3.3.4 for branch-3.3 changes from now on

I've already got a PR up with the changes; going to create an asf
branch-3.3.4 branch mirroring it and kicking off with the pom update.

I'm going to do a dry run of a release this week to build the binaries on
x86 and ARM but not put for a vote as I am away next week. instead it'll be
a validation of my processes, ant-based automation etc. (
https://github.com/steveloughran/validate-hadoop-client-artifacts)

 I will kick off the release the following week, which, being july4 week,
may have more people offline. it does give larry a chance to get
https://issues.apache.org/jira/browse/HADOOP-18074 in, as it may have
security implications.



On Mon, 20 Jun 2022 at 19:02, Steve Loughran  wrote:

> I'm setting things up for a new release
>
> https://issues.apache.org/jira/browse/HADOOP-18305
>
> absolute minimum of fixes. as well as some related to ZK lockdown, i would
> like to include
>
> https://issues.apache.org/jira/browse/HADOOP-18303
> remove shading exclusion of javax.ws.rs-api from hadoop-client-runtime
>
> and
> https://issues.apache.org/jira/browse/HADOOP-18307
> remove hadoop-cos as a dependency of hadoop-cloud-storage
>
> the last one is a lstminute workaround for, a classpath ordering issue due
> to too many libraries having unshaded references to mozilla/prefix-list.txt
> https://issues.apache.org/jira/browse/HADOOP-18159
>
> the proper fixes would be getting that updated library tested (who is set
> up to test tencent cloud?) and ideally (aws, cos, gcs shaded libraries to
> shade their references)
>
> for 3.3.4, i am just going to cut the declaration of the module as a
> dependency of hadoop-cloud-storage so out of downstream apps unless they
> explicitly ask forit.
>
> now, numbering
>
>
>1. I am calling this 3.3.4
>2. I am going to increase the version of branch 3.3. to 3.3.9. that
>leaves space for some more but doesn't confuse jira dropdown dialogs.
>
>
> i do believe branch-3.3. should be renamed branch-3.4 and the release i
> plan to do with mukund called 3.4.0, but that is another bit of project
> organisation.
>
> expect the first RC up soon, I am going to be away on vacation from june
> 28 to july 23 though, which complicates things
>
>
>

[VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

I have put together a release candidate (RC0) for Hadoop 3.3.4

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/

The git tag is release-3.3.4-RC0, commit c679bc76d26

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1356/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.

See the release notes for details.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

cancelling this RC so i can issue a new one with an updated reload4j
https://issues.apache.org/jira/browse/HADOOP-18354

i will also do a pr to update aws sdk, which is less critical (the jackson
databind classes don't seem to get used), -upgrading will stop audit tools
overreacting

On Thu, 21 Jul 2022 at 19:07, Steve Loughran  wrote:

>
> I have put together a release candidate (RC0) for Hadoop 3.3.4
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/
>
> The git tag is release-3.3.4-RC0, commit c679bc76d26
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1356/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC0/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
> See the release notes for details.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>
>
>

any HDFS blockers for a 3.3.x feature release?

[Steve Loughran]

Is there any WiP which people consider a blocker for a  3.3.x feature
release to follow the 3.3.4 security/critical integration issues release?

I want to split off the fork for that next release off branch-3.3 and start
stabilising.

are there things people working on which are nearly ready?

[VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

I have put together a release candidate (RC1) for Hadoop 3.3.4

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/

The git tag is release-3.3.4-RC1, commit a585a73c3e0

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1358/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md

There's a very small number of changes, primarily critical code/packaging
issues and security fixes.

See the release notes for details.

Please try the release and vote. The vote will run for 5 days.

steve

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

my vote for this is +1, binding.

obviously I`m biased, but i do not want to have to issue any more interim
releases before the feature release off branch-3.3, so I am trying to be
ruthless.

my client vaidator ant project has a more targets to help with releasing,
and now builds a lot mor of my local projects
https://github.com/steveloughran/validate-hadoop-client-artifacts
all good as far as my test coverage goes, with these projects validating
the staged dependencies.

now, who else can review

On Fri, 29 Jul 2022 at 19:47, Steve Loughran  wrote:

>
>
> I have put together a release candidate (RC1) for Hadoop 3.3.4
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
>
> The git tag is release-3.3.4-RC1, commit a585a73c3e0
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1358/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md
>
> There's a very small number of changes, primarily critical code/packaging
> issues and security fixes.
>
> See the release notes for details.
>
> Please try the release and vote. The vote will run for 5 days.
>
> steve
>

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

The vote passed with the following result.

Binding PMC members:

+1 Chris Nauroth
+1 Steve Loughran
+1 Masatake Iwasaki

Non binding votes:

+1 Ashutosh Gupta

Cheng Pan was worried about the new transient kotlin dependency. They are
working on a PR there which we can target for the successor to this release
https://issues.apache.org/jira/browse/HDFS-16714

I'm going to publish the artifacts, site docs, maven artifacts, then
announce.

Thank you to all who helped to push this release out.


On Thu, 4 Aug 2022 at 11:48, Ashutosh Gupta 
wrote:

> +1 (non-binding)
>
> * Builds from source looks good.
> * Checksums and signatures are correct.
> * Running basic HDFS commands and running simple MapReduce jobs looks good.
> * Skimmed through the contents of site documentation and it looks good.
>
> Thanks Steve for driving this release.
>
> Ashutosh
>
>
> On Wed, Aug 3, 2022 at 9:39 PM Chris Nauroth  wrote:
>
> > +1 (binding)
> >
> > * Verified all checksums.
> > * Verified all signatures.
> > * Built from source, including native code on Linux.
> > * mvn clean package -Pnative -Psrc -Drequire.openssl -Drequire.snappy
> > -Drequire.zstd -DskipTests
> > * Tests passed.
> > * mvn --fail-never clean test -Pnative -Dparallel-tests
> > -Drequire.snappy -Drequire.zstd -Drequire.openssl
> > -Dsurefire.rerunFailingTestsCount=3 -DtestsThreadCount=8
> > * Checked dependency tree to make sure we have all of the expected
> library
> > updates that are mentioned in the release notes.
> > * mvn -o dependency:tree
> >
> > I saw a LibHDFS test failure, but I know it's something flaky that's
> > already tracked in a JIRA issue. The release looks good. Steve, thank you
> > for driving this.
> >
> > Chris Nauroth
> >
> >
> > On Wed, Aug 3, 2022 at 11:27 AM Steve Loughran
>  > >
> > wrote:
> >
> > > my vote for this is +1, binding.
> > >
> > > obviously I`m biased, but i do not want to have to issue any more
> interim
> > > releases before the feature release off branch-3.3, so I am trying to
> be
> > > ruthless.
> > >
> > > my client vaidator ant project has a more targets to help with
> releasing,
> > > and now builds a lot mor of my local projects
> > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > > all good as far as my test coverage goes, with these projects
> validating
> > > the staged dependencies.
> > >
> > > now, who else can review
> > >
> > > On Fri, 29 Jul 2022 at 19:47, Steve Loughran 
> > wrote:
> > >
> > > >
> > > >
> > > > I have put together a release candidate (RC1) for Hadoop 3.3.4
> > > >
> > > > The RC is available at:
> > > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
> > > >
> > > > The git tag is release-3.3.4-RC1, commit a585a73c3e0
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1358/
> > > >
> > > > You can find my public key at:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Change log
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md
> > > >
> > > > Release notes
> > > >
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md
> > > >
> > > > There's a very small number of changes, primarily critical
> > code/packaging
> > > > issues and security fixes.
> > > >
> > > > See the release notes for details.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > steve
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.4

[Steve Loughran]

thanks.

the release is up for download as is the site; i will do the announcement
now.

also automated a lot more of the work of doing the release, inc testing
across projects
https://github.com/steveloughran/validate-hadoop-client-artifacts

On Thu, 4 Aug 2022 at 19:08, Stack  wrote:

> +1 (Sorry, took me a while)
>
> Ran: ./dev-support/hadoop-vote.sh --source
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
>
> * Signature: ok
>
> * Checksum : failed
>
> * Rat check (17.0.1): ok
>
>  - mvn clean apache-rat:check
>
> * Built from source (17.0.1): ok
>
>  - mvn clean install  -DskipTests
>
> * Built tar from source (17.0.1): ok
>
>  - mvn clean package  -Pdist -DskipTests -Dtar
> -Dmaven.javadoc.skip=true
>
> Took a look at website. Home page says stuff like, “ARM Support: This is
> the first release to support ARM architectures.“, which I don’t think is
> true of 3.3.4 but otherwise, looks fine.
>
> Only played with HDFS. UIs looked right.
>
> Deployed to ten node arm64 cluster. Ran the hbase verification job on top
> of it and all passed. Did some kills, stuff came back.
>
> I didn't spend time on unit tests but one set passed on a local rig here:
>
> [image: image.png]
> Stack
>
> On Fri, Jul 29, 2022 at 11:48 AM Steve Loughran
>  wrote:
>
>> I have put together a release candidate (RC1) for Hadoop 3.3.4
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/
>>
>> The git tag is release-3.3.4-RC1, commit a585a73c3e0
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1358/
>>
>> You can find my public key at:
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Change log
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/CHANGELOG.md
>>
>> Release notes
>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.4-RC1/RELEASENOTES.md
>>
>> There's a very small number of changes, primarily critical code/packaging
>> issues and security fixes.
>>
>> See the release notes for details.
>>
>> Please try the release and vote. The vote will run for 5 days.
>>
>> steve
>>
>

[ANNOUNCE] Apache Hadoop 3.3.4 release

[Steve Loughran]

On behalf of the Apache Hadoop Project Management Committee, I am
pleased to announce the release of Apache Hadoop 3.3.4.

---
This is a release of Apache Hadoop 3.3 line.

It contains a small number of security and critical integration fixes since
3.3.3.

Users of Apache Hadoop 3.3.3 should upgrade to this release.

Users of hadoop 2.x and hadoop 3.2 should also upgrade to the 3.3.x line.
As well as feature enhancements, this is the sole branch currently
receiving fixes for anything other than critical security/data integrity
issues.

Users are encouraged to read the [overview of major changes][1] since
release 3.3.3.
For details of bug fixes, improvements, and other enhancements since the
previous 3.3.3 release,
please check [release notes][2] and [changelog][3].

[1]: http://hadoop.apache.org/docs/r3.3.4/index.html
[2]:
http://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-common/release/3.3.4/RELEASENOTES.3.3.4.html
[3]:
http://hadoop.apache.org/docs/r3.3.4/hadoop-project-dist/hadoop-common/release/3.3.4/CHANGELOG.3.3.4.html


Many thanks to everyone who helped in this release by supplying patches,
reviewing them, helping get this release building and testing and
reviewing the final artifacts.

-Steve

Re: Protobuf-java compatibilty question

[Steve Loughran]

I'm afraid not. protobuf maybe compatible at the wire level -the marshalled
data is good-, but the generated classes in jars Will not link to any
version other than that they were explicitly compiled with.
this is why hadoop 3.3 has a private shaded copy in the hadoop-thirdparty
JAR -how to stop the Hadoop/hdfs code restricting what others can use.

On Tue, 9 Aug 2022 at 16:51, Jason Wen 
wrote:

> Hi team,
>
> Hadooop 3.2.x release is using protobuf-java 2.5.0 version lib. It looks
> like protobuf-java 3.x is binary backward compatible with 2.5.0 version.
> Does that mean Hadoop 3.2.x can work correctly if we replace protobuf-java
> 2.5.0 lib with 3.x version at runtime?
>
> Thanks,
> Jason
>

the next hadoop 3.3.x release

[Steve Loughran]

Hi

Mukund Thakur plans to fork off the next branch-3.3. release in 10 days
time. last chance to get changes in.

I'm away next week...try not to break the branch. Any testing you can do
would be appreciated

Steve

HADOOP-18470 Release hadoop 3.3.5

[Steve Loughran]

Mukund has just created the new Hadoop release JIRA,
https://issues.apache.org/jira/browse/HADOOP-18470, and is doing the first
build/test before pushing it up. This is off branch-3.3, so it will have
more significant changes than the 3.3.3 and 3.3.4 releases, which were just
CVE/integration fixes.

The new branch, branch-3.3.5 has its maven/hadoop.version set to
3.3.5-SNAPSHOT.

All JIRA issues fixed/blocked for 3.3.9 now reference 3.3.5. The next step
of the release is to actually move those wontfix issues back to being
against 3.3.9

There is still a 3.3.9 version; branch-3.3's maven build still refers to
it. Issues found/fixed in branch-3.3 *but not the new branch-3.3.5 branch-
should still refer to this version. Anything targeting the 3.3.5 release
must be committed to the new branch, and have its JIRA version tagged
appropriately.

All changes to be cherrypicked into 3.3.5, except for those ones related to
the release process itself, MUST be in branch-3.3 first, and SHOULD be in
trunk unless there is some fundamental reason they can't apply there
(reload4j etc).

Let's try and stabilise this releases, especially bringing up to date all
the JAR dependencies which we can safely update.

Anyone planning to give talks at ApacheCon about forthcoming features
already in 3.3 SHOULD

   1. reference Hadoop 3.3.5 as the version
   2. make sure their stuff works.

Mukund will be at the conf; find him and offer any help you can in getting
this release out.

I'd like to get that Arm64 build workingdoes anyone else want to get
involved?

-steve

Re: [DISCUSS] Hadoop 3.3.5 release planning

[Steve Loughran]

On Fri, 7 Oct 2022 at 22:36, Wei-Chiu Chuang  wrote:

> Bumping this up. Adding the [DISCUSS] text to make this message stand out
> of your inbox.
>
> I certainly missed this message and only realized 3.3.5 has more than just
> security updates.
>
> What was the issue with the ARM64 build? I was able to publish ARM64 build
> for 3.3.1 release without problems.
>

changes in the underlying dependencies, python libraries in particular.
things wouldn't build on branch-3.3.

Also, building on a macbook m1 wasn't a problem then -it is now and some of
the bash tests needed tuning

fixed now HADOOP-18401



>
>
> On Tue, Sep 27, 2022 at 9:35 AM Steve Loughran  >
> wrote:
>
> > Mukund has just created the new Hadoop release JIRA,
> > https://issues.apache.org/jira/browse/HADOOP-18470, and is doing the
> first
> > build/test before pushing it up. This is off branch-3.3, so it will have
> > more significant changes than the 3.3.3 and 3.3.4 releases, which were
> just
> > CVE/integration fixes.
> >
> > The new branch, branch-3.3.5 has its maven/hadoop.version set to
> > 3.3.5-SNAPSHOT.
> >
> > All JIRA issues fixed/blocked for 3.3.9 now reference 3.3.5. The next
> step
> > of the release is to actually move those wontfix issues back to being
> > against 3.3.9
> >
> > There is still a 3.3.9 version; branch-3.3's maven build still refers to
> > it. Issues found/fixed in branch-3.3 *but not the new branch-3.3.5
> branch-
> > should still refer to this version. Anything targeting the 3.3.5 release
> > must be committed to the new branch, and have its JIRA version tagged
> > appropriately.
> >
> > All changes to be cherrypicked into 3.3.5, except for those ones related
> to
> > the release process itself, MUST be in branch-3.3 first, and SHOULD be in
> > trunk unless there is some fundamental reason they can't apply there
> > (reload4j etc).
> >
> > Let's try and stabilise this releases, especially bringing up to date all
> > the JAR dependencies which we can safely update.
> >
> > Anyone planning to give talks at ApacheCon about forthcoming features
> > already in 3.3 SHOULD
> >
> >1. reference Hadoop 3.3.5 as the version
> >2. make sure their stuff works.
> >
> > Mukund will be at the conf; find him and offer any help you can in
> getting
> > this release out.
> >
> > I'd like to get that Arm64 build workingdoes anyone else want to get
> > involved?
> >
> > -steve
> >
>

exciting new content needed for the 3.3.5 index.md file

[Steve Loughran]

The first "smoke test" RC is about to be up for people to play with, we are
just testing things here and getting that arm build done.

Can I have some content for the index.html page describing what has changed?
hadoop-project/src/site/markdown/index.md.vm

I can (and will) speak highly of stuff I've been involved in, but need
contributions from others for what is new in this release in HDFS, YARN,
and MR (other than the manifest committer).

It'd be good to have a list of CVEs fixed by upgrading jars. Maybe we
should have a transitive-CVE tag for all JIRAs which update a dependency
for this, so that then we could have the release notes explicitly list
these in their own section.

Please submit changes to branch-3.3.5; use HADOOP-18470. as the jira for
all the release notes.

 thanks.

[VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

Mukund and I have put together a release candidate (RC0) for Hadoop 3.3.5.

Given the time of year it's a bit unrealistic to run a 5 day vote and
expect people to be able to test it thoroughly enough to make this the one
we can ship.

What we would like is for anyone who can to verify the tarballs, and test
the binaries, especially anyone who can try the arm64 binaries. We've got
the building of those done and now the build file will incorporate them
into the release -but neither of us have actually tested it yet. Maybe I
should try it on my pi400 over xmas.

The maven artifacts are up on the apache staging repo -they are the ones
from x86 build. Building and testing downstream apps will be incredibly
helpful.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/

The git tag is release-3.3.5-RC0, commit 3262495904d

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1365/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC0/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positive.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.


Please try the release and vote on it, even though i don't know what is a
good timeline here...i'm actually going on holiday in early jan. Mukund is
around and so can drive the process while I'm offline.

Assuming we do have another iteration, the RC1 will not be before mid jan
for that reason

Steve (and mukund)

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

thanks

pulling in a few of the recent changes which seem needed/important, now
wondering about the javadocs.

i will add a new probe for this in our automated release ant bulld so we
can't cut a release without that
https://github.com/steveloughran/validate-hadoop-client-artifacts

On Mon, 2 Jan 2023 at 15:47, Masatake Iwasaki 
wrote:

> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
>
> For HBase, classes under com/sun/jersey/json/* and com/sun/xml/* are not
> expected in hbase-shaded-with-hadoop-check-invariants.
> Updating hbase-shaded/pom.xml is expected to be the fix as done in
> HBASE-27292.
>
> https://github.com/apache/hbase/commit/00612106b5fa78a0dd198cbcaab610bd8b1be277
>
>
are we adding some new dependencies from somewhere then? i never even knew
there was a com.sun.json module

hey, imagine if there was a single, standard, json library with a minimal
O/J mapping (strings, numbers, arrays and maps) -we'd be able to cut out
all of jackson, gson, jettison and maybe even avoid the eternal
jackson-databind CVE homework


>[INFO] --- exec-maven-plugin:1.6.0:exec
> (check-jar-contents-for-stuff-with-hadoop) @
> hbase-shaded-with-hadoop-check-invariants ---
>[ERROR] Found artifact with unexpected contents:
> '/home/rocky/srcs/bigtop/build/hbase/rpm/BUILD/hbase-2.4.13/hbase-shaded/hbase-shaded-client/target/hbase-shaded-client-2.4.13.jar'
>Please check the following and either correct the build or update
>the allowed list with reasoning.
>
>com/
>com/sun/
>com/sun/jersey/
>com/sun/jersey/json/
>...
>
>
> For Hive, classes belonging to org.bouncycastle:bcprov-jdk15on:1.68 seem
> to be problematic.
> Excluding them on hive-jdbc  might be the fix.
>
>[ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.2.1:shade (default) on
> project hive-jdbc: Error creating shaded jar: Problem shading JAR
> /home/rocky/.m2/repository/org/bouncycastle/bcprov-jdk15on/1.68/bcprov-jdk15on-1.68.jar
> entry
> META-INF/versions/15/org/bouncycastle/jcajce/provider/asymmetric/edec/SignatureSpi$EdDSA.class:
> java.lang.IllegalArgumentException: Unsupported class file major version 59
> -> [Help 1]
>...
>
>
ahh, covered in https://issues.apache.org/jira/browse/HADOOP-17563  ... the
maven shade plugin needs to be updated to handle the multi-JVM JAR

>
> On 2023/01/02 22:02, Masatake Iwasaki wrote:
> > Thanks for your great effort for the new release, Steve and Mukund.
> >
> > +1 while it would be nice if we can address missed Javadocs.
> >
> > + verified the signature and checksum.
> > + built from source tarball on Rocky Linux 8 and OpenJDK 8 with native
> profile enabled.
> >+ launched pseudo distributed cluster including kms and httpfs with
> Kerberos and SSL enabled.
> >+ created encryption zone, put and read files via httpfs.
> >+ ran example MR wordcount over encryption zone.
> > + built rpm packages by Bigtop and ran smoke-tests on Rocky Linux 8
> (both x86_64 and aarch64).
> >- building HBase 2.4.13 and Hive 3.1.3 against 3.3.5 failed due to
> dependency change.
> >  # while building HBase 2.4.13 and Hive 3.1.3 against Hadoop 3.3.4
> worked.
> > + skimmed the site contents.
> >- Javadocs are not contained (under r3.3.5/api).
> >  # The issue can be reproduced even if I built site docs from the
> source.
> >
> > Masatake Iwasaki
> >
>

[VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC1) for Hadoop 3.3.5.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/

The git tag is release-3.3.5-RC1, commit 274f91a3259

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1368/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positives.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

Steve and Mukund

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

ok, let me cancel, update those jiras and kick off again. that will save
anyone else having to do their homework

On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma  wrote:

> I'm now -1 as I found the wrong information on the top page (index.md).
>
> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
> Router Based Federation.
>
> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
> though it is actually not in branch-3.3. I corrected the fix version and
> created HDFS-16889 to backport them to branch-3.3 about a month ago.
> Unfortunately, it won't be fixed soon. I should have let you know at that
> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
> So I think we have to delete the description from the top page not to
> confuse Hadoop users.
>
> - Takanobu
>
> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>
> > Thanks for driving the release, Steve and Mukund.
> >
> > I found that there were some jiras with wrong fix versions.
> >
> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
> > - HDFS-16845
> > - HADOOP-18345
> >
> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
> > (and it is not in release-3.3.4) :
> > - HADOOP-17276
> > - HDFS-13293
> > - HDFS-15630
> > - HDFS-16266
> > - HADOOP-18003
> > - HDFS-16310
> > - HADOOP-18014
> >
> > I corrected all the wrong fix versions just now. I'm not sure we should
> > revote it since it only affects the changelog.
> >
> > - Takanobu
> >
> > 2023年2月21日(火) 22:43 Steve Loughran :
> >
> >> Apache Hadoop 3.3.5
> >>
> >> Mukund and I have put together a release candidate (RC1) for Hadoop
> 3.3.5.
> >>
> >> What we would like is for anyone who can to verify the tarballs,
> >> especially
> >> anyone who can try the arm64 binaries as we want to include them too.
> >>
> >> The RC is available at:
> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
> >>
> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
> >>
> >> The maven artifacts are staged at
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
> >>
> >> You can find my public key at:
> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>
> >> Change log
> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
> >>
> >> Release notes
> >>
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
> >>
> >> This is off branch-3.3 and is the first big release since 3.3.2.
> >>
> >> Key changes include
> >>
> >> * Big update of dependencies to try and keep those reports of
> >>   transitive CVEs under control -both genuine and false positives.
> >> * HDFS RBF enhancements
> >> * Critical fix to ABFS input stream prefetching for correct reading.
> >> * Vectored IO API for all FSDataInputStream implementations, with
> >>   high-performance versions for file:// and s3a:// filesystems.
> >>   file:// through java native io
> >>   s3a:// parallel GET requests.
> >> * This release includes Arm64 binaries. Please can anyone with
> >>   compatible systems validate these.
> >>
> >> Note, because the arm64 binaries are built separately on a different
> >> platform and JVM, their jar files may not match those of the x86
> >> release -and therefore the maven artifacts. I don't think this is
> >> an issue (the ASF actually releases source tarballs, the binaries are
> >> there for help only, though with the maven repo that's a bit blurred).
> >>
> >> The only way to be consistent would actually untar the x86.tar.gz,
> >> overwrite its binaries with the arm stuff, retar, sign and push out
> >> for the vote. Even automating that would be risky.
> >>
> >> Please try the release and vote. The vote will run for 5 days.
> >>
> >> Steve and Mukund
> >>
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
maven releases (3.9.0)

on a new local build with maven updated on homebrew (which i needed for
spark). so a code change too. That issue doesn't surface on our
release dockers, but will hit other people. especially over time. Going to
revert HADOOP-18590. Publish SBOM artifacts (#5281)



On Thu, 23 Feb 2023 at 10:29, Steve Loughran  wrote:

> ok, let me cancel, update those jiras and kick off again. that will save
> anyone else having to do their homework
>
> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
> wrote:
>
>> I'm now -1 as I found the wrong information on the top page (index.md).
>>
>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
>> Router Based Federation.
>>
>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
>> though it is actually not in branch-3.3. I corrected the fix version and
>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
>> Unfortunately, it won't be fixed soon. I should have let you know at that
>> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
>> So I think we have to delete the description from the top page not to
>> confuse Hadoop users.
>>
>> - Takanobu
>>
>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>>
>> > Thanks for driving the release, Steve and Mukund.
>> >
>> > I found that there were some jiras with wrong fix versions.
>> >
>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
>> > - HDFS-16845
>> > - HADOOP-18345
>> >
>> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
>> > (and it is not in release-3.3.4) :
>> > - HADOOP-17276
>> > - HDFS-13293
>> > - HDFS-15630
>> > - HDFS-16266
>> > - HADOOP-18003
>> > - HDFS-16310
>> > - HADOOP-18014
>> >
>> > I corrected all the wrong fix versions just now. I'm not sure we should
>> > revote it since it only affects the changelog.
>> >
>> > - Takanobu
>> >
>> > 2023年2月21日(火) 22:43 Steve Loughran :
>> >
>> >> Apache Hadoop 3.3.5
>> >>
>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
>> 3.3.5.
>> >>
>> >> What we would like is for anyone who can to verify the tarballs,
>> >> especially
>> >> anyone who can try the arm64 binaries as we want to include them too.
>> >>
>> >> The RC is available at:
>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>> >>
>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
>> >>
>> >> The maven artifacts are staged at
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>> >>
>> >> You can find my public key at:
>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >>
>> >> Change log
>> >>
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>> >>
>> >> Release notes
>> >>
>> >>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>> >>
>> >> This is off branch-3.3 and is the first big release since 3.3.2.
>> >>
>> >> Key changes include
>> >>
>> >> * Big update of dependencies to try and keep those reports of
>> >>   transitive CVEs under control -both genuine and false positives.
>> >> * HDFS RBF enhancements
>> >> * Critical fix to ABFS input stream prefetching for correct reading.
>> >> * Vectored IO API for all FSDataInputStream implementations, with
>> >>   high-performance versions for file:// and s3a:// filesystems.
>> >>   file:// through java native io
>> >>   s3a:// parallel GET requests.
>> >> * This release includes Arm64 binaries. Please can anyone with
>> >>   compatible systems validate these.
>> >>
>> >> Note, because the arm64 binaries are built separately on a different
>> >> platform and JVM, their jar files may not match those of the x86
>> >> release -and therefore the maven artifacts. I don't think this is
>> >> an issue (the ASF actually releases source tarballs, the binaries are
>> >> there for help only, though with the maven repo that's a bit blurred).
>> >>
>> >> The only way to be consistent would actually untar the x86.tar.gz,
>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>> >> for the vote. Even automating that would be risky.
>> >>
>> >> Please try the release and vote. The vote will run for 5 days.
>> >>
>> >> Steve and Mukund
>> >>
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.5

[Steve Loughran]

 need this pr in too, https://github.com/apache/hadoop/pull/5429

   1. cuts back on some transitive dependencies from hadoop-aliyun
   2. fixes LICENSE-bin to be correct

#2 is the blocker...and it looks like 3.2.x will also need fixup as well as
the later ones -hadoop binaries have shipped without that file being up to
date, but at least all the transitive stuff is correctly licensed. And i
think we need to change the PR template to mention transitive updates in
the license bit too

if this goes in, I will do the rebuild on monday UK time

On Thu, 23 Feb 2023 at 11:18, Steve Loughran  wrote:

>
> And I've just hit HADOOP-18641. cyclonedx maven plugin breaks on recent
> maven releases (3.9.0)
>
> on a new local build with maven updated on homebrew (which i needed for
> spark). so a code change too. That issue doesn't surface on our
> release dockers, but will hit other people. especially over time. Going to
> revert HADOOP-18590. Publish SBOM artifacts (#5281)
>
>
>
> On Thu, 23 Feb 2023 at 10:29, Steve Loughran  wrote:
>
>> ok, let me cancel, update those jiras and kick off again. that will save
>> anyone else having to do their homework
>>
>> On Thu, 23 Feb 2023 at 08:56, Takanobu Asanuma 
>> wrote:
>>
>>> I'm now -1 as I found the wrong information on the top page (index.md).
>>>
>>> > 1. HDFS-13522, HDFS-16767 & Related Jiras: Allow Observer Reads in HDFS
>>> Router Based Federation.
>>>
>>> The fix version of HDFS-13522 and HDFS-16767 also included 3.3.5 before,
>>> though it is actually not in branch-3.3. I corrected the fix version and
>>> created HDFS-16889 to backport them to branch-3.3 about a month ago.
>>> Unfortunately, it won't be fixed soon. I should have let you know at that
>>> time, sorry.  Supporting Observer NameNode in RBF is a prominent feature.
>>> So I think we have to delete the description from the top page not to
>>> confuse Hadoop users.
>>>
>>> - Takanobu
>>>
>>> 2023年2月23日(木) 17:17 Takanobu Asanuma :
>>>
>>> > Thanks for driving the release, Steve and Mukund.
>>> >
>>> > I found that there were some jiras with wrong fix versions.
>>> >
>>> > The fix versions included 3.3.5, but actually, it isn't in 3.3.5-RC1:
>>> > - HDFS-16845
>>> > - HADOOP-18345
>>> >
>>> > The fix versions didn't include 3.3.5, but actually, it is in 3.3.5-RC1
>>> > (and it is not in release-3.3.4) :
>>> > - HADOOP-17276
>>> > - HDFS-13293
>>> > - HDFS-15630
>>> > - HDFS-16266
>>> > - HADOOP-18003
>>> > - HDFS-16310
>>> > - HADOOP-18014
>>> >
>>> > I corrected all the wrong fix versions just now. I'm not sure we should
>>> > revote it since it only affects the changelog.
>>> >
>>> > - Takanobu
>>> >
>>> > 2023年2月21日(火) 22:43 Steve Loughran :
>>> >
>>> >> Apache Hadoop 3.3.5
>>> >>
>>> >> Mukund and I have put together a release candidate (RC1) for Hadoop
>>> 3.3.5.
>>> >>
>>> >> What we would like is for anyone who can to verify the tarballs,
>>> >> especially
>>> >> anyone who can try the arm64 binaries as we want to include them too.
>>> >>
>>> >> The RC is available at:
>>> >> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/
>>> >>
>>> >> The git tag is release-3.3.5-RC1, commit 274f91a3259
>>> >>
>>> >> The maven artifacts are staged at
>>> >>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1368/
>>> >>
>>> >> You can find my public key at:
>>> >> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >>
>>> >> Change log
>>> >>
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/CHANGELOG.md
>>> >>
>>> >> Release notes
>>> >>
>>> >>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC1/RELEASENOTES.md
>>> >>
>>> >> This is off branch-3.3 and is the first big release since 3.3.2.
>>> >>
>>> >> Key changes include
>>> >>
>>> >> * Big update of dependencies to try and keep those reports of
>>> >>   transitive CVEs under control -both genui

[VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

Mukund and I have put together a release candidate (RC2) for Hadoop 3.3.5.

We need anyone who can to verify the source and binary artifacts,
including those JARs staged on maven, the site documentation and the arm64
tar file.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/

The git tag is release-3.3.5-RC2, commit 72f8c2a4888

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

As to what changed since the RC1 attempt last week


   1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
   2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
   3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating build
   issues in maven 3.9.0)
   4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)


Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

Steve and Mukund

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

well, lets see what others say.

we don't want to ship stuff with serious regression to hdfs.

I will highlight that I am completely fed up with doing this  release and
really want to get it out the way -for which I depend on support from as
many other developers as possible.

Erik, right now what you can help by doing is test all the rest of the
release knowing that this issue exists and seeing if you can identify
anything else. That way this update will be the sole blocker and we can get
through that next RC with nothing else surfacing.

I had noticed that the arm64 release somehow missed out the native binaries
and was going to investigate that but didn't consider that a blocker… I was
just going to cut that artefact and, post Darcy, create a new arm64 release
using all the jars of the x86 build but replacing the x86 native libs with
the arm versions. This helps ensure that the JAR files in the wild all
match, which strikes me as a good thing.

Can I also encourage people in the HFS team to put their hand up and
volunteer for leading the next release, with a goal of getting something
out later this year.



On Thu, 2 Mar 2023 at 00:27, Erik Krogen  wrote:

> Hi folks, apologies for being late to the release conversation, but I think
> we need to get HDFS-16923
> <https://issues.apache.org/jira/browse/HDFS-16923> into
> 3.3.5. HDFS-16732 <https://issues.apache.org/jira/browse/HDFS-16732>,
> which
> also went into 3.3.5, introduced an issue whereby Observer NameNodes will
> throw NPE upon any getListing call on a directory that doesn't exist. It
> will make Observer NN pretty much unusable in 3.3.5. Zander put up a patch
> for this and it has been merged to trunk/branch-3.3 as of a few minutes
> ago. I'd like to see about merging to branch-3.3.5 as well.
>
> Thanks for the consideration and sorry for not bringing this up in RC1 or
> earlier.
>
> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran  >
> wrote:
>
> > Mukund and I have put together a release candidate (RC2) for Hadoop
> 3.3.5.
> >
> > We need anyone who can to verify the source and binary artifacts,
> > including those JARs staged on maven, the site documentation and the
> arm64
> > tar file.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
> >
> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > As to what changed since the RC1 attempt last week
> >
> >
> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
> build
> >issues in maven 3.9.0)
> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup. (#5429)
> >
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Steve and Mukund
> >
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

shipping broken hdfs isn't something we'd want to do, but if we can be
confident that all other issues can be addressed in RC3 then I'd be happy.

On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:

> I will highlight that I am completely fed up with doing this  release and
>> really want to get it out the way -for which I depend on support from as
>> many other developers as possible.
>
>
> hmm, I can feel the pain. I tried to find if there is any config or any
> workaround which can dodge this HDFS issue, but unfortunately couldn't find
> any. If someone does a getListing with needLocation and the file doesn't
> exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
> the exception, AFAIK Observer reads have some logic around handling FNF
> specifically, that it attempts Active NN or something like that in such
> cases, So, that will be broken as well for this use case.
>
> Now, there is no denying the fact there is an issue on the HDFS side, and
> it has already been too much work on your side, so you can argue that it
> might not be a very frequent use case or so. It's your call.
>
> Just sharing, no intentions of saying you should do that, But as an RM
> "nobody" can force you for a new iteration of a RC, it is gonna be your
> call and discretion. As far as I know a release can not be vetoed by
> "anybody" as per the apache by laws.(
> https://www.apache.org/legal/release-policy.html#release-approval). Even
> our bylaws say that product release requires a Lazy Majority not a
> Consensus Approval.
>
> So, you have a way out. You guys are 2 already and 1 I will give you a
> pass, in case you are really in a state: ''Get me out of this mess" state,
> my basic validations on x86 & Aarch64 both are passing as of now, couldn't
> reach the end for any of the RC's
>
> -Ayush
>
> On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
>
>> While this RC is not going to be final, I just wanted to share the results
>> of the testing I have done so far with RC1 and RC2.
>>
>> * Signature: ok
>> * Checksum : ok
>> * Rat check (1.8.0_341): ok
>>  - mvn clean apache-rat:check
>> * Built from source (1.8.0_341): ok
>>  - mvn clean install  -DskipTests
>> * Built tar from source (1.8.0_341): ok
>>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>>
>> * Built images using the tarball, installed and started all of Hdfs, JHS
>> and Yarn components
>> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce job
>> * Hdfs CRUD tests
>> * MapReduce wordcount job
>>
>> * Ran S3A tests with scale profile against us-west-2:
>> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>>
>> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s. This
>> is
>> consistently failing, looks like a recent regression.
>> I was also able to repro on trunk, will create Jira.
>>
>>
>> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> 
>> wrote:
>>
>> > Mukund and I have put together a release candidate (RC2) for Hadoop
>> 3.3.5.
>> >
>> > We need anyone who can to verify the source and binary artifacts,
>> > including those JARs staged on maven, the site documentation and the
>> arm64
>> > tar file.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/
>> >
>> > The git tag is release-3.3.5-RC2, commit 72f8c2a4888
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC2/RELEASENOTES.md
>> >
>> > This is off branch-3.3 and is the first big release since 3.3.2.
>> >
>> > As to what changed since the RC1 attempt last week
>> >
>> >
>> >1. Version fixup in JIRA (credit due to Takanobu Asanuma there)
>> >2. HADOOP-18470. Remove HDFS RBF text in the 3.3.5 index.md file
>> >3. Revert "HADOOP-18590. Publish SBOM artifacts (#5281)" (creating
>> build
>> >issues in maven 3.9.0)
>> >4. HADOOP-18641. Cloud connector dependency and LICENSE fixup.
>> (#5429)
>> >
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Steve and Mukund
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:

> Thanks Steve. I see now that the branch cut was way back in October so I
> definitely understand your frustration here!
>
> This made me realize that HDFS-16832
> <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a very
> similar issue as the aforementioned HDFS-16923, is also missing from the
> RC. I erroneously marked it with a fix version of 3.3.5 -- it was before
> the initial 3.3.5 RC was made and I didn't notice the branch was cut. My
> apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> branch-3.3.5, so they are ready if/when an RC3 is cut.
>

thanks.

>
> In the meantime, I tested for RC2 that a local cluster of NN + standby +
> observer + QJM works as expected for some basic HDFS commands.
>

OK. Could you have a go with a (locally built) patch release

>
> On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran 
> wrote:
>
>> shipping broken hdfs isn't something we'd want to do, but if we can be
>> confident that all other issues can be addressed in RC3 then I'd be happy.
>>
>> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
>>
>> > I will highlight that I am completely fed up with doing this  release
>> and
>> >> really want to get it out the way -for which I depend on support from
>> as
>> >> many other developers as possible.
>> >
>> >
>> > hmm, I can feel the pain. I tried to find if there is any config or any
>> > workaround which can dodge this HDFS issue, but unfortunately couldn't
>> find
>> > any. If someone does a getListing with needLocation and the file doesn't
>> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't just
>> > the exception, AFAIK Observer reads have some logic around handling FNF
>> > specifically, that it attempts Active NN or something like that in such
>> > cases, So, that will be broken as well for this use case.
>> >
>> > Now, there is no denying the fact there is an issue on the HDFS side,
>> and
>> > it has already been too much work on your side, so you can argue that it
>> > might not be a very frequent use case or so. It's your call.
>> >
>> > Just sharing, no intentions of saying you should do that, But as an RM
>> > "nobody" can force you for a new iteration of a RC, it is gonna be your
>> > call and discretion. As far as I know a release can not be vetoed by
>> > "anybody" as per the apache by laws.(
>> > https://www.apache.org/legal/release-policy.html#release-approval).
>> Even
>> > our bylaws say that product release requires a Lazy Majority not a
>> > Consensus Approval.
>> >
>> > So, you have a way out. You guys are 2 already and 1 I will give you a
>> > pass, in case you are really in a state: ''Get me out of this mess"
>> state,
>> > my basic validations on x86 & Aarch64 both are passing as of now,
>> couldn't
>> > reach the end for any of the RC's
>> >
>> > -Ayush
>> >
>> > On Fri, 3 Mar 2023 at 08:41, Viraj Jasani  wrote:
>> >
>> >> While this RC is not going to be final, I just wanted to share the
>> results
>> >> of the testing I have done so far with RC1 and RC2.
>> >>
>> >> * Signature: ok
>> >> * Checksum : ok
>> >> * Rat check (1.8.0_341): ok
>> >>  - mvn clean apache-rat:check
>> >> * Built from source (1.8.0_341): ok
>> >>  - mvn clean install  -DskipTests
>> >> * Built tar from source (1.8.0_341): ok
>> >>  - mvn clean package  -Pdist -DskipTests -Dtar
>> -Dmaven.javadoc.skip=true
>> >>
>> >> * Built images using the tarball, installed and started all of Hdfs,
>> JHS
>> >> and Yarn components
>> >> * Ran Hbase (latest 2.5) tests against Hdfs, ran RowCounter Mapreduce
>> job
>> >> * Hdfs CRUD tests
>> >> * MapReduce wordcount job
>> >>
>> >> * Ran S3A tests with scale profile against us-west-2:
>> >> mvn clean verify -Dparallel-tests -DtestsThreadCount=8 -Dscale
>> >>
>> >> ITestS3AConcurrentOps#testParallelRename is timing out after ~960s.
>> This
>> >> is
>> >> consistently failing, looks like a recent regression.
>> >> I was also able to repro on trunk, will create Jira.
>> >>
>> >>
>> >> On Mon, Feb 27, 2023 at 9:59 AM Steve Loughran
>> >&g

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

 i looked at that test and wondered if it it was just being brittle to
time. I'm not a fan of those -there's one in abfs which is particularly bad
for me- maybe we could see if the test can be cut as it is quite a slow one

On Sat, 4 Mar 2023 at 18:28, Viraj Jasani  wrote:

> A minor update on ITestS3AConcurrentOps#testParallelRename
>
> I was previously connected to a vpn due to which bandwidth was getting
> throttled earlier. Ran the test again today without vpn and had no issues
> (earlier only 40% of the overall putObject were able to get completed
> within timeout).
>
>
> On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran  >
> wrote:
>
> > On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
> >
> > > Thanks Steve. I see now that the branch cut was way back in October so
> I
> > > definitely understand your frustration here!
> > >
> > > This made me realize that HDFS-16832
> > > <https://issues.apache.org/jira/browse/HDFS-16832>, which resolves a
> > very
> > > similar issue as the aforementioned HDFS-16923, is also missing from
> the
> > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> before
> > > the initial 3.3.5 RC was made and I didn't notice the branch was cut.
> My
> > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > >
> >
> > thanks.
> >
> > >
> > > In the meantime, I tested for RC2 that a local cluster of NN + standby
> +
> > > observer + QJM works as expected for some basic HDFS commands.
> > >
> >
> > OK. Could you have a go with a (locally built) patch release
> >
> > >
> > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > 
> > > wrote:
> > >
> > >> shipping broken hdfs isn't something we'd want to do, but if we can be
> > >> confident that all other issues can be addressed in RC3 then I'd be
> > happy.
> > >>
> > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena  wrote:
> > >>
> > >> > I will highlight that I am completely fed up with doing this
> release
> > >> and
> > >> >> really want to get it out the way -for which I depend on support
> from
> > >> as
> > >> >> many other developers as possible.
> > >> >
> > >> >
> > >> > hmm, I can feel the pain. I tried to find if there is any config or
> > any
> > >> > workaround which can dodge this HDFS issue, but unfortunately
> couldn't
> > >> find
> > >> > any. If someone does a getListing with needLocation and the file
> > doesn't
> > >> > exist at Observer he is gonna get a NPE rather than a FNF, It isn't
> > just
> > >> > the exception, AFAIK Observer reads have some logic around handling
> > FNF
> > >> > specifically, that it attempts Active NN or something like that in
> > such
> > >> > cases, So, that will be broken as well for this use case.
> > >> >
> > >> > Now, there is no denying the fact there is an issue on the HDFS
> side,
> > >> and
> > >> > it has already been too much work on your side, so you can argue
> that
> > it
> > >> > might not be a very frequent use case or so. It's your call.
> > >> >
> > >> > Just sharing, no intentions of saying you should do that, But as an
> RM
> > >> > "nobody" can force you for a new iteration of a RC, it is gonna be
> > your
> > >> > call and discretion. As far as I know a release can not be vetoed by
> > >> > "anybody" as per the apache by laws.(
> > >> > https://www.apache.org/legal/release-policy.html#release-approval).
> > >> Even
> > >> > our bylaws say that product release requires a Lazy Majority not a
> > >> > Consensus Approval.
> > >> >
> > >> > So, you have a way out. You guys are 2 already and 1 I will give
> you a
> > >> > pass, in case you are really in a state: ''Get me out of this mess"
> > >> state,
> > >> > my basic validations on x86 & Aarch64 both are passing as of now,
> > >> couldn't
> > >> > reach the end for any of the RC's
> > >> >
> > >> > -Ayush
> > >> >
> > >> > On Fri, 3 Mar 

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC2)

[Steve Loughran]

thanks.

now looking at a critical kerby CVE (
https://github.com/apache/hadoop/pull/5458) and revisited one for netty
from last week

i am never a fan of last-minute jar updates, but if we don't ship with them
we will be fielding jiras of "update kerby/netty on 3.3.5" for the next 18
months

On Mon, 6 Mar 2023 at 23:29, Erik Krogen  wrote:

> > OK. Could you have a go with a (locally built) patch release
>
> Just validated the same on the latest HEAD of branch-3.3.5, which includes
> the two HDFS Jiras I mentioned plus one additional one:
>
> * 143fe8095d4 (HEAD -> branch-3.3.5) 2023-03-06 HDFS-16934.
> TestDFSAdmin.testAllDatanodesReconfig regression (#5434) [slfan1989 <
> 55643692+slfan1...@users.noreply.github.com>]
> * d4ea9687a8e 2023-03-03 HDFS-16923. [SBN read] getlisting RPC to observer
> will throw NPE if path does not exist (#5400) [ZanderXu <
> zande...@apache.org
> >]
> * 44bf8aadedf 2023-03-03 HDFS-16832. [SBN READ] Follow-on to HDFS-16732.
> Fix NPE when check the block location of empty directory (#5099)
> [zhengchenyu ]
> * 72f8c2a4888 (tag: release-3.3.5-RC2) 2023-02-25 HADOOP-18641. Cloud
> connector dependency and LICENSE fixup. (#5429) [Steve Loughran <
> ste...@cloudera.com>]
>
> On Mon, Mar 6, 2023 at 2:17 AM Steve Loughran  >
> wrote:
>
> >  i looked at that test and wondered if it it was just being brittle to
> > time. I'm not a fan of those -there's one in abfs which is particularly
> bad
> > for me- maybe we could see if the test can be cut as it is quite a slow
> one
> >
> > On Sat, 4 Mar 2023 at 18:28, Viraj Jasani  wrote:
> >
> > > A minor update on ITestS3AConcurrentOps#testParallelRename
> > >
> > > I was previously connected to a vpn due to which bandwidth was getting
> > > throttled earlier. Ran the test again today without vpn and had no
> issues
> > > (earlier only 40% of the overall putObject were able to get completed
> > > within timeout).
> > >
> > >
> > > On Sat, Mar 4, 2023 at 4:29 AM Steve Loughran
> >  > > >
> > > wrote:
> > >
> > > > On Sat, 4 Mar 2023 at 01:47, Erik Krogen  wrote:
> > > >
> > > > > Thanks Steve. I see now that the branch cut was way back in October
> > so
> > > I
> > > > > definitely understand your frustration here!
> > > > >
> > > > > This made me realize that HDFS-16832
> > > > > <https://issues.apache.org/jira/browse/HDFS-16832>, which
> resolves a
> > > > very
> > > > > similar issue as the aforementioned HDFS-16923, is also missing
> from
> > > the
> > > > > RC. I erroneously marked it with a fix version of 3.3.5 -- it was
> > > before
> > > > > the initial 3.3.5 RC was made and I didn't notice the branch was
> cut.
> > > My
> > > > > apologies for that. I've pushed both HDFS-16832 and HDFS-16932 to
> > > > > branch-3.3.5, so they are ready if/when an RC3 is cut.
> > > > >
> > > >
> > > > thanks.
> > > >
> > > > >
> > > > > In the meantime, I tested for RC2 that a local cluster of NN +
> > standby
> > > +
> > > > > observer + QJM works as expected for some basic HDFS commands.
> > > > >
> > > >
> > > > OK. Could you have a go with a (locally built) patch release
> > > >
> > > > >
> > > > > On Fri, Mar 3, 2023 at 2:52 AM Steve Loughran
> > > > 
> > > > > wrote:
> > > > >
> > > > >> shipping broken hdfs isn't something we'd want to do, but if we
> can
> > be
> > > > >> confident that all other issues can be addressed in RC3 then I'd
> be
> > > > happy.
> > > > >>
> > > > >> On Fri, 3 Mar 2023 at 05:09, Ayush Saxena 
> > wrote:
> > > > >>
> > > > >> > I will highlight that I am completely fed up with doing this
> > > release
> > > > >> and
> > > > >> >> really want to get it out the way -for which I depend on
> support
> > > from
> > > > >> as
> > > > >> >> many other developers as possible.
> > > > >> >
> > > > >> >
> > > > >> > hmm, I can feel the pain. I tried to find if there is any config
> > or
> > > > any
> > > > >> > w

[VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

Apache Hadoop 3.3.5

Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/

The git tag is release-3.3.5-RC3, commit 706d88266ab

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1369/

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md

This is off branch-3.3 and is the first big release since 3.3.2.

Key changes include

* Big update of dependencies to try and keep those reports of
  transitive CVEs under control -both genuine and false positives.
* HDFS RBF enhancements
* Critical fix to ABFS input stream prefetching for correct reading.
* Vectored IO API for all FSDataInputStream implementations, with
  high-performance versions for file:// and s3a:// filesystems.
  file:// through java native io
  s3a:// parallel GET requests.
* This release includes Arm64 binaries. Please can anyone with
  compatible systems validate these.
* and compared to the previous RC, all the major changes are
  HDFS issues.

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

Please try the release and vote. The vote will run for 5 days.

-Steve

Re: [DISCUSS] Move HDFS specific APIs to FileSystem abstration

[Steve Loughran]

   1. I think a new interface would be good as FileContext could do the
   same thing
   2. using PathCapabilities probes should still be mandatory as for
   FileContext it would depend on the back end
   3. Whoever does this gets to specify what the API does and write the
   contract tests. Saying "just to do what HDFS does" isn't enough as it's not
   always clear the HDFS team no how much of that behaviour is intentional
   (rename, anyone?).


For any new API (a better rename, a better delete,...) I would normally
insist on making it cloud friendly, with an extensible builder API and an
emphasis on asynchronous IO. However this is existing code and does target
HDFS and Ozone -pulling the existing APIs up into a new interface seems the
right thing to do here.

 I have a WiP project to do a shim library to offer new FS APIs two older
Hadoop releases by way of reflection, so that we can get new APIs taken up
across projects where we cannot choreograph version updates across the
entire stack. (hello parquet, spark,...). My goal is to actually make this
a Hadoop managed project, with its own release schedule. You could add an
equivalent of the new interface in here, which would then use reflection
behind-the-scenes to invoke the underlying HDFS methods when the FS client
has them.

https://github.com/steveloughran/fs-api-shim

I've just added vector IO API there; the next step is to copy over a lot of
the contract tests from hadoop common and apply them through the shim -to
hadoop 3.2, 3.3.0-3.3.5. That testing against many backends is actually as
tricky as the reflection itself. However without this library it is going
to take a long long time for the open source applications to pick up the
higher performance/Cloud ready Apis. Yes, those of us who can build the
entire stack can do it, but that gradually adds more divergence from the
open source libraries, reduces the test coverage overall and only increases
maintenance costs over time.

steve

On Thu, 16 Mar 2023 at 20:56, Wei-Chiu Chuang  wrote:

> Hi,
>
> Stephen and I are working on a project to make HBase to run on Ozone.
>
> HBase, born out of the Hadoop project, depends on a number of HDFS specific
> APIs, including recoverLease() and isInSafeMode(). The HBase community [1]
> strongly voiced that they don't want the project to have direct dependency
> on additional FS implementations due to dependency and vulnerability
> management concerns.
>
> To make this project successful, we're exploring options, to push up these
> APIs to the FileSystem abstraction. Eventually, it would make HBase FS
> implementation agnostic, and perhaps enable HBase to support other storage
> systems in the future.
>
> We'd use the PathCapabilities API to probe if the underlying FS
> implementation supports these APIs, and would then invoke the corresponding
> FileSystem APIs. This is straightforward but the FileSystem would become
> bloated.
>
> Another option is to create a "RecoverableFileSystem" interface, and have
> both DistributedFileSystem (HDFS) and RootedOzoneFileSystem (Ozone). This
> way the impact to the Hadoop project and the FileSystem abstraction is even
> smaller.
>
> Thoughts?
>
> [1] https://lists.apache.org/thread/tcrp8vxxs3z12y36mpzx35txhpp7tvxv
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

and my vote

My vote

+1 binding

I've been using the RCs for a while as my CLI entry point, and testing it
through other builds

for this RC
* Local builds of cloudstore
* fs-api-shim
* spark
* built and ran my cloud integration tests, which now include large CVS
file jobs which should show the Azure prefetch bug if it still existed.

downloaded the tar, expanded it, ran command line code with it, including
cloudstore against the stores. we need to get hadoop-azure and its
dependencies onto the path by default, to make abfs io easier.


I have the arm binaries building, and did a checknative to make sure all
was good

stevel@0da162643f99:~/hadoop/patchprocess/hadoop-3.3.5$ bin/hadoop
checknative
2023-03-17 13:00:27,107 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2023-03-17 13:00:27,112 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2023-03-17 13:00:27,121 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2023-03-17 13:00:27,156 INFO nativeio.NativeIO: The native code was built
without PMDK support.
Native library checking:
hadoop:  true
/home/stevel/hadoop/patchprocess/hadoop-3.3.5/lib/native/libhadoop.so.1.0.0
zlib:true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: true /lib/aarch64-linux-gnu/libcrypto.so
ISA-L:   false libhadoop was built without ISA-L support
PMDK:false The native code was built without PMDK support.

---


On Wed, 15 Mar 2023 at 19:47, Steve Loughran  wrote:

>
> Apache Hadoop 3.3.5
>
> Mukund and I have put together a release candidate (RC3) for Hadoop 3.3.5.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
>
> The git tag is release-3.3.5-RC3, commit 706d88266ab
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1369/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
>
> This is off branch-3.3 and is the first big release since 3.3.2.
>
> Key changes include
>
> * Big update of dependencies to try and keep those reports of
>   transitive CVEs under control -both genuine and false positives.
> * HDFS RBF enhancements
> * Critical fix to ABFS input stream prefetching for correct reading.
> * Vectored IO API for all FSDataInputStream implementations, with
>   high-performance versions for file:// and s3a:// filesystems.
>   file:// through java native io
>   s3a:// parallel GET requests.
> * This release includes Arm64 binaries. Please can anyone with
>   compatible systems validate these.
> * and compared to the previous RC, all the major changes are
>   HDFS issues.
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> Please try the release and vote. The vote will run for 5 days.
>
> -Steve
>

Re: [VOTE] Release Apache Hadoop 3.3.5 (RC3)

[Steve Loughran]

Thank you for this!

Can anyone else with time do a review too? i really want to get this one
done, now the HDFS issues are all resolved.

I do not want this release to fall by the wayside through lack of votes
alone. In fact, I would be very unhappy



On Sat, 18 Mar 2023 at 06:47, Viraj Jasani  wrote:

> +1 (non-binding)
>
> * Signature/Checksum: ok
> * Rat check (1.8.0_341): ok
>  - mvn clean apache-rat:check
> * Built from source (1.8.0_341): ok
>  - mvn clean install  -DskipTests
> * Built tar from source (1.8.0_341): ok
>  - mvn clean package  -Pdist -DskipTests -Dtar -Dmaven.javadoc.skip=true
>
> Containerized deployments:
> * Deployed and started Hdfs - NN, DN, JN with Hbase 2.5 and Zookeeper 3.7
> * Deployed and started JHS, RM, NM
> * Hbase, hdfs CRUD looks good
> * Sample RowCount MapReduce job looks good
>
> * S3A tests with scale profile looks good
>
>
> On Wed, Mar 15, 2023 at 12:48 PM Steve Loughran
> 
> wrote:
>
> > Apache Hadoop 3.3.5
> >
> > Mukund and I have put together a release candidate (RC3) for Hadoop
> 3.3.5.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/
> >
> > The git tag is release-3.3.5-RC3, commit 706d88266ab
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1369/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.3.5-RC3/RELEASENOTES.md
> >
> > This is off branch-3.3 and is the first big release since 3.3.2.
> >
> > Key changes include
> >
> > * Big update of dependencies to try and keep those reports of
> >   transitive CVEs under control -both genuine and false positives.
> > * HDFS RBF enhancements
> > * Critical fix to ABFS input stream prefetching for correct reading.
> > * Vectored IO API for all FSDataInputStream implementations, with
> >   high-performance versions for file:// and s3a:// filesystems.
> >   file:// through java native io
> >   s3a:// parallel GET requests.
> > * This release includes Arm64 binaries. Please can anyone with
> >   compatible systems validate these.
> > * and compared to the previous RC, all the major changes are
> >   HDFS issues.
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > -Steve
> >
>