Re: [External Sender] Re: JDK17 support for Hadoop

[slfan1989]

Hi, Jason

Thank you for your question! I am currently working on enabling Hadoop
support for JDK 17, and several community members have contributed PRs for
it. I am addressing the issue of HADOOP-15984, and once that is resolved,
progress on other JDK 17-related tasks will be quicker. I expect to have
basic support for compiling under JDK 17 by Q4 2024. As far as I know,
there is no more detailed timeline available at this moment.

Shilun Fan.


On Fri, Sep 20, 2024 at 2:01 AM Jason Wen  wrote:

> What’s the latest status on Java 17 runtime support for Hadoop? Do we have
> a roadmap when and what the next Hadoop release will support it?
>
>
>
> Thanks,
>
> Jason
>
>
>
> *From: *slfan1989 
> *Date: *Wednesday, February 7, 2024 at 12:22 AM
> *To: *Xiaoqiao He , Ayush Saxena <
> ayush...@gmail.com>, ste...@cloudera.com , Takanobu
> Asanuma , Hadoop Common ,
> yarn-dev , mapreduce-dev <
> mapreduce-...@hadoop.apache.org>, Hdfs-dev 
> *Subject: *[External Sender] Re: JDK17 support for Hadoop
>
> Thank you all for your interest in this topic!
>
> In my opinion, upgrading is very beneficial. Based on the roadmap for Spark
> 4.0, we can see that JDK8/JDK11 is no longer the default version.
> Therefore, I believe it is important for us to fully support
> JDK11/JDK17/JDK21 in Hadoop.
>
> From the information I have gathered, we currently do not have full support
> for JDK11. I think it is necessary for us to address this issue first and
> gradually move towards supporting JDK17.
>
> I'd like to hear other members' opinions.
>
> I am willing to start this work to make hadoop support JDK11 and JDK17
> after the release of hadoop-3.4.0.
>
> Before that, we need to hear the opinions of other team members.
>
> CC: @Xiaoqiao He  @Ayush Saxena  >
> @ste...@cloudera.com  @Takanobu Asanuma
> 
>
> Original
>
> From:"Battula, Brahma Reddy"< bbatt...@visa.com.INVALID >;
>
> Date:2024/2/7 15:45
>
> To:"bilwa st"< stbi...@gmail.com >;"Hadoop Common"<
> common-...@hadoop.apache.org >;"Hdfs-dev"< hdfs-...@hadoop.apache.org >;
> "yarn-dev"< yarn-dev@hadoop.apache.org >;"mapreduce-dev"<
> mapreduce-...@hadoop.apache.org >;
>
> Subject:Re: JDK17 support for Hadoop
>
> Thanks, Bilwa, for bringing up the issue of JDK17 support for Hadoop.
>
> It's true that we need to have pipelines for JDK17 builds, and it's good to
> see that some Jira’s are already in progress to support it.
>
> We may need to check with the infrastructure team as well. Does anyone else
> have any thoughts or concerns on this?
>
> It seems like this topic is not very active.
>
>
> From: bilwa st
> Date: Monday, February 5, 2024 at 13:22
> To: Hadoop Common , Hdfs-dev , yarn-dev , mapreduce-dev
> Subject: Jdk17
> Hi folks,
>
> This is regarding jdk17 pending work to be done. Can we have a new pipeline
> for jdk17 builds? I can see that all jira's under HADOOP-17177 which got
> merged are not run on jdk17 as of now. It would be helpful if we have
> dedicated pipeline.
>
> Thanks,
> Bilwa
>

Re: Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

[slfan1989]

+1 (binding)

* Successfully built from source with `mvn clean install`
* Verified signatures and checksums

Thanks Steve and Mukund for driving this release.

Hope the release goes smoothly.

Shilun Fan.

On Fri, Aug 23, 2024 at 11:41 PM Shilun Fan wrote:

+1 (Binding)
>
> * Built from source
> * Verified Signature(Signed by Mukund)
> * Verified Checksums
> * No diff b/w the git tag & src tar
> * Verified the NOTICE & LICENSE files
> * Skimmed over the contents of the site jar
> * Built Hadoop trunk with 1.3.0 version
> * Checked the hadoop test results here [1], nothing looks related
>
> Thanx Steve & Mukund for driving the release, Good Luck!!!
>
> -Ayush
>
> [1]
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6707/28/testReport/
>
> On Fri, 23 Aug 2024 at 12:15, Xiaoqiao He wrote:
> >
> > +1 (binding).
> >
> > Thanks Steve and Mukund for driving this release.
> >
> > Best Regards
> > - He Xiaoqiao
> >
> > On Wed, Aug 21, 2024 at 11:51 PM Mukund Madhav Thakur
> > wrote:
> >
> > > +1 ( binding)
> > >
> > > On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran
>  > > >
> > > wrote:
> > >
> > > > Mukund and I have built a release candidate (RC1) for
> Hadoop-Thirdparty
> > > > 1.3.0.
> > > >
> > > > The RC is available at:
> > > >
> > >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/
> > > >
> > > > The git tag is release-1.3.0-RC1, commit
> > > > 0fd62903b071b5186f31b7030ce42e1c00f6bb6a
> > > >
> > > > The maven artifacts are staged at
> > > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1420/
> > > >
> > > > We've already been using the snapshot artifacts on hadoop trunk for
> the
> > > > protobuf update for a while, so we know that bit seems good.
> > > > Once this release is out we can adopt it in the 3.4.1 release (for
> better
> > > > java 8 compatibility as well as updates).
> > > >
> > > > I have created a hadoop PR for the switch
> > > > https://github.com/apache/hadoop/pull/7007
> > > >
> > > > This also updates the (transitive) binary license. The latest patch
> also
> > > > tries to update docker to the same version of protoc -which hasn't
> been
> > > > done until now. Let's see what happens.
> > > >
> > > > Please try the release and vote. The vote will run for 5 days.
> > > >
> > > > Here is my vote:
> > > >
> > > > +1 (binding)
> > > >
> > >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

[Ayush Saxena]

+1 (Binding)

* Built from source
* Verified Signature(Signed by Mukund)
* Verified Checksums
* No diff b/w the git tag & src tar
* Verified the NOTICE & LICENSE files
* Skimmed over the contents of the site jar
* Built Hadoop trunk with 1.3.0 version
* Checked the hadoop test results here [1], nothing looks related

Thanx Steve & Mukund for driving the release, Good Luck!!!

-Ayush

[1] 
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6707/28/testReport/

On Fri, 23 Aug 2024 at 12:15, Xiaoqiao He  wrote:
>
> +1 (binding).
>
> Thanks Steve and Mukund for driving this release.
>
> Best Regards
> - He Xiaoqiao
>
> On Wed, Aug 21, 2024 at 11:51 PM Mukund Madhav Thakur
>  wrote:
>
> > +1 ( binding)
> >
> > On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran  > >
> > wrote:
> >
> > > Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty
> > > 1.3.0.
> > >
> > > The RC is available at:
> > >
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/
> > >
> > > The git tag is release-1.3.0-RC1, commit
> > > 0fd62903b071b5186f31b7030ce42e1c00f6bb6a
> > >
> > > The maven artifacts are staged at
> > > https://repository.apache.org/content/repositories/orgapachehadoop-1420/
> > >
> > > We've already been using the snapshot artifacts on hadoop trunk for the
> > > protobuf update for a while, so we know that bit seems good.
> > > Once this release is out we can adopt it in the 3.4.1 release (for better
> > > java 8 compatibility as well as updates).
> > >
> > > I have created a hadoop PR for the switch
> > > https://github.com/apache/hadoop/pull/7007
> > >
> > > This also updates the (transitive) binary license. The latest patch also
> > > tries to update docker to the same version of protoc -which hasn't been
> > > done until now. Let's see what happens.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > Here is my vote:
> > >
> > > +1 (binding)
> > >
> >

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

[Xiaoqiao He]

+1 (binding).

Thanks Steve and Mukund for driving this release.

Best Regards
- He Xiaoqiao

On Wed, Aug 21, 2024 at 11:51 PM Mukund Madhav Thakur
 wrote:

> +1 ( binding)
>
> On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran  >
> wrote:
>
> > Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty
> > 1.3.0.
> >
> > The RC is available at:
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/
> >
> > The git tag is release-1.3.0-RC1, commit
> > 0fd62903b071b5186f31b7030ce42e1c00f6bb6a
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1420/
> >
> > We've already been using the snapshot artifacts on hadoop trunk for the
> > protobuf update for a while, so we know that bit seems good.
> > Once this release is out we can adopt it in the 3.4.1 release (for better
> > java 8 compatibility as well as updates).
> >
> > I have created a hadoop PR for the switch
> > https://github.com/apache/hadoop/pull/7007
> >
> > This also updates the (transitive) binary license. The latest patch also
> > tries to update docker to the same version of protoc -which hasn't been
> > done until now. Let's see what happens.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Here is my vote:
> >
> > +1 (binding)
> >
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

[slfan1989]

My oversight led to executing the wrong command.

The command I actually needed to run was `mvn clean install -DskipTests`,
which is the expected one. I will continue with my verification.

[INFO]

[INFO] Reactor Summary for Apache Hadoop Third-party Libs 1.3.0:
[INFO]
[INFO] Apache Hadoop Third-party Libs . SUCCESS [
 0.958 s]
[INFO] Apache Hadoop shaded Protobuf .. SUCCESS [
 1.481 s]
[INFO] Apache Hadoop shaded Guava . SUCCESS [
 1.043 s]
[INFO] Apache Hadoop shaded Avro 1.11 . SUCCESS [
 0.350 s]
[INFO]

[INFO] BUILD SUCCESS
[INFO]

[INFO] Total time:  4.385 s
[INFO] Finished at: 2024-08-22T22:54:38+08:00
[INFO]




On Thu, Aug 22, 2024 at 9:35 AM slfan1989  wrote:

>
> I encountered some issues while trying to compile with version
> hadoop-thirdparty-1.3.0-RC1.
>
> Using mvn package clean -DskipTests succeeds in compiling, but using mvn
> install package -DskipTests fails to compile.
>
> mvn clean package -DskipTests
>
> [INFO]
> 
> [INFO] Reactor Summary for Apache Hadoop Third-party Libs 1.3.0:
> [INFO]
> [INFO] Apache Hadoop Third-party Libs . SUCCESS [
>  0.862 s]
> [INFO] Apache Hadoop shaded Protobuf .. SUCCESS [
>  1.521 s]
> [INFO] Apache Hadoop shaded Guava . SUCCESS [
>  1.027 s]
> [INFO] Apache Hadoop shaded Avro 1.11 . SUCCESS [
>  0.408 s]
> [INFO]
> 
> [INFO] BUILD SUCCESS
> [INFO]
> 
> [INFO] Total time:  4.261 s
> [INFO] Finished at: 2024-08-22T09:32:46+08:00
> [INFO]
> 
>
> mvn install package -DskipTests
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (shade-protobuf) on
> project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate
> entry: META-INF/LICENSE.txt -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :hadoop-shaded-protobuf_3_25
>
>
> Shilun Fan.
>
> On Thu, Aug 22, 2024 at 9:07 AM Mukund Madhav Thakur
>
> +1 ( binding)
>>
>> On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran
>> wrote:
>>
>> > Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty
>> > 1.3.0.
>> >
>> > The RC is available at:
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/
>> >
>> > The git tag is release-1.3.0-RC1, commit
>> > 0fd62903b071b5186f31b7030ce42e1c00f6bb6a
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1420/
>> >
>> > We've already been using the snapshot artifacts on hadoop trunk for the
>> > protobuf update for a while, so we know that bit seems good.
>> > Once this release is out we can adopt it in the 3.4.1 release (for
>> better
>> > java 8 compatibility as well as updates).
>> >
>> > I have created a hadoop PR for the switch
>> > https://github.com/apache/hadoop/pull/7007
>> >
>> > This also updates the (transitive) binary license. The latest patch also
>> > tries to update docker to the same version of protoc -which hasn't been
>> > done until now. Let's see what happens.
>> >
>> > Please try the release and vote. The vote will run for 5 days.
>> >
>> > Here is my vote:
>> >
>> > +1 (binding)
>> >
>>
>>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

[Steve Loughran]

looks like there is some inability of maven to handle the double invocation
where the manifest LICENSE.txt is built by copying in /LICENSE-binary


I've just checked and this happens on branch 1.2.0 as well (64b6721). So it
is not a regression, just a way for the build process to fail.

because its not a regression, I don't think its a blocker. but we should
mention it in our release notes about how to build this
- arm64 doesn't work
- you can't do package and install at the same tiime.

Doesn't "mvn package" do an install first? I get confused here

On Thu, 22 Aug 2024 at 02:36, slfan1989  wrote:

> I encountered some issues while trying to compile with version
> hadoop-thirdparty-1.3.0-RC1.
>
> Using mvn package clean -DskipTests succeeds in compiling, but using mvn
> install package -DskipTests fails to compile.
>
> mvn clean package -DskipTests
>
> [INFO]
> 
> [INFO] Reactor Summary for Apache Hadoop Third-party Libs 1.3.0:
> [INFO]
> [INFO] Apache Hadoop Third-party Libs . SUCCESS [
>  0.862 s]
> [INFO] Apache Hadoop shaded Protobuf .. SUCCESS [
>  1.521 s]
> [INFO] Apache Hadoop shaded Guava . SUCCESS [
>  1.027 s]
> [INFO] Apache Hadoop shaded Avro 1.11 . SUCCESS [
>  0.408 s]
> [INFO]
> 
> [INFO] BUILD SUCCESS
> [INFO]
> 
> [INFO] Total time:  4.261 s
> [INFO] Finished at: 2024-08-22T09:32:46+08:00
> [INFO]
> 
>
> mvn install package -DskipTests
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (shade-protobuf) on
> project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate
> entry: META-INF/LICENSE.txt -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR]   mvn  -rf :hadoop-shaded-protobuf_3_25
>
>
> Shilun Fan.
>
> On Thu, Aug 22, 2024 at 9:07 AM Mukund Madhav Thakur
>
> +1 ( binding)
> >
> > On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran
> > wrote:
> >
> > > Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty
> > > 1.3.0.
> > >
> > > The RC is available at:
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/
> > >
> > > The git tag is release-1.3.0-RC1, commit
> > > 0fd62903b071b5186f31b7030ce42e1c00f6bb6a
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1420/
> > >
> > > We've already been using the snapshot artifacts on hadoop trunk for the
> > > protobuf update for a while, so we know that bit seems good.
> > > Once this release is out we can adopt it in the 3.4.1 release (for
> better
> > > java 8 compatibility as well as updates).
> > >
> > > I have created a hadoop PR for the switch
> > > https://github.com/apache/hadoop/pull/7007
> > >
> > > This also updates the (transitive) binary license. The latest patch
> also
> > > tries to update docker to the same version of protoc -which hasn't been
> > > done until now. Let's see what happens.
> > >
> > > Please try the release and vote. The vote will run for 5 days.
> > >
> > > Here is my vote:
> > >
> > > +1 (binding)
> > >
> >
> >
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

[slfan1989]

I encountered some issues while trying to compile with version
hadoop-thirdparty-1.3.0-RC1.

Using mvn package clean -DskipTests succeeds in compiling, but using mvn
install package -DskipTests fails to compile.

mvn clean package -DskipTests

[INFO]

[INFO] Reactor Summary for Apache Hadoop Third-party Libs 1.3.0:
[INFO]
[INFO] Apache Hadoop Third-party Libs . SUCCESS [
 0.862 s]
[INFO] Apache Hadoop shaded Protobuf .. SUCCESS [
 1.521 s]
[INFO] Apache Hadoop shaded Guava . SUCCESS [
 1.027 s]
[INFO] Apache Hadoop shaded Avro 1.11 . SUCCESS [
 0.408 s]
[INFO]

[INFO] BUILD SUCCESS
[INFO]

[INFO] Total time:  4.261 s
[INFO] Finished at: 2024-08-22T09:32:46+08:00
[INFO]


mvn install package -DskipTests

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (shade-protobuf) on
project hadoop-shaded-protobuf_3_25: Error creating shaded jar: duplicate
entry: META-INF/LICENSE.txt -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions,
please read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the
command
[ERROR]   mvn  -rf :hadoop-shaded-protobuf_3_25


Shilun Fan.

On Thu, Aug 22, 2024 at 9:07 AM Mukund Madhav Thakur

+1 ( binding)
>
> On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran
> wrote:
>
> > Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty
> > 1.3.0.
> >
> > The RC is available at:
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/
> >
> > The git tag is release-1.3.0-RC1, commit
> > 0fd62903b071b5186f31b7030ce42e1c00f6bb6a
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1420/
> >
> > We've already been using the snapshot artifacts on hadoop trunk for the
> > protobuf update for a while, so we know that bit seems good.
> > Once this release is out we can adopt it in the 3.4.1 release (for better
> > java 8 compatibility as well as updates).
> >
> > I have created a hadoop PR for the switch
> > https://github.com/apache/hadoop/pull/7007
> >
> > This also updates the (transitive) binary license. The latest patch also
> > tries to update docker to the same version of protoc -which hasn't been
> > done until now. Let's see what happens.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > Here is my vote:
> >
> > +1 (binding)
> >
>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.3.0

[Mukund Madhav Thakur]

+1 ( binding)

On Tue, Aug 20, 2024 at 9:16 AM Steve Loughran 
wrote:

> Mukund and I have built a release candidate (RC1) for Hadoop-Thirdparty
> 1.3.0.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.3.0-RC1/
>
> The git tag is release-1.3.0-RC1, commit
> 0fd62903b071b5186f31b7030ce42e1c00f6bb6a
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1420/
>
> We've already been using the snapshot artifacts on hadoop trunk for the
> protobuf update for a while, so we know that bit seems good.
> Once this release is out we can adopt it in the 3.4.1 release (for better
> java 8 compatibility as well as updates).
>
> I have created a hadoop PR for the switch
> https://github.com/apache/hadoop/pull/7007
>
> This also updates the (transitive) binary license. The latest patch also
> tries to update docker to the same version of protoc -which hasn't been
> done until now. Let's see what happens.
>
> Please try the release and vote. The vote will run for 5 days.
>
> Here is my vote:
>
> +1 (binding)
>

Re: Need help with gpg signing while creating 3.4.1 release using MacOS M3

[Mukund Madhav Thakur]

And it works when I run the command manually inside the docker container.


On Thu, Aug 1, 2024 at 1:52 PM Mukund Madhav Thakur 
wrote:

> yes it was not backported to the trunk branch.
>
> After skipping the sign step, the build fails in yarn-ui with this error.
> I have tried a few times by now but it always fails so it's not
> intermittent. There was some python failure as well before this which I
> fixed by setting the PYTHON env variable in the docker file. Any idea about
> this error? Thanks
>
> [INFO] --- exec-maven-plugin:1.3.1:exec (ember build) @ hadoop-yarn-ui ---
> (node:10063) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use
> os.tmpdir() instead.
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-array-contains-helper -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-bootstrap -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-bootstrap -> ember-wormhole -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-cli-app-version -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-cli-htmlbars-inline-precompile -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-cli-jquery-ui -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> moment ->
> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> ember-d3 ->
> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-disable-proxy-controllers -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> lodash ->
> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> ember-resolver
> -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-spin-spinner -> ember-cli-babel
> DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
> least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
> ember-truth-helpers -> ember-cli-babel
> broccoli-babel-transpiler is opting out of caching due to a plugin that
> does not provide a caching strategy: `function(babel) {
> var t = babel.types;
> ...
> ..
> ..
> ..
>
> ENOENT: no such file or directory, stat
> '/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/tmp/broccoli_merge_trees-input_base_path-rp0oGH9N.tmp/.bin/phantomjs'
>
> Error: ENOENT: no such file or directory, stat
> '/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/tmp/broccoli_merge_trees-input_base_path-rp0oGH9N.tmp/.bin/phantomjs'
>
> at Object.statSync (fs.js:1016:3)
>
> at buildEntry
> (/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:343:17)
>
> at BroccoliMergeTrees._mergeRelativePath
> (/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:251:19)
>
> at BroccoliMergeTrees._mergeRelativePath
> (/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:304:31)
>
> at BroccoliMergeTrees.build
> (/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:83:24)
>
> at
> /build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/broccoli-plugin/read_compat.js:93:34
>
> at tryCatch
> (/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/rsvp/dist/rsvp.js:525:12)
>
> at invokeCallback
> (/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/rsvp/dist/rsvp.js:538:13)
>
> at publish
> (/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/rsvp/dist/rsvp.js:508:7)
>
> at flush
> (/build/sou

Re: Need help with gpg signing while creating 3.4.1 release using MacOS M3

[Mukund Madhav Thakur]

yes it was not backported to the trunk branch.

After skipping the sign step, the build fails in yarn-ui with this error. I
have tried a few times by now but it always fails so it's not intermittent.
There was some python failure as well before this which I fixed by setting
the PYTHON env variable in the docker file. Any idea about this error?
Thanks

[INFO] --- exec-maven-plugin:1.3.1:exec (ember build) @ hadoop-yarn-ui ---
(node:10063) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use
os.tmpdir() instead.
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-array-contains-helper -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-bootstrap -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-bootstrap -> ember-wormhole -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-cli-app-version -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-cli-htmlbars-inline-precompile -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-cli-jquery-ui -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> moment ->
ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> ember-d3 ->
ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-disable-proxy-controllers -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> lodash ->
ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui -> ember-resolver
-> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-spin-spinner -> ember-cli-babel
DEPRECATION: ember-cli-babel 5.x has been deprecated. Please upgrade to at
least ember-cli-babel 6.6. Version 5.2.8 located: yarn-ui ->
ember-truth-helpers -> ember-cli-babel
broccoli-babel-transpiler is opting out of caching due to a plugin that
does not provide a caching strategy: `function(babel) {
var t = babel.types;
...
..
..
..

ENOENT: no such file or directory, stat
'/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/tmp/broccoli_merge_trees-input_base_path-rp0oGH9N.tmp/.bin/phantomjs'

Error: ENOENT: no such file or directory, stat
'/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/tmp/broccoli_merge_trees-input_base_path-rp0oGH9N.tmp/.bin/phantomjs'

at Object.statSync (fs.js:1016:3)

at buildEntry
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:343:17)

at BroccoliMergeTrees._mergeRelativePath
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:251:19)

at BroccoliMergeTrees._mergeRelativePath
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:304:31)

at BroccoliMergeTrees.build
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/ember-cli/node_modules/broccoli-merge-trees/index.js:83:24)

at
/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/broccoli-plugin/read_compat.js:93:34

at tryCatch
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/rsvp/dist/rsvp.js:525:12)

at invokeCallback
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/rsvp/dist/rsvp.js:538:13)

at publish
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/rsvp/dist/rsvp.js:508:7)

at flush
(/build/source/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp/node_modules/rsvp/dist/rsvp.js:2415:5)

error Command failed with exit code 1.

yarn run v1.22.5

$ TMPDIR=tmp node/node ./node_modules/ember-cli/bin/ember build -prod

WARNING: Node v12.22.1 has currently not been tested against Ember CLI an

Re: [DISCUSS] what needs to be done to switch to java17

[Ayush Saxena]

>  One thing I know is that it is necessary to upgrade JUnit 4 to JUnit 5

Do you know any specific reason or docs mentioning that? Just curious
as to why we need to migrate to JUnit 5 for JDK-17, Looking here [1],
it seems like it supports JDK-21 as well and even [2] mentions like it
does support jdk-17

> By the way, there's a JIRA that we have had to revert twice 
> (HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin.

I am bit more curious on this one as well, the maven surefire doc [3]
just mentions the minimum version for all the releases, does the
latest one on 3.0.0-M* gives us a pass, IIRC 3.0.0-M4 is working for
Hive JDK-17 (In Progress) and atleast that isn't creating a problem
over there. Looking at this ticket [4], it seems like 3.0.0-M6 with
Junit-4 works with JDK-17 or am I missing something? Can we attempt
that version & unblock JDK-17 efforts, if it indeed blocks us?
3.0.0-M8 in future if we want to push for JDK-21[5]

> Additionally, we are using Hadoop compiled with JDK 17 internally (mainly on 
> the current trunk branch with --add-opens=). We have replaced over 600 NM 
> instances, and this part is functioning normally. Therefore, I believe we are 
> essentially ready to upgrade to JDK 17.

That sounds cool, that means you folks have already solved all the
problems like Jersey + any other on trunk & it is just a matter of
rebasing those fixes & just pushing it to the apache repo or is there
some catch which I missed?


-Ayush

[1] 
https://github.com/junit-team/junit4/blob/main/.github/workflows/main.yml#L22
[2] https://github.com/junit-team/junit4/issues/1770
[3] https://maven.apache.org/surefire/maven-surefire-plugin/plugin-info.html
[4] https://issues.apache.org/jira/browse/SUREFIRE-2007
[5] https://issues.apache.org/jira/browse/SUREFIRE-2135

On Tue, 30 Jul 2024 at 07:58, Ashutosh Gupta  wrote:
>
> Thanks, everyone, for starting this discussion. This sounds like a good plan 
> to start with. As I was working on the JUnit 4 to 5 upgrade, I paused for a 
> while as I got occupied with other stuff. But I would be happy to complete it 
> as part of the Java switch process.
>
> On Tue, Jul 30, 2024 at 4:21 AM slfan1989  wrote:
>>
>> Thank you very much for initiating this discussion! I am also very much
>> looking forward to JDK 17. I have observed that using --add-opens= is quite
>> common in other Apache projects. One thing I know is that it is necessary
>> to upgrade JUnit 4 to JUnit 5. There are some JIRA issues currently being
>> worked on for this. For YARN MapReduce, some pull requests have already
>> been completed for this feature. Upgrading Jersey is a very challenging
>> project. Currently, our upgrade strategy is to perform it all at once. If
>> we can find a way to break Jersey down into individual modules for
>> upgrading, it might be much easier.
>>
>> I have successfully debugged most of the unit tests for YARN's
>> ResourceManager and NodeManager.(Upgrade Jersey 2.4.1)  Additionally, we
>> are using Hadoop compiled with JDK 17 internally (mainly on the current
>> trunk branch with --add-opens=). We have replaced over 600 NM instances,
>> and this part is functioning normally. Therefore, I believe we are
>> essentially ready to upgrade to JDK 17.
>>
>> My idea is to create a separate branch specifically for upgrading to JDK
>> 17. After debugging and ensuring everything works, we can then merge this
>> branch into the trunk branch.
>>
>> By the way, there's a JIRA that we have had to revert twice
>> (HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin.
>>
>> [1] https://issues.apache.org/jira/browse/HADOOP-19071
>>
>> - Shilun Fan
>>
>> On Tue, 30 Jul 2024 at 06:37, Shilun Fan wrote:
>>
>> > Depends on what we mean by switching to JDK-17, Compile time support
>> > or Runtime support, We don't have compile time support for JDK-11 too,
>> > it is just runtime, We have a daily build as well for JDK-11, which
>> > has now some genuine failures now, need to check [1], as of now the
>> > version is hardcoded here [2], unless we change here it but just
>> > change our local JAVA_HOME, it will compile with JDK-17 as well I
>> > believe
>>
>> > So, If we are to chase the runtime support we can setup a regular CI
>> > with JDK-17 like we have one for JDK-11, there were some test failures
>> > reported as part of HADOOP-18716, most of them should be fixable by
>> > some dependency upgrades or some hacks like adding one of these
>> > ```
>> > --add-opens=java.base/java.net=ALL-UNNAMED
>> > --add-opens=java.base/java.util=ALL-UNNAMED
>> > --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
>> > --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
>> > --add-opens=java.base/java.lang=ALL-UNNAMED
>> > --add-opens=java.base/java.util.regex=ALL-UNNAMED
>> > --add-opens=java.base/java.io=ALL-UNNAMED
>> > ```
>>
>> > That is some typical JDK-17 hack
>>
>> > For the compile support, that is we change the value at [2] to 17, I
>> > think the

Re: [DISCUSS] what needs to be done to switch to java17

[Ashutosh Gupta]

Thanks, everyone, for starting this discussion. This sounds like a good
plan to start with. As I was working on the JUnit 4 to 5 upgrade, I paused
for a while as I got occupied with other stuff. But I would be happy to
complete it as part of the Java switch process.

On Tue, Jul 30, 2024 at 4:21 AM slfan1989  wrote:

> Thank you very much for initiating this discussion! I am also very much
> looking forward to JDK 17. I have observed that using --add-opens= is quite
> common in other Apache projects. One thing I know is that it is necessary
> to upgrade JUnit 4 to JUnit 5. There are some JIRA issues currently being
> worked on for this. For YARN MapReduce, some pull requests have already
> been completed for this feature. Upgrading Jersey is a very challenging
> project. Currently, our upgrade strategy is to perform it all at once. If
> we can find a way to break Jersey down into individual modules for
> upgrading, it might be much easier.
>
> I have successfully debugged most of the unit tests for YARN's
> ResourceManager and NodeManager.(Upgrade Jersey 2.4.1)  Additionally, we
> are using Hadoop compiled with JDK 17 internally (mainly on the current
> trunk branch with --add-opens=). We have replaced over 600 NM instances,
> and this part is functioning normally. Therefore, I believe we are
> essentially ready to upgrade to JDK 17.
>
> My idea is to create a separate branch specifically for upgrading to JDK
> 17. After debugging and ensuring everything works, we can then merge this
> branch into the trunk branch.
>
> By the way, there's a JIRA that we have had to revert twice
> (HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin.
>
> [1] https://issues.apache.org/jira/browse/HADOOP-19071
>
> - Shilun Fan
>
> On Tue, 30 Jul 2024 at 06:37, Shilun Fan wrote:
>
> > Depends on what we mean by switching to JDK-17, Compile time support
> > or Runtime support, We don't have compile time support for JDK-11 too,
> > it is just runtime, We have a daily build as well for JDK-11, which
> > has now some genuine failures now, need to check [1], as of now the
> > version is hardcoded here [2], unless we change here it but just
> > change our local JAVA_HOME, it will compile with JDK-17 as well I
> > believe
>
> > So, If we are to chase the runtime support we can setup a regular CI
> > with JDK-17 like we have one for JDK-11, there were some test failures
> > reported as part of HADOOP-18716, most of them should be fixable by
> > some dependency upgrades or some hacks like adding one of these
> > ```
> > --add-opens=java.base/java.net=ALL-UNNAMED
> > --add-opens=java.base/java.util=ALL-UNNAMED
> > --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
> > --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
> > --add-opens=java.base/java.lang=ALL-UNNAMED
> > --add-opens=java.base/java.util.regex=ALL-UNNAMED
> > --add-opens=java.base/java.io=ALL-UNNAMED
> > ```
>
> > That is some typical JDK-17 hack
>
> > For the compile support, that is we change the value at [2] to 17, I
> > think then some bigger problems will surface, One I know is Jersey
> > creates some issues for sure, We need to upgrade Jersey, As per this
> > [3] to Jersey 2.35+ or do some hack to use a patched jersey version
> > with minimal change which can unblock the JDK-17 initiative
>
> > -Ayush
>
>
> > [1]
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/709/
> > [2]
>
> https://github.com/apache/hadoop/blob/038636a1b5250e06622cac7ee11b12965c9e/hadoop-project/pom.xml#L160
> > [3] https://github.com/eclipse-ee4j/jersey/wiki/Road-Map
>
> On Tue, 30 Jul 2024 at 00:53, PJ Fanning wrote:
> >
> > I think this is worth considering. I think it would require a minor
> > release like 3.5.0 as opposed to considering it for future 3.4.x patch
> > releases.
> > I tend to build locally with Java 11, by default and I haven't hit
> > major issues building Hadoop. There may be some gotcha somewhere but
> > it is likely to be easy enough to fix.
> >
> > It's probably only a matter of time before some important dependency
> > jars for Hadoop require Java 11 or 17.
> >
> > On Mon, 29 Jul 2024 at 20:04, Steve Loughran
> > wrote:
> > >
> > > A lot of projects are moving off java8. making java17 the new baseline
> > >
> > > what do we need to there that is blocker rather than just "nice"'?
> >
> > -
> > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

Re: [DISCUSS] what needs to be done to switch to java17

[slfan1989]

Thank you very much for initiating this discussion! I am also very much
looking forward to JDK 17. I have observed that using --add-opens= is quite
common in other Apache projects. One thing I know is that it is necessary
to upgrade JUnit 4 to JUnit 5. There are some JIRA issues currently being
worked on for this. For YARN MapReduce, some pull requests have already
been completed for this feature. Upgrading Jersey is a very challenging
project. Currently, our upgrade strategy is to perform it all at once. If
we can find a way to break Jersey down into individual modules for
upgrading, it might be much easier.

I have successfully debugged most of the unit tests for YARN's
ResourceManager and NodeManager.(Upgrade Jersey 2.4.1)  Additionally, we
are using Hadoop compiled with JDK 17 internally (mainly on the current
trunk branch with --add-opens=). We have replaced over 600 NM instances,
and this part is functioning normally. Therefore, I believe we are
essentially ready to upgrade to JDK 17.

My idea is to create a separate branch specifically for upgrading to JDK
17. After debugging and ensuring everything works, we can then merge this
branch into the trunk branch.

By the way, there's a JIRA that we have had to revert twice
(HADOOP-19071)[1], which involves upgrading the maven-surefire-plugin.

[1] https://issues.apache.org/jira/browse/HADOOP-19071

- Shilun Fan

On Tue, 30 Jul 2024 at 06:37, Shilun Fan wrote:

> Depends on what we mean by switching to JDK-17, Compile time support
> or Runtime support, We don't have compile time support for JDK-11 too,
> it is just runtime, We have a daily build as well for JDK-11, which
> has now some genuine failures now, need to check [1], as of now the
> version is hardcoded here [2], unless we change here it but just
> change our local JAVA_HOME, it will compile with JDK-17 as well I
> believe

> So, If we are to chase the runtime support we can setup a regular CI
> with JDK-17 like we have one for JDK-11, there were some test failures
> reported as part of HADOOP-18716, most of them should be fixable by
> some dependency upgrades or some hacks like adding one of these
> ```
> --add-opens=java.base/java.net=ALL-UNNAMED
> --add-opens=java.base/java.util=ALL-UNNAMED
> --add-opens=java.base/java.util.concurrent=ALL-UNNAMED
> --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
> --add-opens=java.base/java.lang=ALL-UNNAMED
> --add-opens=java.base/java.util.regex=ALL-UNNAMED
> --add-opens=java.base/java.io=ALL-UNNAMED
> ```

> That is some typical JDK-17 hack

> For the compile support, that is we change the value at [2] to 17, I
> think then some bigger problems will surface, One I know is Jersey
> creates some issues for sure, We need to upgrade Jersey, As per this
> [3] to Jersey 2.35+ or do some hack to use a patched jersey version
> with minimal change which can unblock the JDK-17 initiative

> -Ayush


> [1]
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/709/
> [2]
https://github.com/apache/hadoop/blob/038636a1b5250e06622cac7ee11b12965c9e/hadoop-project/pom.xml#L160
> [3] https://github.com/eclipse-ee4j/jersey/wiki/Road-Map

On Tue, 30 Jul 2024 at 00:53, PJ Fanning wrote:
>
> I think this is worth considering. I think it would require a minor
> release like 3.5.0 as opposed to considering it for future 3.4.x patch
> releases.
> I tend to build locally with Java 11, by default and I haven't hit
> major issues building Hadoop. There may be some gotcha somewhere but
> it is likely to be easy enough to fix.
>
> It's probably only a matter of time before some important dependency
> jars for Hadoop require Java 11 or 17.
>
> On Mon, 29 Jul 2024 at 20:04, Steve Loughran
> wrote:
> >
> > A lot of projects are moving off java8. making java17 the new baseline
> >
> > what do we need to there that is blocker rather than just "nice"'?
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] what needs to be done to switch to java17

[Ayush Saxena]

Depends on what we mean by switching to JDK-17, Compile time support
or Runtime support, We don't have compile time support for JDK-11 too,
it is just runtime, We have a daily build as well for JDK-11, which
has now some genuine failures now, need to check [1], as of now the
version is hardcoded here [2], unless we change here it but just
change our local JAVA_HOME, it will compile with JDK-17 as well I
believe

So, If we are to chase the runtime support we can setup a regular CI
with JDK-17 like we have one for JDK-11, there were some test failures
reported as part of HADOOP-18716, most of them should be fixable by
some dependency upgrades or some hacks like adding one of these
```
--add-opens=java.base/java.net=ALL-UNNAMED
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent=ALL-UNNAMED
--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
--add-opens=java.base/java.util.regex=ALL-UNNAMED
--add-opens=java.base/java.io=ALL-UNNAMED
```

That is some typical JDK-17 hack

For the compile support, that is we change the value at  [2] to 17, I
think then some bigger problems will surface, One I know is Jersey
creates some issues for sure, We need to upgrade Jersey, As per this
[3] to Jersey 2.35+ or do some hack to use a patched jersey version
with minimal change which can unblock the JDK-17 initiative

-Ayush


[1] https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/709/
[2] 
https://github.com/apache/hadoop/blob/038636a1b5250e06622cac7ee11b12965c9e/hadoop-project/pom.xml#L160
[3] https://github.com/eclipse-ee4j/jersey/wiki/Road-Map

On Tue, 30 Jul 2024 at 00:53, PJ Fanning  wrote:
>
> I think this is worth considering. I think it would require a minor
> release like 3.5.0 as opposed to considering it for future 3.4.x patch
> releases.
> I tend to build locally with Java 11, by default and I haven't hit
> major issues building Hadoop. There may be some gotcha somewhere but
> it is likely to be easy enough to fix.
>
> It's probably only a matter of time before some important dependency
> jars for Hadoop require Java 11 or 17.
>
> On Mon, 29 Jul 2024 at 20:04, Steve Loughran
>  wrote:
> >
> > A lot of projects are moving off java8. making java17 the new baseline
> >
> > what do we need to there that is blocker rather than just "nice"'?
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] what needs to be done to switch to java17

[PJ Fanning]

I think this is worth considering. I think it would require a minor
release like 3.5.0 as opposed to considering it for future 3.4.x patch
releases.
I tend to build locally with Java 11, by default and I haven't hit
major issues building Hadoop. There may be some gotcha somewhere but
it is likely to be easy enough to fix.

It's probably only a matter of time before some important dependency
jars for Hadoop require Java 11 or 17.

On Mon, 29 Jul 2024 at 20:04, Steve Loughran
 wrote:
>
> A lot of projects are moving off java8. making java17 the new baseline
>
> what do we need to there that is blocker rather than just "nice"'?

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

[Steve Loughran]

On Thu, 30 May 2024 at 03:47, Xiaoqiao He  wrote:

> Strong +1. One concerns, how we define 'unnecessary branches', which
> mean how to distinguish the branches someone accidentally created, I try
> to traverse some of them but didn't get one obvious rule. Thanks.
>


If we take a snapshot into an archive repository then nothing will be lost
forever (actually, github fork should let us do this, so someone could just
log in as hadoop yetus and fork it there, perhaps)

Then we'd just look to see

   1. what feature branches are merged. e.g MR-379 is actually yarn itself.
   2. what feature branches are abandoned
   3. what release branches can we retire (all of 0.x, 1.x, early 2.x).
   maybe every branch not getting active maintenance, leaving only release
   tags?


Keeping all the tags will still result in a large repo. but removing
feature branches will potentially be good as the final merge was inevitably
a squashed merge...the intermediate chain of commits can be purged provided
there aren't tags associated with them




> Best Regards,
> - He Xiaoqiao
>
>
>
> On Wed, May 29, 2024 at 10:11 PM Ayush Saxena  wrote:
>
>> +1 for the proposal, the first thing might be to just drop the
>> unnecessary branches which are either from dependabot or someone
>> accidentally created a branch in the main repo rather than in their
>> fork, there are many, I don't think we need them in the archived repo
>> either
>>
>> Regarding (3) If you mean just branches, then should be ok, maybe lets
>> not touch the release tags for now IMO
>>
>> Regrading the regex, I tried this locally & it works
>> ```
>> ayushsaxena@ayushsaxena hadoop % git tag --delete  `git tag --list
>> 'ozone*'`
>> Deleted tag 'ozone-0.2.1-alpha-RC0' (was 90b070452bc7)
>> Deleted tag 'ozone-0.3.0-alpha' (was e9921ebf7e8d)
>> Deleted tag 'ozone-0.3.0-alpha-RC0' (was 3fbd1f15b894)
>> Deleted tag 'ozone-0.3.0-alpha-RC1' (was cdad29240e52)
>> Deleted tag 'ozone-0.4.0-alpha-RC0' (was 07fd26ef6d8c)
>> Deleted tag 'ozone-0.4.0-alpha-RC1' (was c4f9a20bbe55)
>> Deleted tag 'ozone-0.4.0-alpha-RC2' (was 6860c595ed19)
>> Deleted tag 'ozone-0.4.1-alpha' (was 687173ff4be4)
>> Deleted tag 'ozone-0.4.1-alpha-RC0' (was 9062dac447c8)
>> ayushsaxena@ayushsaxena hadoop %
>>
>> ```
>>
>> -Ayush
>>
>> On Wed, 29 May 2024 at 18:07, Steve Loughran
>>  wrote:
>> >
>> > I'm waiting for a git update of trunk to complete, not having done it
>> since
>> > last week. The 1.8 GB download is taking a long time over a VPN.
>> >
>> > Updating files: 100% (8518/8518), done.
>> > Switched to branch 'trunk'
>> > Your branch is up to date with 'apache/trunk'.
>> > remote: Enumerating objects: 4142992, done.
>> > remote: Counting objects: 100% (4142972/4142972), done.
>> > remote: Compressing objects: 100% (503038/503038), done.
>> > ^Receiving objects:  11% (483073/4142404), 204.18 MiB | 7.05 MiB/s
>> > remote: Total 4142404 (delta 3583765), reused 4140936 (delta 3582453)
>> > Receiving objects: 100% (4142404/4142404), 1.80 GiB | 6.36 MiB/s, done.
>> > Resolving deltas:  42% (1505182/3583765)
>> > ...
>> >
>> >
>> > We have too many branches and too many tags, which makes for big
>> downloads
>> > and slow clones, as well as complaints from git whenever I manually push
>> > things to gitbox.apache.org.
>> >
>> > I think we can/should clean up, which can be done as
>> >
>> >
>> >1. Create a hadoop-source-archive repository,
>> >2. into which we add all of the current hadoop-repository. This
>> ensures
>> >all the history is preserved.
>> >3. Delete all the old release branches, where old is defined as,
>> maybe <
>> >2.6?
>> >4. feature branches which are merged/abandoned
>> >5. all the single JIRA branches which are the same thing, "MR-279"
>> being
>> >a key example. ozone-* probably too.
>> >6. Do some tag pruning too. (Is there a way to do this with
>> wildcards? I
>> >could use it locally...)
>> >
>> > With an archive repo, all the old development history for branches off
>> the
>> > current release chain + tags are still available, but the core repo is
>> > much, much smaller.
>> >
>> > What do people think?
>> >
>> > If others are interested, I'll need some help carefully getting the
>> > hadoop-source-archive repo up. We'd need to somehow get all of hadoop
>> trunk
>> > into it.
>> >
>> > Meanwhile, I will cull some merged feature branches.
>> >
>> > Steve
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>

Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

[Xiaoqiao He]

Strong +1. One concerns, how we define 'unnecessary branches', which
mean how to distinguish the branches someone accidentally created, I try
to traverse some of them but didn't get one obvious rule. Thanks.

Best Regards,
- He Xiaoqiao



On Wed, May 29, 2024 at 10:11 PM Ayush Saxena  wrote:

> +1 for the proposal, the first thing might be to just drop the
> unnecessary branches which are either from dependabot or someone
> accidentally created a branch in the main repo rather than in their
> fork, there are many, I don't think we need them in the archived repo
> either
>
> Regarding (3) If you mean just branches, then should be ok, maybe lets
> not touch the release tags for now IMO
>
> Regrading the regex, I tried this locally & it works
> ```
> ayushsaxena@ayushsaxena hadoop % git tag --delete  `git tag --list
> 'ozone*'`
> Deleted tag 'ozone-0.2.1-alpha-RC0' (was 90b070452bc7)
> Deleted tag 'ozone-0.3.0-alpha' (was e9921ebf7e8d)
> Deleted tag 'ozone-0.3.0-alpha-RC0' (was 3fbd1f15b894)
> Deleted tag 'ozone-0.3.0-alpha-RC1' (was cdad29240e52)
> Deleted tag 'ozone-0.4.0-alpha-RC0' (was 07fd26ef6d8c)
> Deleted tag 'ozone-0.4.0-alpha-RC1' (was c4f9a20bbe55)
> Deleted tag 'ozone-0.4.0-alpha-RC2' (was 6860c595ed19)
> Deleted tag 'ozone-0.4.1-alpha' (was 687173ff4be4)
> Deleted tag 'ozone-0.4.1-alpha-RC0' (was 9062dac447c8)
> ayushsaxena@ayushsaxena hadoop %
>
> ```
>
> -Ayush
>
> On Wed, 29 May 2024 at 18:07, Steve Loughran
>  wrote:
> >
> > I'm waiting for a git update of trunk to complete, not having done it
> since
> > last week. The 1.8 GB download is taking a long time over a VPN.
> >
> > Updating files: 100% (8518/8518), done.
> > Switched to branch 'trunk'
> > Your branch is up to date with 'apache/trunk'.
> > remote: Enumerating objects: 4142992, done.
> > remote: Counting objects: 100% (4142972/4142972), done.
> > remote: Compressing objects: 100% (503038/503038), done.
> > ^Receiving objects:  11% (483073/4142404), 204.18 MiB | 7.05 MiB/s
> > remote: Total 4142404 (delta 3583765), reused 4140936 (delta 3582453)
> > Receiving objects: 100% (4142404/4142404), 1.80 GiB | 6.36 MiB/s, done.
> > Resolving deltas:  42% (1505182/3583765)
> > ...
> >
> >
> > We have too many branches and too many tags, which makes for big
> downloads
> > and slow clones, as well as complaints from git whenever I manually push
> > things to gitbox.apache.org.
> >
> > I think we can/should clean up, which can be done as
> >
> >
> >1. Create a hadoop-source-archive repository,
> >2. into which we add all of the current hadoop-repository. This
> ensures
> >all the history is preserved.
> >3. Delete all the old release branches, where old is defined as,
> maybe <
> >2.6?
> >4. feature branches which are merged/abandoned
> >5. all the single JIRA branches which are the same thing, "MR-279"
> being
> >a key example. ozone-* probably too.
> >6. Do some tag pruning too. (Is there a way to do this with
> wildcards? I
> >could use it locally...)
> >
> > With an archive repo, all the old development history for branches off
> the
> > current release chain + tags are still available, but the core repo is
> > much, much smaller.
> >
> > What do people think?
> >
> > If others are interested, I'll need some help carefully getting the
> > hadoop-source-archive repo up. We'd need to somehow get all of hadoop
> trunk
> > into it.
> >
> > Meanwhile, I will cull some merged feature branches.
> >
> > Steve
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] pruning surplus branches/tags from the main hadoop repo

[Ayush Saxena]

+1 for the proposal, the first thing might be to just drop the
unnecessary branches which are either from dependabot or someone
accidentally created a branch in the main repo rather than in their
fork, there are many, I don't think we need them in the archived repo
either

Regarding (3) If you mean just branches, then should be ok, maybe lets
not touch the release tags for now IMO

Regrading the regex, I tried this locally & it works
```
ayushsaxena@ayushsaxena hadoop % git tag --delete  `git tag --list 'ozone*'`
Deleted tag 'ozone-0.2.1-alpha-RC0' (was 90b070452bc7)
Deleted tag 'ozone-0.3.0-alpha' (was e9921ebf7e8d)
Deleted tag 'ozone-0.3.0-alpha-RC0' (was 3fbd1f15b894)
Deleted tag 'ozone-0.3.0-alpha-RC1' (was cdad29240e52)
Deleted tag 'ozone-0.4.0-alpha-RC0' (was 07fd26ef6d8c)
Deleted tag 'ozone-0.4.0-alpha-RC1' (was c4f9a20bbe55)
Deleted tag 'ozone-0.4.0-alpha-RC2' (was 6860c595ed19)
Deleted tag 'ozone-0.4.1-alpha' (was 687173ff4be4)
Deleted tag 'ozone-0.4.1-alpha-RC0' (was 9062dac447c8)
ayushsaxena@ayushsaxena hadoop %

```

-Ayush

On Wed, 29 May 2024 at 18:07, Steve Loughran
 wrote:
>
> I'm waiting for a git update of trunk to complete, not having done it since
> last week. The 1.8 GB download is taking a long time over a VPN.
>
> Updating files: 100% (8518/8518), done.
> Switched to branch 'trunk'
> Your branch is up to date with 'apache/trunk'.
> remote: Enumerating objects: 4142992, done.
> remote: Counting objects: 100% (4142972/4142972), done.
> remote: Compressing objects: 100% (503038/503038), done.
> ^Receiving objects:  11% (483073/4142404), 204.18 MiB | 7.05 MiB/s
> remote: Total 4142404 (delta 3583765), reused 4140936 (delta 3582453)
> Receiving objects: 100% (4142404/4142404), 1.80 GiB | 6.36 MiB/s, done.
> Resolving deltas:  42% (1505182/3583765)
> ...
>
>
> We have too many branches and too many tags, which makes for big downloads
> and slow clones, as well as complaints from git whenever I manually push
> things to gitbox.apache.org.
>
> I think we can/should clean up, which can be done as
>
>
>1. Create a hadoop-source-archive repository,
>2. into which we add all of the current hadoop-repository. This ensures
>all the history is preserved.
>3. Delete all the old release branches, where old is defined as, maybe <
>2.6?
>4. feature branches which are merged/abandoned
>5. all the single JIRA branches which are the same thing, "MR-279" being
>a key example. ozone-* probably too.
>6. Do some tag pruning too. (Is there a way to do this with wildcards? I
>could use it locally...)
>
> With an archive repo, all the old development history for branches off the
> current release chain + tags are still available, but the core repo is
> much, much smaller.
>
> What do people think?
>
> If others are interested, I'll need some help carefully getting the
> hadoop-source-archive repo up. We'd need to somehow get all of hadoop trunk
> into it.
>
> Meanwhile, I will cull some merged feature branches.
>
> Steve

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] New Hadoop Committer - Haiyang Hu

[haiyang hu]

Thanks!  Looking forward to participating more in the Apache Hadoop
community!

Yuanbo Liu  于2024年4月22日周一 15:26写道：

> Congratulations
>
> On Mon, Apr 22, 2024 at 12:14 PM Ayush Saxena  wrote:
>
>> Congratulations Haiyang!!!
>>
>> -Ayush
>>
>> > On 22 Apr 2024, at 9:41 AM, Xiaoqiao He  wrote:
>> >
>> > I am pleased to announce that Haiyang Hu has been elected as
>> > a committer on the Apache Hadoop project. We appreciate all of
>> > Haiyang's work, and look forward to her/his continued contributions.
>> >
>> > Congratulations and Welcome, Haiyang!
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> > (On behalf of the Apache Hadoop PMC)
>> >
>> > -
>> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>>
>>

Re: [ANNOUNCE] New Hadoop Committer - Haiyang Hu

[Yuanbo Liu]

Congratulations

On Mon, Apr 22, 2024 at 12:14 PM Ayush Saxena  wrote:

> Congratulations Haiyang!!!
>
> -Ayush
>
> > On 22 Apr 2024, at 9:41 AM, Xiaoqiao He  wrote:
> >
> > I am pleased to announce that Haiyang Hu has been elected as
> > a committer on the Apache Hadoop project. We appreciate all of
> > Haiyang's work, and look forward to her/his continued contributions.
> >
> > Congratulations and Welcome, Haiyang!
> >
> > Best Regards,
> > - He Xiaoqiao
> > (On behalf of the Apache Hadoop PMC)
> >
> > -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

Re: [ANNOUNCE] New Hadoop Committer - Haiyang Hu

[Ayush Saxena]

Congratulations Haiyang!!!

-Ayush

> On 22 Apr 2024, at 9:41 AM, Xiaoqiao He  wrote:
> 
> I am pleased to announce that Haiyang Hu has been elected as
> a committer on the Apache Hadoop project. We appreciate all of
> Haiyang's work, and look forward to her/his continued contributions.
> 
> Congratulations and Welcome, Haiyang!
> 
> Best Regards,
> - He Xiaoqiao
> (On behalf of the Apache Hadoop PMC)
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Apache Hadoop 3.4.0 release

[Sammi Chen]

Xiaoqiao He and Shilun Fan

Awesome!  Thanks for leading the effort to release the Hadoop 3.4.0 !

Bests,
Sammi

On Tue, 19 Mar 2024 at 21:12, slfan1989  wrote:

> On behalf of the Apache Hadoop Project Management Committee, We are
> pleased to announce the release of Apache Hadoop 3.4.0.
>
> This is a release of Apache Hadoop 3.4 line.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> This is the first release of Apache Hadoop 3.4 line. It contains 2888 bug
> fixes, improvements and enhancements since 3.3.
>
> Users are encouraged to read the [overview of major changes][1].
> For details of please check [release notes][2] and [changelog][3].
>
> [1]: http://hadoop.apache.org/docs/r3.4.0/index.html
> [2]:
>
> http://hadoop.apache.org/docs/r3.4.0/hadoop-project-dist/hadoop-common/release/3.4.0/RELEASENOTES.3.4.0.html
> [3]:
>
> http://hadoop.apache.org/docs/r3.4.0/hadoop-project-dist/hadoop-common/release/3.4.0/CHANGELOG.3.4.0.html
>
> Many thanks to everyone who helped in this release by supplying patches,
> reviewing them, helping get this release building and testing and
> reviewing the final artifacts.
>
> Best Regards,
> Xiaoqiao He And Shilun Fan.
>

Re: ContainerId starts with 1 ?

[李响]

Dear Hadoop/Yarn community,

I still beg your help for the question above.

Additionally, I might have other questions.
The target is to get the driver container id of a Spark app, from Yarn
Aggregation Log. I would like to call
LogAggregationIndexedFileController#readAggregatedLogsMeta()

then get the first ContainLogMeta from the list returned, then call
getContainerId() from it.
The questions are:

   1. Is the first ContainerLogMeta always the driver container?
   2. If the driver failed to get up for the first time somehow, but
   succeed in its second try. The container id will be added by 1 if I
   understand it correctly. Under this case, will the first ContainLogMeta
   returned by that function above be the first failed container, or the
   second successful container? Or the container id gets not changed after a
   failure?

Thanks!


On Fri, Feb 23, 2024 at 4:21 PM 李响  wrote:

> Dear Hadoop/Yarn community,
>
> In Yarn, a container is represented as
> container_e*epoch*_*clusterTimestamp*_*appId*_*attemptId*_*containerId*
>
> Regarding the last section, "containerId", as the sequential number of
> containers, I notice it does not start with 0, but 1.
>
> My question is:
> 1. Is that observation correct?
> 2. Sorry I do not find the code to support that. I read ContainerId.java
> and ContainerIdPBImpl.java but does not find the answer. Could you please
> show me the code path to support it staring with 1?
> 3. It seems counter-intuitive for me, as a programmer ^_^, who thinks the
> index should start with 0, rather than 1. If it is designed to start with
> 1, any background / thought / discussion to share?
>
> Thanks !!!
>
>
>
> --
>
>李响 Xiang Li
>
>
>

-- 

   李响 Xiang Li

手机 cellphone ：+86-136-8113-8972
邮件 e-mail  ：wate...@gmail.com

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

Hi folks,

The voting for hadoop-3.4.0-RC3 has ended, and we have received a total of
8 affirmative votes, with 6 binding and 2 non-binding.

In the order of the voting timeline:

binding (+6)

Masatake Iwasaki
Xiaoqiao He
Mukund Madhav Thakur
Steve Loughran
Takanobu Asanuma
Shilun Fan

non-binding(+2)

Suhail, Ahmar
Dongjoon Hyun

Thank you very much for helping with the review and validation!

We have received 6 binding PMC affirmative votes. hadoop-3.4.0 will be
officially released in the next 1-2 days.

Best Regards,
Shilun Fan.

On Fri, Mar 15, 2024 at 8:32 AM slfan1989  wrote:

> +1 (binding).
>
> * Verified the signature and checksum of all tarballs.
> * Built from the source tarball on macOS 10.14 and OpenJDK 8.
> * Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
> * Check the Yarn federation page and basic functionalities.
> * Read the basic help documentation.
>
> Best Regards,
> Shilun Fan.
>
> On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma 
> wrote:
>
>> +1 (binding).
>>
>> Thanks for the great work, Shilun Fan.
>>
>> * Verified signatures and checksums
>> * Successfully built from source with native code
>> * Deployed a distributed cluster (on K8s)
>> * Successfully ran some Erasure Coding operations with ISA-L codec
>> * Successfully ran some HDFS RBF operations
>>
>> Regards,
>> - Takanobu Asanuma
>>
>> 2024年3月14日(木) 15:19 Xiaoqiao He :
>>
>>> Thanks Ayush for highlighting this information. Absolutely true, we
>>> should
>>> count RM's vote when explicit +1 here.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:
>>>
>>> > >  Counter should be with yourself vote, where the current summary
>>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>>> >
>>> > Just on the process: The release manager needs to "explicitly" vote
>>> like
>>> > any other before counting their own vote, there has been a lot of
>>> > discussions around that at multiple places & the official apache doc
>>> has
>>> > been updated as well [1], the last paragraph reads:
>>> >
>>> > "Note that there is no implicit +1 from the release manager, or from
>>> > anyone in any ASF vote. Only explicit votes are valid. The release
>>> manager
>>> > is encouraged to vote on releases, like any reviewer would do."
>>> >
>>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>>> >
>>> > -Ayush
>>> >
>>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>>> >
>>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>>> 
>>> > wrote:
>>> >
>>> >> followup: overnight work happy too.
>>> >>
>>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>>> >> complains that libcrypto is missing
>>> >>
>>> >> > bin/hadoop checknative
>>> >>
>>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>>> >> initialized native-bzip2 library system-native
>>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>>> >> initialized native-zlib library
>>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>>> support
>>> >> is not available in your platform... using builtin-java codec where
>>> >> applicable
>>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>>> built
>>> >> without PMDK support.
>>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>>> OpenSSL
>>> >> Cipher.
>>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so
>>> (libcrypto.so:
>>> >> cannot open shared object file: No such file or directory)!
>>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>>> Method)
>>> >> at
>>> >> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
>>> >> at
>>> >>
>>> >>
>>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>>> >> Native library checking:
>>> >> hadoop:  true
>>> >>
>>> >>
>>>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

+1 (binding).

* Verified the signature and checksum of all tarballs.
* Built from the source tarball on macOS 10.14 and OpenJDK 8.
* Execute basic HDFS/YARN commands and run the WordCount MapReduce job.
* Check the Yarn federation page and basic functionalities.
* Read the basic help documentation.

Best Regards,
Shilun Fan.

On Thu, Mar 14, 2024 at 4:40 PM Takanobu Asanuma 
wrote:

> +1 (binding).
>
> Thanks for the great work, Shilun Fan.
>
> * Verified signatures and checksums
> * Successfully built from source with native code
> * Deployed a distributed cluster (on K8s)
> * Successfully ran some Erasure Coding operations with ISA-L codec
> * Successfully ran some HDFS RBF operations
>
> Regards,
> - Takanobu Asanuma
>
> 2024年3月14日(木) 15:19 Xiaoqiao He :
>
>> Thanks Ayush for highlighting this information. Absolutely true, we should
>> count RM's vote when explicit +1 here.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:
>>
>> > >  Counter should be with yourself vote, where the current summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> >
>> > Just on the process: The release manager needs to "explicitly" vote like
>> > any other before counting their own vote, there has been a lot of
>> > discussions around that at multiple places & the official apache doc has
>> > been updated as well [1], the last paragraph reads:
>> >
>> > "Note that there is no implicit +1 from the release manager, or from
>> > anyone in any ASF vote. Only explicit votes are valid. The release
>> manager
>> > is encouraged to vote on releases, like any reviewer would do."
>> >
>> > So, do put an explicit +1, before you count yourself. Good Luck!!!
>> >
>> > -Ayush
>> >
>> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>> >
>> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran
>> 
>> > wrote:
>> >
>> >> followup: overnight work happy too.
>> >>
>> >> one interesting pain point is that on a raspberry pi 64 os checknative
>> >> complains that libcrypto is missing
>> >>
>> >> > bin/hadoop checknative
>> >>
>> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> >> initialized native-bzip2 library system-native
>> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> >> initialized native-zlib library
>> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
>> support
>> >> is not available in your platform... using builtin-java codec where
>> >> applicable
>> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
>> built
>> >> without PMDK support.
>> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
>> OpenSSL
>> >> Cipher.
>> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> >> cannot open shared object file: No such file or directory)!
>> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native
>> Method)
>> >> at
>> >> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
>> >> at
>> >>
>> >>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> >> Native library checking:
>> >> hadoop:  true
>> >>
>> >>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> >> zlib:true /lib/aarch64-linux-gnu/libz.so.1
>> >> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
>> >> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
>> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
>> shared
>> >> object file: No such file or directory)!
>> >> ISA-L:   false libhadoop was built without ISA-L support
>> >> PMDK:false The native code was built without PMDK support.
>> >>
>> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> >> /usr/lib/aarch64-linux-gnu/l
>> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> &

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Takanobu Asanuma]

+1 (binding).

Thanks for the great work, Shilun Fan.

* Verified signatures and checksums
* Successfully built from source with native code
* Deployed a distributed cluster (on K8s)
* Successfully ran some Erasure Coding operations with ISA-L codec
* Successfully ran some HDFS RBF operations

Regards,
- Takanobu Asanuma

2024年3月14日(木) 15:19 Xiaoqiao He :

> Thanks Ayush for highlighting this information. Absolutely true, we should
> count RM's vote when explicit +1 here.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:
>
> > >  Counter should be with yourself vote, where the current summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> >
> > Just on the process: The release manager needs to "explicitly" vote like
> > any other before counting their own vote, there has been a lot of
> > discussions around that at multiple places & the official apache doc has
> > been updated as well [1], the last paragraph reads:
> >
> > "Note that there is no implicit +1 from the release manager, or from
> > anyone in any ASF vote. Only explicit votes are valid. The release
> manager
> > is encouraged to vote on releases, like any reviewer would do."
> >
> > So, do put an explicit +1, before you count yourself. Good Luck!!!
> >
> > -Ayush
> >
> > [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
> >
> > On Tue, 12 Mar 2024 at 17:27, Steve Loughran  >
> > wrote:
> >
> >> followup: overnight work happy too.
> >>
> >> one interesting pain point is that on a raspberry pi 64 os checknative
> >> complains that libcrypto is missing
> >>
> >> > bin/hadoop checknative
> >>
> >> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> >> initialized native-bzip2 library system-native
> >> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> >> initialized native-zlib library
> >> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L
> support
> >> is not available in your platform... using builtin-java codec where
> >> applicable
> >> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was
> built
> >> without PMDK support.
> >> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load
> OpenSSL
> >> Cipher.
> >> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> >> cannot open shared object file: No such file or directory)!
> >> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> >> at
> >> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
> >> at
> >>
> >>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> >> Native library checking:
> >> hadoop:  true
> >>
> >>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> >> zlib:true /lib/aarch64-linux-gnu/libz.so.1
> >> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
> >> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
> >> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open
> shared
> >> object file: No such file or directory)!
> >> ISA-L:   false libhadoop was built without ISA-L support
> >> PMDK:false The native code was built without PMDK support.
> >>
> >> which happens because its not in /lib/aarch64-linux-gnu but instead in
> >> /usr/lib/aarch64-linux-gnu/l
> >> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> >> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> >> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> >> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
> >>
> >> Anyone got any insights on how I should set up this (debian-based) OS
> >> here?
> >> I know it's only a small box but with arm64 VMs becoming available in
> >> cloud
> >> infras, it'd be good to know if they are similar.
> >>
> >> Note: checknative itself is happy; but checknative -a will fail because
> of
> >> this -though it's an OS setup issue, nothing related to the hadoop
> >> binaries.
> >>
> >> steve
> >>
> >> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He 
> wrote:
> >>
> >> > Hi Shilun, Counter should be with yourself vote, where the current
> >&g

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

Thanks Ayush for highlighting this information. Absolutely true, we should
count RM's vote when explicit +1 here.

Best Regards,
- He Xiaoqiao

On Thu, Mar 14, 2024 at 3:55 AM Ayush Saxena  wrote:

> >  Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>
> Just on the process: The release manager needs to "explicitly" vote like
> any other before counting their own vote, there has been a lot of
> discussions around that at multiple places & the official apache doc has
> been updated as well [1], the last paragraph reads:
>
> "Note that there is no implicit +1 from the release manager, or from
> anyone in any ASF vote. Only explicit votes are valid. The release manager
> is encouraged to vote on releases, like any reviewer would do."
>
> So, do put an explicit +1, before you count yourself. Good Luck!!!
>
> -Ayush
>
> [1] https://www.apache.org/foundation/voting.html#ReleaseVotes
>
> On Tue, 12 Mar 2024 at 17:27, Steve Loughran 
> wrote:
>
>> followup: overnight work happy too.
>>
>> one interesting pain point is that on a raspberry pi 64 os checknative
>> complains that libcrypto is missing
>>
>> > bin/hadoop checknative
>>
>> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
>> initialized native-bzip2 library system-native
>> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
>> initialized native-zlib library
>> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
>> is not available in your platform... using builtin-java codec where
>> applicable
>> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
>> without PMDK support.
>> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
>> Cipher.
>> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
>> cannot open shared object file: No such file or directory)!
>> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
>> at
>> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
>> at
>>
>> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
>> Native library checking:
>> hadoop:  true
>>
>> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
>> zlib:true /lib/aarch64-linux-gnu/libz.so.1
>> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
>> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
>> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
>> object file: No such file or directory)!
>> ISA-L:   false libhadoop was built without ISA-L support
>> PMDK:false The native code was built without PMDK support.
>>
>> which happens because its not in /lib/aarch64-linux-gnu but instead in
>> /usr/lib/aarch64-linux-gnu/l
>> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
>> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
>> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
>> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>>
>> Anyone got any insights on how I should set up this (debian-based) OS
>> here?
>> I know it's only a small box but with arm64 VMs becoming available in
>> cloud
>> infras, it'd be good to know if they are similar.
>>
>> Note: checknative itself is happy; but checknative -a will fail because of
>> this -though it's an OS setup issue, nothing related to the hadoop
>> binaries.
>>
>> steve
>>
>> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:
>>
>> > Hi Shilun, Counter should be with yourself vote, where the current
>> summary
>> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
>> > Thanks again.
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
>> >
>> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
>> > and
>> > > 1 vote non-binding.
>> > >
>> > > Thank you very much for voting and verifying!
>> > >
>> > > This voting will continue until March 15th, this Friday.
>> > >
>> > > Best Regards,
>> > > Shilun Fan.
>> > >
>> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>> > > > > >
>> > > wrote:
>> > >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Ayush Saxena]

>  Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.

Just on the process: The release manager needs to "explicitly" vote like
any other before counting their own vote, there has been a lot of
discussions around that at multiple places & the official apache doc has
been updated as well [1], the last paragraph reads:

"Note that there is no implicit +1 from the release manager, or from anyone
in any ASF vote. Only explicit votes are valid. The release manager is
encouraged to vote on releases, like any reviewer would do."

So, do put an explicit +1, before you count yourself. Good Luck!!!

-Ayush

[1] https://www.apache.org/foundation/voting.html#ReleaseVotes

On Tue, 12 Mar 2024 at 17:27, Steve Loughran 
wrote:

> followup: overnight work happy too.
>
> one interesting pain point is that on a raspberry pi 64 os checknative
> complains that libcrypto is missing
>
> > bin/hadoop checknative
>
> 2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
> initialized native-bzip2 library system-native
> 2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
> initialized native-zlib library
> 2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
> is not available in your platform... using builtin-java codec where
> applicable
> 2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
> without PMDK support.
> 2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
> Cipher.
> java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
> cannot open shared object file: No such file or directory)!
> at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
> at
> org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
> at
>
> org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
> Native library checking:
> hadoop:  true
>
> /home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
> zlib:true /lib/aarch64-linux-gnu/libz.so.1
> zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
> bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
> openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
> object file: No such file or directory)!
> ISA-L:   false libhadoop was built without ISA-L support
> PMDK:false The native code was built without PMDK support.
>
> which happens because its not in /lib/aarch64-linux-gnu but instead in
> /usr/lib/aarch64-linux-gnu/l
> ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
> -rw-r--r-- 1 root root 2739952 Sep 19 13:09
> /usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
> -rw-r--r-- 1 root root 4466856 Oct 27 13:40
> /usr/lib/aarch64-linux-gnu/libcrypto.so.3
>
> Anyone got any insights on how I should set up this (debian-based) OS here?
> I know it's only a small box but with arm64 VMs becoming available in cloud
> infras, it'd be good to know if they are similar.
>
> Note: checknative itself is happy; but checknative -a will fail because of
> this -though it's an OS setup issue, nothing related to the hadoop
> binaries.
>
> steve
>
> On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:
>
> > Hi Shilun, Counter should be with yourself vote, where the current
> summary
> > is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> > Thanks again.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
> >
> > > As of now, we have collected 5 affirmative votes, with 4 votes binding
> > and
> > > 1 vote non-binding.
> > >
> > > Thank you very much for voting and verifying!
> > >
> > > This voting will continue until March 15th, this Friday.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
> >  > > >
> > > wrote:
> > >
> > > > +1 binding
> > > >
> > > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have
> seen
> > it
> > > > earlier. been testing it this afternoon:
> > > >
> > > > pulled the latest version of
> > > > https://github.com/apache/hadoop-release-support
> > > > (note, this module is commit-then-review; whoever is working
> > > on/validating
> > > > a release can commit as they go along. This is not production
> code...)
> > > >
> > > > * went through the "validating a release&q

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Dongjoon Hyun]

+1

I tested with Apache ORC repo.

Thank you for the new release.

Dongjoon.

On 2024/03/11 20:28:22 Steve Loughran wrote:
> +1 binding
> 
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
> 
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
> 
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
> 
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
> 
> The core changes I needed in are at the dependency level and I've
> verified they are good.
> 
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
> 
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
> 
> 
> 
> 
> 
> On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
> 
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs, especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
> 

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Steve Loughran]

followup: overnight work happy too.

one interesting pain point is that on a raspberry pi 64 os checknative
complains that libcrypto is missing

> bin/hadoop checknative

2024-03-12 11:50:24,359 INFO bzip2.Bzip2Factory: Successfully loaded &
initialized native-bzip2 library system-native
2024-03-12 11:50:24,363 INFO zlib.ZlibFactory: Successfully loaded &
initialized native-zlib library
2024-03-12 11:50:24,370 WARN erasurecode.ErasureCodeNative: ISA-L support
is not available in your platform... using builtin-java codec where
applicable
2024-03-12 11:50:24,429 INFO nativeio.NativeIO: The native code was built
without PMDK support.
2024-03-12 11:50:24,431 WARN crypto.OpensslCipher: Failed to load OpenSSL
Cipher.
java.lang.UnsatisfiedLinkError: Cannot load libcrypto.so (libcrypto.so:
cannot open shared object file: No such file or directory)!
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)
at
org.apache.hadoop.crypto.OpensslCipher.(OpensslCipher.java:90)
at
org.apache.hadoop.util.NativeLibraryChecker.main(NativeLibraryChecker.java:111)
Native library checking:
hadoop:  true
/home/stevel/Projects/hadoop-release-support/target/arm-untar/hadoop-3.4.0/lib/native/libhadoop.so.1.0.0
zlib:true /lib/aarch64-linux-gnu/libz.so.1
zstd  :  true /lib/aarch64-linux-gnu/libzstd.so.1
bzip2:   true /lib/aarch64-linux-gnu/libbz2.so.1
openssl: false Cannot load libcrypto.so (libcrypto.so: cannot open shared
object file: No such file or directory)!
ISA-L:   false libhadoop was built without ISA-L support
PMDK:false The native code was built without PMDK support.

which happens because its not in /lib/aarch64-linux-gnu but instead in
/usr/lib/aarch64-linux-gnu/l
ls -l /usr/lib/aarch64-linux-gnu/libcrypto*
-rw-r--r-- 1 root root 2739952 Sep 19 13:09
/usr/lib/aarch64-linux-gnu/libcrypto.so.1.1
-rw-r--r-- 1 root root 4466856 Oct 27 13:40
/usr/lib/aarch64-linux-gnu/libcrypto.so.3

Anyone got any insights on how I should set up this (debian-based) OS here?
I know it's only a small box but with arm64 VMs becoming available in cloud
infras, it'd be good to know if they are similar.

Note: checknative itself is happy; but checknative -a will fail because of
this -though it's an OS setup issue, nothing related to the hadoop binaries.

steve

On Tue, 12 Mar 2024 at 02:26, Xiaoqiao He  wrote:

> Hi Shilun, Counter should be with yourself vote, where the current summary
> is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
> Thanks again.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:
>
> > As of now, we have collected 5 affirmative votes, with 4 votes binding
> and
> > 1 vote non-binding.
> >
> > Thank you very much for voting and verifying!
> >
> > This voting will continue until March 15th, this Friday.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran
>  > >
> > wrote:
> >
> > > +1 binding
> > >
> > > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen
> it
> > > earlier. been testing it this afternoon:
> > >
> > > pulled the latest version of
> > > https://github.com/apache/hadoop-release-support
> > > (note, this module is commit-then-review; whoever is working
> > on/validating
> > > a release can commit as they go along. This is not production code...)
> > >
> > > * went through the "validating a release" step, validating maven
> > artifacts
> > > * building the same downstream modules which built for me last time
> (avro
> > > too complex; hboss not aws v2 in apache yet)
> > >
> > > spark build is still ongoing, but I'm not going to wait. It is
> building,
> > > which is key.
> > >
> > > The core changes I needed in are at the dependency level and I've
> > > verified they are good.
> > >
> > > Oh, and I've also got my raspberry p5 doing the download of the arm
> > > stuff for its checknative; not expecting problems.
> > >
> > > So: i've got some stuff still ongoing, but the core changes to
> packaging
> > > are in and the rest I'm not worried about -they shouldn't block the
> > release
> > > as I already validated them on RC2
> > >
> > >
> > >
> > >
> > >
> > > On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC3) for
> > Hadoop
> > > > 3.4.0.
> > > >
> > > > What we would like i

Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop

[Ayush Saxena]

Thanx Everyone. I think we have an agreement around dropping the Hbase v1
support.

I have created a ticket: https://issues.apache.org/jira/browse/HADOOP-19107

FYI. The HBase v2 build is broken now. I tried " mvn clean install
-DskipTests -Dhbase.profile=2.0" & it didn't work (It worked couple of
months back IIRC), Some sl4j exclusion needs to be added I believe

Maybe an upgrade of the HBase v2 version might solve it, if not we need to
handle it in our code. We can chase the upgrade of v2 version as Bryan
mentioned either in a different ticket parallely or together with this as
well :-)

In case anyone has objections/suggestions do shout out here or on the
ticket or both

-Ayush

On Mon, 11 Mar 2024 at 19:22, Steve Loughran 
wrote:

>  +1 for cutting hbase 1; it only reduces dependency pain (no more protobuf
> 2.5!)
>
> Created the JIRA on that a few days back
> https://issues.apache.org/jira/browse/YARN-11658
>
> On Tue, 5 Mar 2024 at 12:08, Bryan Beaudreault 
> wrote:
>
> > Hbase v1 is EOL for a while now, so option 2 probably makes sense. While
> > you are at it you should probably update the hbase2 version, because
> 2.2.x
> > is also very old and EOL. 2.5.x is the currently maintained release for
> > hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as
> > well.
> >
> > On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena  wrote:
> >
> > > Hi Folks,
> > > As of now we have two profiles for HBase: one for HBase v1(1.7.1) &
> other
> > > for v2(2.2.4). The versions are specified over here: [1], how to build
> is
> > > mentioned over here: [2]
> > >
> > > As of now we by default run our Jenkins "only" for HBase v1, so we have
> > > seen HBase v2 profile silently breaking a couple of times.
> > >
> > > Considering there are stable versions for HBase v2 as per [3] & HBase
> v2
> > > seems not too new, I have some suggestions, we can consider:
> > >
> > > * Make HBase v2 profile as the default profile & let HBase v1 profile
> > stay
> > > in our code.
> > > * Ditch HBase v1 profile & just lets support HBase v2 profile.
> > > * Let everything stay as is, just add a Jenkins job/ Github action
> which
> > > compiles HBase v2 as well, so we make sure no change breaks it.
> > >
> > > Personally I would go with the second option, the last HBase v1 release
> > > seems to be 2 years back, it might be pulling in some
> > > problematic transitive dependencies & it will open scope for us to
> > support
> > > HBase 3.x when they have a stable release in future.
> > >
> > >
> > > Let me know your thoughts!!!
> > >
> > > -Ayush
> > >
> > >
> > > [1]
> > >
> > >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207
> > >
> > > [2]
> > >
> > >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172
> > >
> > > [3] https://hbase.apache.org/downloads.html
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

Hi Shilun, Counter should be with yourself vote, where the current summary
is 5 +1 binding and 1 +1 non-binding. Let's re-count when deadline.
Thanks again.

Best Regards,
- He Xiaoqiao

On Tue, Mar 12, 2024 at 9:00 AM slfan1989  wrote:

> As of now, we have collected 5 affirmative votes, with 4 votes binding and
> 1 vote non-binding.
>
> Thank you very much for voting and verifying!
>
> This voting will continue until March 15th, this Friday.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran  >
> wrote:
>
> > +1 binding
> >
> > (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> > earlier. been testing it this afternoon:
> >
> > pulled the latest version of
> > https://github.com/apache/hadoop-release-support
> > (note, this module is commit-then-review; whoever is working
> on/validating
> > a release can commit as they go along. This is not production code...)
> >
> > * went through the "validating a release" step, validating maven
> artifacts
> > * building the same downstream modules which built for me last time (avro
> > too complex; hboss not aws v2 in apache yet)
> >
> > spark build is still ongoing, but I'm not going to wait. It is building,
> > which is key.
> >
> > The core changes I needed in are at the dependency level and I've
> > verified they are good.
> >
> > Oh, and I've also got my raspberry p5 doing the download of the arm
> > stuff for its checknative; not expecting problems.
> >
> > So: i've got some stuff still ongoing, but the core changes to packaging
> > are in and the rest I'm not worried about -they shouldn't block the
> release
> > as I already validated them on RC2
> >
> >
> >
> >
> >
> > On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC3) for
> Hadoop
> > > 3.4.0.
> > >
> > > What we would like is for anyone who can to verify the tarballs,
> > especially
> > > anyone who can try the arm64 binaries as we want to include them too.
> > >
> > > The RC is available at:
> > > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> > >
> > > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
> > >
> > > You can find my public key at:
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Change log
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> > >
> > > Release notes
> > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> > >
> > > This is off branch-3.4.0 and is the first big release since 3.3.6.
> > >
> > > Key changes include
> > >
> > > * S3A: Upgrade AWS SDK to V2
> > > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > > * YARN Federation improvements
> > > * YARN Capacity Scheduler improvements
> > > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > > * HDFS EC: Code Enhancements and Bug Fixes
> > > * Transitive CVE fixes
> > >
> > > Differences from Hadoop-3.4.0-RC2
> > >
> > > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > > (80b4bb68159c)
> > > * Use hadoop-release-support[1] for packaging and verification.
> > > * Add protobuf compatibility issue description
> > >
> > > Note, because the arm64 binaries are built separately on a different
> > > platform and JVM, their jar files may not match those of the x86
> > > release -and therefore the maven artifacts. I don't think this is
> > > an issue (the ASF actually releases source tarballs, the binaries are
> > > there for help only, though with the maven repo that's a bit blurred).
> > >
> > > The only way to be consistent would actually untar the x86.tar.gz,
> > > overwrite its binaries with the arm stuff, retar, sign and push out
> > > for the vote. Even automating that would be risky.
> > >
> > > [1] hadoop-release-support:
> > > https://github.com/apache/hadoop-release-support
> > > Thanks to steve for providing hadoop-release-support.
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

As of now, we have collected 5 affirmative votes, with 4 votes binding and
1 vote non-binding.

Thank you very much for voting and verifying!

This voting will continue until March 15th, this Friday.

Best Regards,
Shilun Fan.

On Tue, Mar 12, 2024 at 4:29 AM Steve Loughran 
wrote:

> +1 binding
>
> (sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
> earlier. been testing it this afternoon:
>
> pulled the latest version of
> https://github.com/apache/hadoop-release-support
> (note, this module is commit-then-review; whoever is working on/validating
> a release can commit as they go along. This is not production code...)
>
> * went through the "validating a release" step, validating maven artifacts
> * building the same downstream modules which built for me last time (avro
> too complex; hboss not aws v2 in apache yet)
>
> spark build is still ongoing, but I'm not going to wait. It is building,
> which is key.
>
> The core changes I needed in are at the dependency level and I've
> verified they are good.
>
> Oh, and I've also got my raspberry p5 doing the download of the arm
> stuff for its checknative; not expecting problems.
>
> So: i've got some stuff still ongoing, but the core changes to packaging
> are in and the rest I'm not worried about -they shouldn't block the release
> as I already validated them on RC2
>
>
>
>
>
> On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Steve Loughran]

+1 binding

(sorry, this had ended in the yarn-dev folder, otherwise I'd have seen it
earlier. been testing it this afternoon:

pulled the latest version of
https://github.com/apache/hadoop-release-support
(note, this module is commit-then-review; whoever is working on/validating
a release can commit as they go along. This is not production code...)

* went through the "validating a release" step, validating maven artifacts
* building the same downstream modules which built for me last time (avro
too complex; hboss not aws v2 in apache yet)

spark build is still ongoing, but I'm not going to wait. It is building,
which is key.

The core changes I needed in are at the dependency level and I've
verified they are good.

Oh, and I've also got my raspberry p5 doing the download of the arm
stuff for its checknative; not expecting problems.

So: i've got some stuff still ongoing, but the core changes to packaging
are in and the rest I'm not worried about -they shouldn't block the release
as I already validated them on RC2





On Mon, 4 Mar 2024 at 22:08, slfan1989  wrote:

> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Mukund Madhav Thakur]

+1 (binding)
Thanks for driving the release, Shilun Fan.


   - Verified checksums and signatures.
   - Build the source on macOS and Java 8.
   - Ran AWS and Azure tests ( some known failures, created Jira's
   HADOOP-19106 it is a test side issue only mostly related to test configs,
   also as Ahmar pointed out)
   - Untar the binaries and ran hadoop fs commands on S3 and ABFS buckets.
   - Compiled gcs-connector and ran tests using the 3.4.0 version.


On Mon, Mar 11, 2024 at 4:10 AM Xiaoqiao He  wrote:

> +1 (binding)
>
> [x] Verified signature and checksum.
> [x] LICENSE files exist and NOTICE is included.
> [x] Rat check is ok. mvn clean apache-rat:check
> [x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
> -DskipTests -Pnative -Pdist -Dtar`.
> [x] Setup pseudo cluster with HDFS and YARN.
> [x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
> result.
> [x] Run example mr jobs and check the result - Pi & wordcount.
> [x] Spot-check and run some unit tests.
> [x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager
> etc.
> [x] Skimmed over the contents of site documentation.
> [x] Skimmed over the contents of maven repo.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar  wrote:
>
> > +1 (non-binding)
> >
> >
> > Thanks for driving the release, Shilun.
> >
> >
> > * Verified sha512 checksum and signature was correct for source tarball
> > * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> > * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> > * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> > agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> > * untar the binaries and able to access objects in S3 (for both standard
> > and S3 express) via hadoop fs commands.
> >
> > We have some failures in S3A tests, but these are expected due to one of
> > the public landsat buckets we relied on no longer being available, these
> > tests were fixed in HADOOP-19057
> > .
> >
> > --
> > *From:* slfan1989 
> > *Sent:* Monday, March 4, 2024 10:07:05 PM
> > *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> > priv...@hadoop.apache.org>
> > *Cc:* Steve Loughran; Xiaoqiao He
> > *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
> >
> > CAUTION: This email originated from outside of the organization. Do not
> > click links or open attachments unless you can confirm the sender and
> know
> > the content is safe.
> >
> >
> >
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> 

Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop

[Steve Loughran]

 +1 for cutting hbase 1; it only reduces dependency pain (no more protobuf
2.5!)

Created the JIRA on that a few days back
https://issues.apache.org/jira/browse/YARN-11658

On Tue, 5 Mar 2024 at 12:08, Bryan Beaudreault 
wrote:

> Hbase v1 is EOL for a while now, so option 2 probably makes sense. While
> you are at it you should probably update the hbase2 version, because 2.2.x
> is also very old and EOL. 2.5.x is the currently maintained release for
> hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as
> well.
>
> On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena  wrote:
>
> > Hi Folks,
> > As of now we have two profiles for HBase: one for HBase v1(1.7.1) & other
> > for v2(2.2.4). The versions are specified over here: [1], how to build is
> > mentioned over here: [2]
> >
> > As of now we by default run our Jenkins "only" for HBase v1, so we have
> > seen HBase v2 profile silently breaking a couple of times.
> >
> > Considering there are stable versions for HBase v2 as per [3] & HBase v2
> > seems not too new, I have some suggestions, we can consider:
> >
> > * Make HBase v2 profile as the default profile & let HBase v1 profile
> stay
> > in our code.
> > * Ditch HBase v1 profile & just lets support HBase v2 profile.
> > * Let everything stay as is, just add a Jenkins job/ Github action which
> > compiles HBase v2 as well, so we make sure no change breaks it.
> >
> > Personally I would go with the second option, the last HBase v1 release
> > seems to be 2 years back, it might be pulling in some
> > problematic transitive dependencies & it will open scope for us to
> support
> > HBase 3.x when they have a stable release in future.
> >
> >
> > Let me know your thoughts!!!
> >
> > -Ayush
> >
> >
> > [1]
> >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207
> >
> > [2]
> >
> >
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172
> >
> > [3] https://hbase.apache.org/downloads.html
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

+1 (binding)

[x] Verified signature and checksum.
[x] LICENSE files exist and NOTICE is included.
[x] Rat check is ok. mvn clean apache-rat:check
[x] Built the source code on Ubuntu and OpenJDK 11 by `mvn clean package
-DskipTests -Pnative -Pdist -Dtar`.
[x] Setup pseudo cluster with HDFS and YARN.
[x] Run simple FsShell - mkdir/put/get/mv/rm (include EC) and check the
result.
[x] Run example mr jobs and check the result - Pi & wordcount.
[x] Spot-check and run some unit tests.
[x] Skimmed the Web UI of NameNode/DataNode/Resourcemanager/NodeManager etc.
[x] Skimmed over the contents of site documentation.
[x] Skimmed over the contents of maven repo.

Best Regards,
- He Xiaoqiao


On Sat, Mar 9, 2024 at 9:22 PM Suhail, Ahmar  wrote:

> +1 (non-binding)
>
>
> Thanks for driving the release, Shilun.
>
>
> * Verified sha512 checksum and signature was correct for source tarball
> * Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
> * Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
> * Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile
> agains Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
> * untar the binaries and able to access objects in S3 (for both standard
> and S3 express) via hadoop fs commands.
>
> We have some failures in S3A tests, but these are expected due to one of
> the public landsat buckets we relied on no longer being available, these
> tests were fixed in HADOOP-19057
> .
>
> --
> *From:* slfan1989 
> *Sent:* Monday, March 4, 2024 10:07:05 PM
> *To:* Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; <
> priv...@hadoop.apache.org>
> *Cc:* Steve Loughran; Xiaoqiao He
> *Subject:* [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)
>
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and know
> the content is safe.
>
>
>
> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Suhail, Ahmar]

+1 (non-binding)


Thanks for driving the release, Shilun.


* Verified sha512 checksum and signature was correct for source tarball
* Built source code on Amazon-Linux 2 and OpenJDK 8 in Amazon EC2
* Verified S3A (hadoop-tools/hadoop-aws) unit tests passing
* Ran S3A (hadoop-tools/hadoop-aws) integ tests with scale profile agains 
Amazon S3 in eu-west-1, and in us-east-1 for directory buckets.
* untar the binaries and able to access objects in S3 (for both standard and S3 
express) via hadoop fs commands.

We have some failures in S3A tests, but these are expected due to one of the 
public landsat buckets we relied on no longer being available, these tests were 
fixed in HADOOP-19057.



From: slfan1989 
Sent: Monday, March 4, 2024 10:07:05 PM
To: Hadoop Common; Hdfs-dev; yarn-dev; mapreduce-dev; 

Cc: Steve Loughran; Xiaoqiao He
Subject: [EXTERNAL] [VOTE] Release Apache Hadoop 3.4.0 (RC3)

CAUTION: This email originated from outside of the organization. Do not click 
links or open attachments unless you can confirm the sender and know the 
content is safe.



Hi folks,

Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/

The git tag is release-3.4.0-RC3, commit bd8b77f398f

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md

This is off branch-3.4.0 and is the first big release since 3.3.6.

Key changes include

* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes

Differences from Hadoop-3.4.0-RC2

* From branch-3.4 to branch-3.4.0 backport 2 Prs
* HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
* HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.

Best Regards,
Shilun Fan.

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Masatake Iwasaki]

+1 (binding)

Thanks for the great work, Shilun Fan.

+ verified checksums and signatures of src and site tarballs.
+ built from source tarball with native profile enabled on Rocky Linux 8
  and Ubuntu 22.04 (x86_64).
+ deployed pseudo cluster with and without kerberos enabled.
  + ran some example jobs on encryption zone.
  + ran some simple query against httpfs.
+ built site documentation and skimmed the contents.
+ skimmed the contents of pre-built site documentation.

- downstream like HBase can not be built against 3.4.0 without addressing
  changes of transitive dependencies.
- OpenSSL feature of libhadoop.so does not work on the platform without
  SM4 support (e.g. Red Hat) while it falls back to Java implementation.
  I want HADOOP-17609 be in the next release (or OpenSSL 3 support considering 
this)::

$ bin/hdfs dfs -put README.txt /zone1/
2024-03-08 15:57:42,685 WARN crypto.OpensslCipher: Failed to load OpenSSL 
Cipher.
java.lang.UnsatisfiedLinkError: Cannot find AES-CTR/SM4-CTR support, is 
your version of Openssl new enough?
at org.apache.hadoop.crypto.OpensslCipher.initIDs(Native Method)

Masatake Iwasaki

On 2024/03/05 7:07, slfan1989 wrote:

Hi folks,

Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
3.4.0.

What we would like is for anyone who can to verify the tarballs, especially
anyone who can try the arm64 binaries as we want to include them too.

The RC is available at:
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/

The git tag is release-3.4.0-RC3, commit bd8b77f398f

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1408

You can find my public key at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Change log
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md

Release notes
https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md

This is off branch-3.4.0 and is the first big release since 3.3.6.

Key changes include

* S3A: Upgrade AWS SDK to V2
* HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
* YARN Federation improvements
* YARN Capacity Scheduler improvements
* HDFS RBF: Code Enhancements, New Features, and Bug Fixes
* HDFS EC: Code Enhancements and Bug Fixes
* Transitive CVE fixes

Differences from Hadoop-3.4.0-RC2

* From branch-3.4 to branch-3.4.0 backport 2 Prs
 * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
 * HADOOP-19084: Pruning hadoop-common transitive dependencies.
(80b4bb68159c)
* Use hadoop-release-support[1] for packaging and verification.
* Add protobuf compatibility issue description

Note, because the arm64 binaries are built separately on a different
platform and JVM, their jar files may not match those of the x86
release -and therefore the maven artifacts. I don't think this is
an issue (the ASF actually releases source tarballs, the binaries are
there for help only, though with the maven repo that's a bit blurred).

The only way to be consistent would actually untar the x86.tar.gz,
overwrite its binaries with the arm stuff, retar, sign and push out
for the vote. Even automating that would be risky.

[1] hadoop-release-support: https://github.com/apache/hadoop-release-support
Thanks to steve for providing hadoop-release-support.

Best Regards,
Shilun Fan.



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop

[Masatake Iwasaki]

+1 on supporting only HBase v2.

Masatake Iwasaki

On 2024/03/06 13:18, slfan1989 wrote:

+1 Option2

I also agree with the idea of upgrading hbase 2.2 to 2.5.

Shilun Fan.


Hbase v1 is EOL for a while now, so option 2 probably makes sense. While
you are at it you should probably update the hbase2 version, because 2.2.x
is also very old and EOL. 2.5.x is the currently maintained release for
hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as
well.


On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena wrote:


Hi Folks,
As of now we have two profiles for HBase: one for HBase v1(1.7.1) & other
for v2(2.2.4). The versions are specified over here: [1], how to build is
mentioned over here: [2]

As of now we by default run our Jenkins "only" for HBase v1, so we have
seen HBase v2 profile silently breaking a couple of times.

Considering there are stable versions for HBase v2 as per [3] & HBase v2
seems not too new, I have some suggestions, we can consider:

* Make HBase v2 profile as the default profile & let HBase v1 profile stay
in our code.
* Ditch HBase v1 profile & just lets support HBase v2 profile.
* Let everything stay as is, just add a Jenkins job/ Github action which
compiles HBase v2 as well, so we make sure no change breaks it.

Personally I would go with the second option, the last HBase v1 release
seems to be 2 years back, it might be pulling in some
problematic transitive dependencies & it will open scope for us to support
HBase 3.x when they have a stable release in future.


Let me know your thoughts!!!

-Ayush


[1]



https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207


[2]



https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172


[3] https://hbase.apache.org/downloads.html





-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Ayush Saxena]

Just gave a quick try, Hive master didn't compile. I gave up post that...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
(default-compile) on project hive-common: Compilation failure
[ERROR]
/Users/ayushsaxena/code/hive/common/src/java/org/apache/hadoop/hive/common/JvmMetrics.java:[23,37]
package org.apache.hadoop.log.metrics does not exist

I think it due to HADOOP-17524
, it will be quite a
effort to upgrade from 3.3.x to 3.4.x, I don't think we would chase that
upgrade in hive anytime soon

I will abstain

-Ayush

On Thu, 7 Mar 2024 at 08:33, Xiaoqiao He  wrote:

> cc @Chao Sun  @zhang...@apache.org
>  @Ayush Saxena 
> Would you mind helping to check the upstream systems(Spark, HBase, Hive)
> dependency with
> the new hadoop release version if necessary? Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He  wrote:
>
>> Shilun, Thanks for your great work. I am checking but not finished yet.
>> Will confirm once done.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>> On Thu, Mar 7, 2024 at 7:23 AM slfan1989  wrote:
>>
>>> @Xiaoqiao He  @Ayush Saxena 
>>> @Steve
>>> Loughran  @Mukund Madhav Thakur
>>>  @Takanobu
>>> Asanuma  @Masatake Iwasaki 
>>>
>>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:
>>>
>>> > Hi folks,
>>> >
>>> > Xiaoqiao He and I have put together a release candidate (RC3) for
>>> Hadoop
>>> > 3.4.0.
>>> >
>>> > What we would like is for anyone who can to verify the tarballs,
>>> especially
>>> > anyone who can try the arm64 binaries as we want to include them too.
>>> >
>>> > The RC is available at:
>>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>>> >
>>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>>> >
>>> > The maven artifacts are staged at
>>> >
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>>> >
>>> > You can find my public key at:
>>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >
>>> > Change log
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>>> >
>>> > Release notes
>>> >
>>> >
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>>> >
>>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>>> >
>>> > Key changes include
>>> >
>>> > * S3A: Upgrade AWS SDK to V2
>>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>>> > * YARN Federation improvements
>>> > * YARN Capacity Scheduler improvements
>>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>>> > * HDFS EC: Code Enhancements and Bug Fixes
>>> > * Transitive CVE fixes
>>> >
>>> > Differences from Hadoop-3.4.0-RC2
>>> >
>>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>>> > (80b4bb68159c)
>>> > * Use hadoop-release-support[1] for packaging and verification.
>>> > * Add protobuf compatibility issue description
>>> >
>>> > Note, because the arm64 binaries are built separately on a different
>>> > platform and JVM, their jar files may not match those of the x86
>>> > release -and therefore the maven artifacts. I don't think this is
>>> > an issue (the ASF actually releases source tarballs, the binaries are
>>> > there for help only, though with the maven repo that's a bit blurred).
>>> >
>>> > The only way to be consistent would actually untar the x86.tar.gz,
>>> > overwrite its binaries with the arm stuff, retar, sign and push out
>>> > for the vote. Even automating that would be risky.
>>> >
>>> > [1] hadoop-release-support:
>>> > https://github.com/apache/hadoop-release-support
>>> > Thanks to steve for providing hadoop-release-support.
>>> >
>>> > Best Regards,
>>> > Shilun Fan.
>>> >
>>> >
>>>
>>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

cc @Chao Sun  @zhang...@apache.org 
 @Ayush Saxena 
Would you mind helping to check the upstream systems(Spark, HBase, Hive)
dependency with
the new hadoop release version if necessary? Thanks.

Best Regards,
- He Xiaoqiao


On Thu, Mar 7, 2024 at 10:23 AM Xiaoqiao He  wrote:

> Shilun, Thanks for your great work. I am checking but not finished yet.
> Will confirm once done.
>
> Best Regards,
> - He Xiaoqiao
>
> On Thu, Mar 7, 2024 at 7:23 AM slfan1989  wrote:
>
>> @Xiaoqiao He  @Ayush Saxena 
>> @Steve
>> Loughran  @Mukund Madhav Thakur
>>  @Takanobu
>> Asanuma  @Masatake Iwasaki 
>>
>> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:
>>
>> > Hi folks,
>> >
>> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
>> > 3.4.0.
>> >
>> > What we would like is for anyone who can to verify the tarballs,
>> especially
>> > anyone who can try the arm64 binaries as we want to include them too.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>> >
>> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
>> >
>> > The maven artifacts are staged at
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>> >
>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> >
>> > Key changes include
>> >
>> > * S3A: Upgrade AWS SDK to V2
>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > * YARN Federation improvements
>> > * YARN Capacity Scheduler improvements
>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > * HDFS EC: Code Enhancements and Bug Fixes
>> > * Transitive CVE fixes
>> >
>> > Differences from Hadoop-3.4.0-RC2
>> >
>> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
>> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
>> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
>> > (80b4bb68159c)
>> > * Use hadoop-release-support[1] for packaging and verification.
>> > * Add protobuf compatibility issue description
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > [1] hadoop-release-support:
>> > https://github.com/apache/hadoop-release-support
>> > Thanks to steve for providing hadoop-release-support.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[Xiaoqiao He]

Shilun, Thanks for your great work. I am checking but not finished yet.
Will confirm once done.

Best Regards,
- He Xiaoqiao

On Thu, Mar 7, 2024 at 7:23 AM slfan1989  wrote:

> @Xiaoqiao He  @Ayush Saxena 
> @Steve
> Loughran  @Mukund Madhav Thakur
>  @Takanobu
> Asanuma  @Masatake Iwasaki 
>
> Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
> >
> > The git tag is release-3.4.0-RC3, commit bd8b77f398f
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1408
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from Hadoop-3.4.0-RC2
> >
> > * From branch-3.4 to branch-3.4.0 backport 2 Prs
> > * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> > * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> > (80b4bb68159c)
> > * Use hadoop-release-support[1] for packaging and verification.
> > * Add protobuf compatibility issue description
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] hadoop-release-support:
> > https://github.com/apache/hadoop-release-support
> > Thanks to steve for providing hadoop-release-support.
> >
> > Best Regards,
> > Shilun Fan.
> >
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC3)

[slfan1989]

@Xiaoqiao He  @Ayush Saxena  @Steve
Loughran  @Mukund Madhav Thakur
 @Takanobu
Asanuma  @Masatake Iwasaki 

Can you help review and vote for hadoop-3.4.0-RC3? Thank you very much!

Best Regards,
Shilun Fan.

On Tue, Mar 5, 2024 at 6:07 AM slfan1989  wrote:

> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC3) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/
>
> The git tag is release-3.4.0-RC3, commit bd8b77f398f
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1408
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC3/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from Hadoop-3.4.0-RC2
>
> * From branch-3.4 to branch-3.4.0 backport 2 Prs
> * HADOOP-18088: Replacing log4j 1.x with reload4j. (ad8b6541117b)
> * HADOOP-19084: Pruning hadoop-common transitive dependencies.
> (80b4bb68159c)
> * Use hadoop-release-support[1] for packaging and verification.
> * Add protobuf compatibility issue description
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] hadoop-release-support:
> https://github.com/apache/hadoop-release-support
> Thanks to steve for providing hadoop-release-support.
>
> Best Regards,
> Shilun Fan.
>
>

Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop

[slfan1989]

+1 Option2

I also agree with the idea of upgrading hbase 2.2 to 2.5.

Shilun Fan.

> Hbase v1 is EOL for a while now, so option 2 probably makes sense. While
> you are at it you should probably update the hbase2 version, because 2.2.x
> is also very old and EOL. 2.5.x is the currently maintained release for
> hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as
> well.

On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena wrote:

> Hi Folks,
> As of now we have two profiles for HBase: one for HBase v1(1.7.1) & other
> for v2(2.2.4). The versions are specified over here: [1], how to build is
> mentioned over here: [2]
>
> As of now we by default run our Jenkins "only" for HBase v1, so we have
> seen HBase v2 profile silently breaking a couple of times.
>
> Considering there are stable versions for HBase v2 as per [3] & HBase v2
> seems not too new, I have some suggestions, we can consider:
>
> * Make HBase v2 profile as the default profile & let HBase v1 profile stay
> in our code.
> * Ditch HBase v1 profile & just lets support HBase v2 profile.
> * Let everything stay as is, just add a Jenkins job/ Github action which
> compiles HBase v2 as well, so we make sure no change breaks it.
>
> Personally I would go with the second option, the last HBase v1 release
> seems to be 2 years back, it might be pulling in some
> problematic transitive dependencies & it will open scope for us to support
> HBase 3.x when they have a stable release in future.
>
>
> Let me know your thoughts!!!
>
> -Ayush
>
>
> [1]
>
>
https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207
>
> [2]
>
>
https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172
>
> [3] https://hbase.apache.org/downloads.html
>

Re: [DISCUSS] Support/Fate of HBase v1 in Hadoop

[Bryan Beaudreault]

Hbase v1 is EOL for a while now, so option 2 probably makes sense. While
you are at it you should probably update the hbase2 version, because 2.2.x
is also very old and EOL. 2.5.x is the currently maintained release for
hbase2, with 2.5.7 being the latest. We’re soon going to release 2.6.0 as
well.

On Tue, Mar 5, 2024 at 6:56 AM Ayush Saxena  wrote:

> Hi Folks,
> As of now we have two profiles for HBase: one for HBase v1(1.7.1) & other
> for v2(2.2.4). The versions are specified over here: [1], how to build is
> mentioned over here: [2]
>
> As of now we by default run our Jenkins "only" for HBase v1, so we have
> seen HBase v2 profile silently breaking a couple of times.
>
> Considering there are stable versions for HBase v2 as per [3] & HBase v2
> seems not too new, I have some suggestions, we can consider:
>
> * Make HBase v2 profile as the default profile & let HBase v1 profile stay
> in our code.
> * Ditch HBase v1 profile & just lets support HBase v2 profile.
> * Let everything stay as is, just add a Jenkins job/ Github action which
> compiles HBase v2 as well, so we make sure no change breaks it.
>
> Personally I would go with the second option, the last HBase v1 release
> seems to be 2 years back, it might be pulling in some
> problematic transitive dependencies & it will open scope for us to support
> HBase 3.x when they have a stable release in future.
>
>
> Let me know your thoughts!!!
>
> -Ayush
>
>
> [1]
>
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/hadoop-project/pom.xml#L206-L207
>
> [2]
>
> https://github.com/apache/hadoop/blob/dae871e3e0783e1fe6ea09131c3f4650abfa8a1d/BUILDING.txt#L168-L172
>
> [3] https://hbase.apache.org/downloads.html
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[slfan1989]

I am preparing hadoop-3.4.0-RC3 as we have already released 3 RC versions
before, and I hope hadoop-3.4.0-RC3 will receive the approval of the
members.

Compared to hadoop-3.4.0-RC2, my plan is to backport 2 PRs from branch-3.4
to branch-3.4.0:

HADOOP-18088: Replacing log4j 1.x with reload4j.
HADOOP-19084: Pruning hadoop-common transitive dependencies.

I will use hadoop-release-support to package the arm version.

I plan to release hadoop-3.4.0-RC3 next Monday.

Best Regards,
Shilun Fan.

On Sat, Feb 24, 2024 at 11:28 AM slfan1989  wrote:

> Thank you very much for Steve's detailed test report and issue description!
>
>  I appreciate your time spent helping with validation. I am currently
> trying to use hadoop-release-support to prepare hadoop-3.4.0-RC3.
>
> After completing the hadoop-3.4.0 version, I will document some of the
> issues encountered in the "how to release" document, so that future members
> can refer to it during the release process.
>
> Once again, thank you to all members involved in the hadoop-3.4.0 release.
>
> Let's hope for a smooth release process.
>
> Best Regards,
> Shilun Fan.
>
> On Sat, Feb 24, 2024 at 2:29 AM Steve Loughran 
> wrote:
>
>> I have been testing this all week, and a -1 until some very minor changes
>> go in.
>>
>>
>>1. build the arm64 binaries with the same jar artifacts as the x86 one
>>2. include ad8b6541117b HADOOP-18088. Replace log4j 1.x with reload4j.
>>3. include 80b4bb68159c HADOOP-19084. Prune hadoop-common transitive
>>dependencies
>>
>>
>> For #1 we have automation there in my client-validator module, which I
>> have
>> moved to be a hadoop-managed project and tried to make more
>> manageable
>> https://github.com/apache/hadoop-release-support
>>
>> This contains an ant project to perform a lot of the documented build
>> stages, including using SCP to copy down an x86 release tarball and make a
>> signed copy of this containing (locally built) arm artifacts.
>>
>> Although that only works with my development environment (macbook m1
>> laptop
>> and remote ec2 server), it should be straightforward to make it more
>> flexible.
>>
>> It also includes and tests a maven project which imports many of the
>> hadoop-* pom files and run some test with it; this caught some problems
>> with exported slf4j and log4j2 artifacts getting into the classpath. That
>> is: hadoop-common pulling in log4j 1.2 and 2.x bindings.
>>
>> HADOOP-19084 fixes this; the build file now includes a target to scan the
>> dependencies and fail if "forbidden" artifacts are found. I have not been
>> able to stop logback ending on the transitive dependency list, but at
>> least
>> there is only one slf4j there.
>>
>> HADOOP-18088. Replace log4j 1.x with reload4j switches over to reload4j
>> while the move to v2 is still something we have to consider a WiP.
>>
>> I have tried doing some other changes to the packaging this week
>> - creating a lean distro without the AWS SDK
>> - trying to get protobuf-2.5 out of yarn-api
>> However, I think it is too late to try applying patches this risky.
>>
>> I Believe we should get the 3.4.0 release out for people to start playing
>> with while we rapidly iterate 3.4.1 release out with
>> - updated dependencies (where possible)
>> - separate "lean" and "full" installations, where "full" includes all the
>> cloud connectors and their dependencies; the default is lean and doesn't.
>> That will cut the default download size in half.
>> - critical issues which people who use the 3.4.0 release raise with us.
>>
>> That is: a packaging and bugs release, with a minimal number of new
>> features.
>>
>> I've created HADOOP-19087
>>  to cover this,
>> I'm willing to get my hands dirty here -Shilun Fan and Xiaoqiao He have
>> put
>> a lot of work on 3.4.0 and probably need other people to take up the work
>> for next release. Who else is willing to participate? (Yes Mukund, I have
>> you in mind too)
>>
>> One thing I would like to visit is: what hadoop-tools modules can we cut?
>> Are rumen and hadoop-streaming being actively used? Or can we consider
>> them
>> implicitly EOL and strip. Just think of the maintenance effort we would
>> save.
>>
>> ---
>>
>> Incidentally, I have tested the arm stuff on my raspberry pi5 which is now
>> running 64 bit linux. I believe it is the first time we have qualified a
>> Hadoop release with the media player under someone's television.
>>
>> On Thu, 15 Feb 2024 at 20:41, Mukund Madhav Thakur 
>> wrote:
>>
>> > Thanks, Shilun for putting this together.
>> >
>> > Tried the below things and everything worked for me.
>> >
>> > validated checksum and gpg signature.
>> > compiled from source.
>> > Ran AWS integration tests.
>> > untar the binaries and able to access objects in S3 via hadoop fs
>> commands.
>> > compiled gcs-connector successfully using the 3.4.0 version.
>> >
>> > qq: what is the difference between RC1 and RC2? apart from some extra
>> > patches.
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[slfan1989]

Thank you very much for Steve's detailed test report and issue description!

 I appreciate your time spent helping with validation. I am currently
trying to use hadoop-release-support to prepare hadoop-3.4.0-RC3.

After completing the hadoop-3.4.0 version, I will document some of the
issues encountered in the "how to release" document, so that future members
can refer to it during the release process.

Once again, thank you to all members involved in the hadoop-3.4.0 release.

Let's hope for a smooth release process.

Best Regards,
Shilun Fan.

On Sat, Feb 24, 2024 at 2:29 AM Steve Loughran 
wrote:

> I have been testing this all week, and a -1 until some very minor changes
> go in.
>
>
>1. build the arm64 binaries with the same jar artifacts as the x86 one
>2. include ad8b6541117b HADOOP-18088. Replace log4j 1.x with reload4j.
>3. include 80b4bb68159c HADOOP-19084. Prune hadoop-common transitive
>dependencies
>
>
> For #1 we have automation there in my client-validator module, which I have
> moved to be a hadoop-managed project and tried to make more
> manageable
> https://github.com/apache/hadoop-release-support
>
> This contains an ant project to perform a lot of the documented build
> stages, including using SCP to copy down an x86 release tarball and make a
> signed copy of this containing (locally built) arm artifacts.
>
> Although that only works with my development environment (macbook m1 laptop
> and remote ec2 server), it should be straightforward to make it more
> flexible.
>
> It also includes and tests a maven project which imports many of the
> hadoop-* pom files and run some test with it; this caught some problems
> with exported slf4j and log4j2 artifacts getting into the classpath. That
> is: hadoop-common pulling in log4j 1.2 and 2.x bindings.
>
> HADOOP-19084 fixes this; the build file now includes a target to scan the
> dependencies and fail if "forbidden" artifacts are found. I have not been
> able to stop logback ending on the transitive dependency list, but at least
> there is only one slf4j there.
>
> HADOOP-18088. Replace log4j 1.x with reload4j switches over to reload4j
> while the move to v2 is still something we have to consider a WiP.
>
> I have tried doing some other changes to the packaging this week
> - creating a lean distro without the AWS SDK
> - trying to get protobuf-2.5 out of yarn-api
> However, I think it is too late to try applying patches this risky.
>
> I Believe we should get the 3.4.0 release out for people to start playing
> with while we rapidly iterate 3.4.1 release out with
> - updated dependencies (where possible)
> - separate "lean" and "full" installations, where "full" includes all the
> cloud connectors and their dependencies; the default is lean and doesn't.
> That will cut the default download size in half.
> - critical issues which people who use the 3.4.0 release raise with us.
>
> That is: a packaging and bugs release, with a minimal number of new
> features.
>
> I've created HADOOP-19087
>  to cover this,
> I'm willing to get my hands dirty here -Shilun Fan and Xiaoqiao He have put
> a lot of work on 3.4.0 and probably need other people to take up the work
> for next release. Who else is willing to participate? (Yes Mukund, I have
> you in mind too)
>
> One thing I would like to visit is: what hadoop-tools modules can we cut?
> Are rumen and hadoop-streaming being actively used? Or can we consider them
> implicitly EOL and strip. Just think of the maintenance effort we would
> save.
>
> ---
>
> Incidentally, I have tested the arm stuff on my raspberry pi5 which is now
> running 64 bit linux. I believe it is the first time we have qualified a
> Hadoop release with the media player under someone's television.
>
> On Thu, 15 Feb 2024 at 20:41, Mukund Madhav Thakur 
> wrote:
>
> > Thanks, Shilun for putting this together.
> >
> > Tried the below things and everything worked for me.
> >
> > validated checksum and gpg signature.
> > compiled from source.
> > Ran AWS integration tests.
> > untar the binaries and able to access objects in S3 via hadoop fs
> commands.
> > compiled gcs-connector successfully using the 3.4.0 version.
> >
> > qq: what is the difference between RC1 and RC2? apart from some extra
> > patches.
> >
> >
> >
> > On Thu, Feb 15, 2024 at 10:58 AM slfan1989  wrote:
> >
> >> Thank you for explaining this part!
> >>
> >> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to
> >> generate
> >> the ARM tar package, which should meet expectations.
> >>
> >> We also look forward to other members helping to verify.
> >>
> >> Best Regards,
> >> Shilun Fan.
> >>
> >> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran 
> >> wrote:
> >>
> >> >
> >> >
> >> > On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:
> >> >
> >> >>
> >> >>
> >> >> Note, because the arm64 binaries are built separately on a different
> >> >> platform and JVM, their jar files may not match those of th

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[Steve Loughran]

I have been testing this all week, and a -1 until some very minor changes
go in.


   1. build the arm64 binaries with the same jar artifacts as the x86 one
   2. include ad8b6541117b HADOOP-18088. Replace log4j 1.x with reload4j.
   3. include 80b4bb68159c HADOOP-19084. Prune hadoop-common transitive
   dependencies


For #1 we have automation there in my client-validator module, which I have
moved to be a hadoop-managed project and tried to make more
manageable
https://github.com/apache/hadoop-release-support

This contains an ant project to perform a lot of the documented build
stages, including using SCP to copy down an x86 release tarball and make a
signed copy of this containing (locally built) arm artifacts.

Although that only works with my development environment (macbook m1 laptop
and remote ec2 server), it should be straightforward to make it more
flexible.

It also includes and tests a maven project which imports many of the
hadoop-* pom files and run some test with it; this caught some problems
with exported slf4j and log4j2 artifacts getting into the classpath. That
is: hadoop-common pulling in log4j 1.2 and 2.x bindings.

HADOOP-19084 fixes this; the build file now includes a target to scan the
dependencies and fail if "forbidden" artifacts are found. I have not been
able to stop logback ending on the transitive dependency list, but at least
there is only one slf4j there.

HADOOP-18088. Replace log4j 1.x with reload4j switches over to reload4j
while the move to v2 is still something we have to consider a WiP.

I have tried doing some other changes to the packaging this week
- creating a lean distro without the AWS SDK
- trying to get protobuf-2.5 out of yarn-api
However, I think it is too late to try applying patches this risky.

I Believe we should get the 3.4.0 release out for people to start playing
with while we rapidly iterate 3.4.1 release out with
- updated dependencies (where possible)
- separate "lean" and "full" installations, where "full" includes all the
cloud connectors and their dependencies; the default is lean and doesn't.
That will cut the default download size in half.
- critical issues which people who use the 3.4.0 release raise with us.

That is: a packaging and bugs release, with a minimal number of new
features.

I've created HADOOP-19087
 to cover this,
I'm willing to get my hands dirty here -Shilun Fan and Xiaoqiao He have put
a lot of work on 3.4.0 and probably need other people to take up the work
for next release. Who else is willing to participate? (Yes Mukund, I have
you in mind too)

One thing I would like to visit is: what hadoop-tools modules can we cut?
Are rumen and hadoop-streaming being actively used? Or can we consider them
implicitly EOL and strip. Just think of the maintenance effort we would
save.

---

Incidentally, I have tested the arm stuff on my raspberry pi5 which is now
running 64 bit linux. I believe it is the first time we have qualified a
Hadoop release with the media player under someone's television.

On Thu, 15 Feb 2024 at 20:41, Mukund Madhav Thakur 
wrote:

> Thanks, Shilun for putting this together.
>
> Tried the below things and everything worked for me.
>
> validated checksum and gpg signature.
> compiled from source.
> Ran AWS integration tests.
> untar the binaries and able to access objects in S3 via hadoop fs commands.
> compiled gcs-connector successfully using the 3.4.0 version.
>
> qq: what is the difference between RC1 and RC2? apart from some extra
> patches.
>
>
>
> On Thu, Feb 15, 2024 at 10:58 AM slfan1989  wrote:
>
>> Thank you for explaining this part!
>>
>> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to
>> generate
>> the ARM tar package, which should meet expectations.
>>
>> We also look forward to other members helping to verify.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran 
>> wrote:
>>
>> >
>> >
>> > On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:
>> >
>> >>
>> >>
>> >> Note, because the arm64 binaries are built separately on a different
>> >> platform and JVM, their jar files may not match those of the x86
>> >> release -and therefore the maven artifacts. I don't think this is
>> >> an issue (the ASF actually releases source tarballs, the binaries are
>> >> there for help only, though with the maven repo that's a bit blurred).
>> >>
>> >> The only way to be consistent would actually untar the x86.tar.gz,
>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>> >> for the vote.
>> >
>> >
>> >
>> > that's exactly what the "arm.release" target in my client-validator
>> does.
>> > builds an arm tar with the x86 binaries but the arm native libs, signs
>> it.
>> >
>> >
>> >
>> >> Even automating that would be risky.
>> >>
>> >>
>> > automating is the *only* way to do it; apache ant has everything needed
>> > for this including the ability to run gpg.
>> >
>> > we did this on the relevan

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[slfan1989]

@Ayush Saxena  @Takanobu Asanuma
 @Xiaoqiao
He  @inigo...@apache.org  @Masatake
Iwasaki  @Akira Ajisaka

Could you please assist with the Hadoop-3.4.0-RC2 vote?

Best Regards,
Shilun Fan.

On Fri, Feb 16, 2024 at 7:27 AM slfan1989  wrote:

> Thank you for helping review hadoop-3.4.0-RC2.
>
> Compared to RC1, we have made two improvements:
> 1. Merged some patches from the branch-3.4 branch to branch-3.4.0.
> 2. Upgrade the version of hadoop-thirdparty to 1.2.0.
>
> Best Regards,
> Shilun Fan.
>
> On Fri, Feb 16, 2024 at 4:41 AM Mukund Madhav Thakur 
> wrote:
>
>> Thanks, Shilun for putting this together.
>>
>> Tried the below things and everything worked for me.
>>
>> validated checksum and gpg signature.
>> compiled from source.
>> Ran AWS integration tests.
>> untar the binaries and able to access objects in S3 via hadoop fs
>> commands.
>> compiled gcs-connector successfully using the 3.4.0 version.
>>
>> qq: what is the difference between RC1 and RC2? apart from some extra
>> patches.
>>
>>
>>
>> On Thu, Feb 15, 2024 at 10:58 AM slfan1989  wrote:
>>
>>> Thank you for explaining this part!
>>>
>>> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to
>>> generate
>>> the ARM tar package, which should meet expectations.
>>>
>>> We also look forward to other members helping to verify.
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran 
>>> wrote:
>>>
>>> >
>>> >
>>> > On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:
>>> >
>>> >>
>>> >>
>>> >> Note, because the arm64 binaries are built separately on a different
>>> >> platform and JVM, their jar files may not match those of the x86
>>> >> release -and therefore the maven artifacts. I don't think this is
>>> >> an issue (the ASF actually releases source tarballs, the binaries are
>>> >> there for help only, though with the maven repo that's a bit blurred).
>>> >>
>>> >> The only way to be consistent would actually untar the x86.tar.gz,
>>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>>> >> for the vote.
>>> >
>>> >
>>> >
>>> > that's exactly what the "arm.release" target in my client-validator
>>> does.
>>> > builds an arm tar with the x86 binaries but the arm native libs, signs
>>> it.
>>> >
>>> >
>>> >
>>> >> Even automating that would be risky.
>>> >>
>>> >>
>>> > automating is the *only* way to do it; apache ant has everything needed
>>> > for this including the ability to run gpg.
>>> >
>>> > we did this on the relevant 3.3.x releases and nobody has yet
>>> complained...
>>> >
>>>
>>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[slfan1989]

Thank you for helping review hadoop-3.4.0-RC2.

Compared to RC1, we have made two improvements:
1. Merged some patches from the branch-3.4 branch to branch-3.4.0.
2. Upgrade the version of hadoop-thirdparty to 1.2.0.

Best Regards,
Shilun Fan.

On Fri, Feb 16, 2024 at 4:41 AM Mukund Madhav Thakur 
wrote:

> Thanks, Shilun for putting this together.
>
> Tried the below things and everything worked for me.
>
> validated checksum and gpg signature.
> compiled from source.
> Ran AWS integration tests.
> untar the binaries and able to access objects in S3 via hadoop fs commands.
> compiled gcs-connector successfully using the 3.4.0 version.
>
> qq: what is the difference between RC1 and RC2? apart from some extra
> patches.
>
>
>
> On Thu, Feb 15, 2024 at 10:58 AM slfan1989  wrote:
>
>> Thank you for explaining this part!
>>
>> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to
>> generate
>> the ARM tar package, which should meet expectations.
>>
>> We also look forward to other members helping to verify.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran 
>> wrote:
>>
>> >
>> >
>> > On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:
>> >
>> >>
>> >>
>> >> Note, because the arm64 binaries are built separately on a different
>> >> platform and JVM, their jar files may not match those of the x86
>> >> release -and therefore the maven artifacts. I don't think this is
>> >> an issue (the ASF actually releases source tarballs, the binaries are
>> >> there for help only, though with the maven repo that's a bit blurred).
>> >>
>> >> The only way to be consistent would actually untar the x86.tar.gz,
>> >> overwrite its binaries with the arm stuff, retar, sign and push out
>> >> for the vote.
>> >
>> >
>> >
>> > that's exactly what the "arm.release" target in my client-validator
>> does.
>> > builds an arm tar with the x86 binaries but the arm native libs, signs
>> it.
>> >
>> >
>> >
>> >> Even automating that would be risky.
>> >>
>> >>
>> > automating is the *only* way to do it; apache ant has everything needed
>> > for this including the ability to run gpg.
>> >
>> > we did this on the relevant 3.3.x releases and nobody has yet
>> complained...
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[Mukund Madhav Thakur]

Thanks, Shilun for putting this together.

Tried the below things and everything worked for me.

validated checksum and gpg signature.
compiled from source.
Ran AWS integration tests.
untar the binaries and able to access objects in S3 via hadoop fs commands.
compiled gcs-connector successfully using the 3.4.0 version.

qq: what is the difference between RC1 and RC2? apart from some extra
patches.



On Thu, Feb 15, 2024 at 10:58 AM slfan1989  wrote:

> Thank you for explaining this part!
>
> hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to generate
> the ARM tar package, which should meet expectations.
>
> We also look forward to other members helping to verify.
>
> Best Regards,
> Shilun Fan.
>
> On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran 
> wrote:
>
> >
> >
> > On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:
> >
> >>
> >>
> >> Note, because the arm64 binaries are built separately on a different
> >> platform and JVM, their jar files may not match those of the x86
> >> release -and therefore the maven artifacts. I don't think this is
> >> an issue (the ASF actually releases source tarballs, the binaries are
> >> there for help only, though with the maven repo that's a bit blurred).
> >>
> >> The only way to be consistent would actually untar the x86.tar.gz,
> >> overwrite its binaries with the arm stuff, retar, sign and push out
> >> for the vote.
> >
> >
> >
> > that's exactly what the "arm.release" target in my client-validator does.
> > builds an arm tar with the x86 binaries but the arm native libs, signs
> it.
> >
> >
> >
> >> Even automating that would be risky.
> >>
> >>
> > automating is the *only* way to do it; apache ant has everything needed
> > for this including the ability to run gpg.
> >
> > we did this on the relevant 3.3.x releases and nobody has yet
> complained...
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[slfan1989]

Thank you for explaining this part!

hadoop-3.4.0-RC2 used the validate-hadoop-client-artifacts tool to generate
the ARM tar package, which should meet expectations.

We also look forward to other members helping to verify.

Best Regards,
Shilun Fan.

On Fri, Feb 16, 2024 at 12:22 AM Steve Loughran  wrote:

>
>
> On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:
>
>>
>>
>> Note, because the arm64 binaries are built separately on a different
>> platform and JVM, their jar files may not match those of the x86
>> release -and therefore the maven artifacts. I don't think this is
>> an issue (the ASF actually releases source tarballs, the binaries are
>> there for help only, though with the maven repo that's a bit blurred).
>>
>> The only way to be consistent would actually untar the x86.tar.gz,
>> overwrite its binaries with the arm stuff, retar, sign and push out
>> for the vote.
>
>
>
> that's exactly what the "arm.release" target in my client-validator does.
> builds an arm tar with the x86 binaries but the arm native libs, signs it.
>
>
>
>> Even automating that would be risky.
>>
>>
> automating is the *only* way to do it; apache ant has everything needed
> for this including the ability to run gpg.
>
> we did this on the relevant 3.3.x releases and nobody has yet complained...
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC2)

[Steve Loughran]

On Mon, 12 Feb 2024 at 15:32, slfan1989  wrote:

>
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote.



that's exactly what the "arm.release" target in my client-validator does.
builds an arm tar with the x86 binaries but the arm native libs, signs it.



> Even automating that would be risky.
>
>
automating is the *only* way to do it; apache ant has everything needed for
this including the ability to run gpg.

we did this on the relevant 3.3.x releases and nobody has yet complained...

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC1)

[slfan1989]

We will backport all the PRs from the branch-3.4 branch to branch-3.4.0,
and use this to release hadoop-3.4.0-RC2. It should include the s3a
checksum and region stuff.

Best Regards,
Shilun Fan.

On Mon, Feb 12, 2024 at 7:10 PM Steve Loughran  wrote:

> so it'll have the s3a checksum and region stuff? that'd be wonderful! I've
> been assuming i'd have to push out a 3.4.1 release for that alone.
>
> On Sat, 10 Feb 2024 at 23:36, slfan1989  wrote:
>
>> We will end the voting for hadoop-3.4.0-RC1 and will open the voting for
>> hadoop-3.4.0-RC2 in 1-2 days.
>>
>> * hadoop-3.4.0-RC2 will contain all commits in branch-3.4, and newly
>> submitted(in branch-3.4) commits will be cherry-picked to branch-3.4.0.
>> * hadoop-3.4.0-RC2 will use hadoop-thirdparty-1.2.0.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Mon, Jan 29, 2024 at 10:33 PM slfan1989  wrote:
>>
>> > Apache Hadoop 3.4.0
>> >
>> > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop
>> > 3.4.0.
>> >
>> > What we would like is for anyone who can to verify the tarballs,
>> especially
>> > anyone who can try the arm64 binaries as we want to include them too.
>> >
>> > The RC is available at:
>> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/
>> >
>> > The git tag is release-3.4.0-RC1, commit 7e2edd8c5d1
>> >
>> > The maven artifacts are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1395/
>> >
>> > You can find my public key at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Change log
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/CHANGELOG.md
>> >
>> > Release notes
>> >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/RELEASENOTES.md
>> >
>> > This is off branch-3.4.0 and is the first big release since 3.3.6.
>> >
>> > Key changes include
>> >
>> > * S3A: Upgrade AWS SDK to V2
>> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
>> > * YARN Federation improvements
>> > * YARN Capacity Scheduler improvements
>> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
>> > * HDFS EC: Code Enhancements and Bug Fixes
>> > * Transitive CVE fixes
>> >
>> > Differences from RC0
>> >
>> > * We've improved Hadoop 3.4.0 Highlight big features and improvements.
>> > * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE
>> modules.
>> > * Use validate-hadoop-client-artifacts[1] for packaging and
>> verification.
>> >
>> > Note, because the arm64 binaries are built separately on a different
>> > platform and JVM, their jar files may not match those of the x86
>> > release -and therefore the maven artifacts. I don't think this is
>> > an issue (the ASF actually releases source tarballs, the binaries are
>> > there for help only, though with the maven repo that's a bit blurred).
>> >
>> > The only way to be consistent would actually untar the x86.tar.gz,
>> > overwrite its binaries with the arm stuff, retar, sign and push out
>> > for the vote. Even automating that would be risky.
>> >
>> > [1] validate-hadoop-client-artifacts:
>> > https://github.com/steveloughran/validate-hadoop-client-artifacts
>> > Thanks to steve for providing validate-hadoop-client-artifacts.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC1)

[Steve Loughran]

so it'll have the s3a checksum and region stuff? that'd be wonderful! I've
been assuming i'd have to push out a 3.4.1 release for that alone.

On Sat, 10 Feb 2024 at 23:36, slfan1989  wrote:

> We will end the voting for hadoop-3.4.0-RC1 and will open the voting for
> hadoop-3.4.0-RC2 in 1-2 days.
>
> * hadoop-3.4.0-RC2 will contain all commits in branch-3.4, and newly
> submitted(in branch-3.4) commits will be cherry-picked to branch-3.4.0.
> * hadoop-3.4.0-RC2 will use hadoop-thirdparty-1.2.0.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Jan 29, 2024 at 10:33 PM slfan1989  wrote:
>
> > Apache Hadoop 3.4.0
> >
> > Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop
> > 3.4.0.
> >
> > What we would like is for anyone who can to verify the tarballs,
> especially
> > anyone who can try the arm64 binaries as we want to include them too.
> >
> > The RC is available at:
> > https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/
> >
> > The git tag is release-3.4.0-RC1, commit 7e2edd8c5d1
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1395/
> >
> > You can find my public key at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Change log
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/CHANGELOG.md
> >
> > Release notes
> >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/RELEASENOTES.md
> >
> > This is off branch-3.4.0 and is the first big release since 3.3.6.
> >
> > Key changes include
> >
> > * S3A: Upgrade AWS SDK to V2
> > * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> > * YARN Federation improvements
> > * YARN Capacity Scheduler improvements
> > * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> > * HDFS EC: Code Enhancements and Bug Fixes
> > * Transitive CVE fixes
> >
> > Differences from RC0
> >
> > * We've improved Hadoop 3.4.0 Highlight big features and improvements.
> > * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE modules.
> > * Use validate-hadoop-client-artifacts[1] for packaging and verification.
> >
> > Note, because the arm64 binaries are built separately on a different
> > platform and JVM, their jar files may not match those of the x86
> > release -and therefore the maven artifacts. I don't think this is
> > an issue (the ASF actually releases source tarballs, the binaries are
> > there for help only, though with the maven repo that's a bit blurred).
> >
> > The only way to be consistent would actually untar the x86.tar.gz,
> > overwrite its binaries with the arm stuff, retar, sign and push out
> > for the vote. Even automating that would be risky.
> >
> > [1] validate-hadoop-client-artifacts:
> > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > Thanks to steve for providing validate-hadoop-client-artifacts.
> >
> > Best Regards,
> > Shilun Fan.
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 (RC1)

[slfan1989]

We will end the voting for hadoop-3.4.0-RC1 and will open the voting for
hadoop-3.4.0-RC2 in 1-2 days.

* hadoop-3.4.0-RC2 will contain all commits in branch-3.4, and newly
submitted(in branch-3.4) commits will be cherry-picked to branch-3.4.0.
* hadoop-3.4.0-RC2 will use hadoop-thirdparty-1.2.0.

Best Regards,
Shilun Fan.

On Mon, Jan 29, 2024 at 10:33 PM slfan1989  wrote:

> Apache Hadoop 3.4.0
>
> Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop
> 3.4.0.
>
> What we would like is for anyone who can to verify the tarballs, especially
> anyone who can try the arm64 binaries as we want to include them too.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/
>
> The git tag is release-3.4.0-RC1, commit 7e2edd8c5d1
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1395/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Change log
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/CHANGELOG.md
>
> Release notes
>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-3.4.0-RC1/RELEASENOTES.md
>
> This is off branch-3.4.0 and is the first big release since 3.3.6.
>
> Key changes include
>
> * S3A: Upgrade AWS SDK to V2
> * HDFS DataNode Split one FsDatasetImpl lock to volume grain locks
> * YARN Federation improvements
> * YARN Capacity Scheduler improvements
> * HDFS RBF: Code Enhancements, New Features, and Bug Fixes
> * HDFS EC: Code Enhancements and Bug Fixes
> * Transitive CVE fixes
>
> Differences from RC0
>
> * We've improved Hadoop 3.4.0 Highlight big features and improvements.
> * Confirmed the JIRA status of Hadoop, HDFS, YARN, and MAPREDUCE modules.
> * Use validate-hadoop-client-artifacts[1] for packaging and verification.
>
> Note, because the arm64 binaries are built separately on a different
> platform and JVM, their jar files may not match those of the x86
> release -and therefore the maven artifacts. I don't think this is
> an issue (the ASF actually releases source tarballs, the binaries are
> there for help only, though with the maven repo that's a bit blurred).
>
> The only way to be consistent would actually untar the x86.tar.gz,
> overwrite its binaries with the arm stuff, retar, sign and push out
> for the vote. Even automating that would be risky.
>
> [1] validate-hadoop-client-artifacts:
> https://github.com/steveloughran/validate-hadoop-client-artifacts
> Thanks to steve for providing validate-hadoop-client-artifacts.
>
> Best Regards,
> Shilun Fan.
>

Re: JDK17 support for Hadoop

[slfan1989]

Thank you all for your interest in this topic!

In my opinion, upgrading is very beneficial. Based on the roadmap for Spark
4.0, we can see that JDK8/JDK11 is no longer the default version.
Therefore, I believe it is important for us to fully support
JDK11/JDK17/JDK21 in Hadoop.

>From the information I have gathered, we currently do not have full support
for JDK11. I think it is necessary for us to address this issue first and
gradually move towards supporting JDK17.

I'd like to hear other members' opinions.

I am willing to start this work to make hadoop support JDK11 and JDK17
after the release of hadoop-3.4.0.

Before that, we need to hear the opinions of other team members.

CC: @Xiaoqiao He  @Ayush Saxena 
@ste...@cloudera.com  @Takanobu Asanuma


Original

From:"Battula, Brahma Reddy"< bbatt...@visa.com.INVALID >;

Date:2024/2/7 15:45

To:"bilwa st"< stbi...@gmail.com >;"Hadoop Common"<
common-...@hadoop.apache.org >;"Hdfs-dev"< hdfs-...@hadoop.apache.org >;
"yarn-dev"< yarn-dev@hadoop.apache.org >;"mapreduce-dev"<
mapreduce-...@hadoop.apache.org >;

Subject:Re: JDK17 support for Hadoop

Thanks, Bilwa, for bringing up the issue of JDK17 support for Hadoop.

It's true that we need to have pipelines for JDK17 builds, and it's good to
see that some Jira’s are already in progress to support it.

We may need to check with the infrastructure team as well. Does anyone else
have any thoughts or concerns on this?

It seems like this topic is not very active.


From: bilwa st
Date: Monday, February 5, 2024 at 13:22
To: Hadoop Common , Hdfs-dev , yarn-dev , mapreduce-dev
Subject: Jdk17
Hi folks,

This is regarding jdk17 pending work to be done. Can we have a new pipeline
for jdk17 builds? I can see that all jira's under HADOOP-17177 which got
merged are not run on jdk17 as of now. It would be helpful if we have
dedicated pipeline.

Thanks,
Bilwa

Re: JDK17 support for Hadoop

[Battula, Brahma Reddy]

Thanks, Bilwa, for bringing up the issue of JDK17 support for Hadoop.

It's true that we need to have pipelines for JDK17 builds, and it's good to see 
that some Jira’s are already in progress to support it.

We may need to check with the infrastructure team as well. Does anyone else 
have any thoughts or concerns on this?

It seems like this topic is not very active.


From: bilwa st 
Date: Monday, February 5, 2024 at 13:22
To: Hadoop Common , Hdfs-dev 
, yarn-dev , 
mapreduce-dev 
Subject: Jdk17
Hi folks,

This is regarding jdk17 pending work to be done. Can we have a new pipeline
for jdk17 builds? I can see that all jira's under HADOOP-17177 which got
merged are not run on jdk17 as of now. It would be helpful if we have
dedicated pipeline.

Thanks,
Bilwa

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

[slfan1989]

We have completed the release of hadoop-thirdparty-1.2.0 according to the
release process of how to release.
We will start preparing for hadoop-3.4.0-RC2 voting.

Looking forward to everything going smoothly!

Best Regards,
Shilun Fan.

On Wed, Feb 7, 2024 at 10:24 AM slfan1989  wrote:

> +1
>
> Thank you very much for voting!  @Ayush Saxena  @Takanobu
> Asanuma  @Xiaoqiao He 
>
> We have collected 4 PMC votes for hadoop-thirdparty-1.2.0-RC1, and we will
> release hadoop-thirdparty-1.2.0.
>
> Best Regards,
> Shilun Fan.
>
> On Wed, Feb 7, 2024 at 4:24 AM Ayush Saxena  wrote:
>
>> +1 (Binding)
>>
>> * Built from source
>> * Verified Checksums
>> * Verified Signatures
>> * Verified no diff b/w the git tag & src tar
>> * Verified LICENSE & NOTICE files
>> * Compiled hadoop trunk with 1.2.0-RC1*
>>
>> Thanx Shilun Fan & Xiaoqiao He for driving the release. Good Luck!!!
>>
>> * You need to change that protobuf artifact name to compile, that ain't
>> fancy at all, the exclusions and all need to be updated after every
>> release, this thing we should change maybe from next release and let the
>> dependency stay without the version suffix. Incompatible for sure, but the
>> current approach isn't very compatible either, If someone is having a
>> dependency of any hadoop module, which pulls in this shaded jar
>> transitively, that guy needs to update his exclusions post every release
>> and that IMO isn't a very good experience, lets chase that in our next
>> release or have some discussions around it
>>
>> -Ayush
>>
>>
>> On Tue, 6 Feb 2024 at 19:14, Takanobu Asanuma 
>> wrote:
>>
>>> +1 (binding).
>>>
>>> * Verified signatures and checksums
>>> * Reviewed the documents
>>> * Successfully built from source with `mvn clean install`
>>> * Successfully compiled Hadoop trunk and branch-3.4 with `mvn clean
>>> install
>>> -DskipTests` using the thirdparty 1.2.0-RC1
>>> * There are not any diffs between tag and src
>>>
>>> Regards,
>>> - Takanobu Asanuma
>>>
>>> 2024年2月6日(火) 11:03 Xiaoqiao He :
>>>
>>> > cc @PJ Fanning  @Ayush Saxena <
>>> ayush...@gmail.com> @Steve
>>> > Loughran  @Takanobu Asanuma 
>>> @Shuyan
>>> > Zhang  @inigo...@apache.org <
>>> inigo...@apache.org>
>>> >
>>> > On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He 
>>> wrote:
>>> >
>>> >> +1(binding).
>>> >>
>>> >> I checked the following items:
>>> >> - [X] Download links are valid.
>>> >> - [X] Checksums and PGP signatures are valid.
>>> >> - [X] LICENSE and NOTICE files are correct for the repository.
>>> >> - [X] Source code artifacts have correct names matching the current
>>> >> release.
>>> >> - [X] All files have license headers if necessary.
>>> >> - [X] Building is OK using `mvn clean install` on JDK_1.8.0_202.
>>> >> - [X] Built Hadoop trunk successfully with updated thirdparty (include
>>> >> update protobuf shaded path).
>>> >> - [X] No difference between tag and release src tar.
>>> >>
>>> >> Good Luck!
>>> >>
>>> >> Best Regards,
>>> >> - He Xiaoqiao
>>> >>
>>> >>
>>> >> On Sun, Feb 4, 2024 at 10:29 PM slfan1989 
>>> wrote:
>>> >>
>>> >>> Hi folks,
>>> >>>
>>> >>> Xiaoqiao He and I have put together a release candidate (RC1) for
>>> Hadoop
>>> >>> Thirdparty 1.2.0.
>>> >>>
>>> >>> The RC is available at:
>>> >>>
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1
>>> >>>
>>> >>> The RC tag is
>>> >>>
>>> >>>
>>> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC1
>>> >>>
>>> >>> The maven artifacts are staged at
>>> >>>
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1401
>>> >>>
>>> >>> Comparing to 1.1.1, there are three additional fixes:
>>> >>>
>>> >>> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
>>> >>> https://github.com/apache/hadoop-thirdparty/pull/26
>>> >>>
>>> >>> HADOOP-18921. Upgrade to avro 1.11.3
>>> >>> https://github.com/apache/hadoop-thirdparty/pull/24
>>> >>>
>>> >>> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 (#23)
>>> >>> https://github.com/apache/hadoop-thirdparty/pull/23
>>> >>>
>>> >>> You can find my public key at :
>>> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>> >>>
>>> >>> Best Regards,
>>> >>> Shilun Fan.
>>> >>>
>>> >>>
>>>
>>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

[slfan1989]

+1

Thank you very much for voting!  @Ayush Saxena  @Takanobu
Asanuma  @Xiaoqiao He 

We have collected 4 PMC votes for hadoop-thirdparty-1.2.0-RC1, and we will
release hadoop-thirdparty-1.2.0.

Best Regards,
Shilun Fan.

On Wed, Feb 7, 2024 at 4:24 AM Ayush Saxena  wrote:

> +1 (Binding)
>
> * Built from source
> * Verified Checksums
> * Verified Signatures
> * Verified no diff b/w the git tag & src tar
> * Verified LICENSE & NOTICE files
> * Compiled hadoop trunk with 1.2.0-RC1*
>
> Thanx Shilun Fan & Xiaoqiao He for driving the release. Good Luck!!!
>
> * You need to change that protobuf artifact name to compile, that ain't
> fancy at all, the exclusions and all need to be updated after every
> release, this thing we should change maybe from next release and let the
> dependency stay without the version suffix. Incompatible for sure, but the
> current approach isn't very compatible either, If someone is having a
> dependency of any hadoop module, which pulls in this shaded jar
> transitively, that guy needs to update his exclusions post every release
> and that IMO isn't a very good experience, lets chase that in our next
> release or have some discussions around it
>
> -Ayush
>
>
> On Tue, 6 Feb 2024 at 19:14, Takanobu Asanuma  wrote:
>
>> +1 (binding).
>>
>> * Verified signatures and checksums
>> * Reviewed the documents
>> * Successfully built from source with `mvn clean install`
>> * Successfully compiled Hadoop trunk and branch-3.4 with `mvn clean
>> install
>> -DskipTests` using the thirdparty 1.2.0-RC1
>> * There are not any diffs between tag and src
>>
>> Regards,
>> - Takanobu Asanuma
>>
>> 2024年2月6日(火) 11:03 Xiaoqiao He :
>>
>> > cc @PJ Fanning  @Ayush Saxena 
>> @Steve
>> > Loughran  @Takanobu Asanuma 
>> @Shuyan
>> > Zhang  @inigo...@apache.org <
>> inigo...@apache.org>
>> >
>> > On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He 
>> wrote:
>> >
>> >> +1(binding).
>> >>
>> >> I checked the following items:
>> >> - [X] Download links are valid.
>> >> - [X] Checksums and PGP signatures are valid.
>> >> - [X] LICENSE and NOTICE files are correct for the repository.
>> >> - [X] Source code artifacts have correct names matching the current
>> >> release.
>> >> - [X] All files have license headers if necessary.
>> >> - [X] Building is OK using `mvn clean install` on JDK_1.8.0_202.
>> >> - [X] Built Hadoop trunk successfully with updated thirdparty (include
>> >> update protobuf shaded path).
>> >> - [X] No difference between tag and release src tar.
>> >>
>> >> Good Luck!
>> >>
>> >> Best Regards,
>> >> - He Xiaoqiao
>> >>
>> >>
>> >> On Sun, Feb 4, 2024 at 10:29 PM slfan1989 
>> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Xiaoqiao He and I have put together a release candidate (RC1) for
>> Hadoop
>> >>> Thirdparty 1.2.0.
>> >>>
>> >>> The RC is available at:
>> >>>
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1
>> >>>
>> >>> The RC tag is
>> >>>
>> >>>
>> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC1
>> >>>
>> >>> The maven artifacts are staged at
>> >>>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1401
>> >>>
>> >>> Comparing to 1.1.1, there are three additional fixes:
>> >>>
>> >>> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
>> >>> https://github.com/apache/hadoop-thirdparty/pull/26
>> >>>
>> >>> HADOOP-18921. Upgrade to avro 1.11.3
>> >>> https://github.com/apache/hadoop-thirdparty/pull/24
>> >>>
>> >>> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 (#23)
>> >>> https://github.com/apache/hadoop-thirdparty/pull/23
>> >>>
>> >>> You can find my public key at :
>> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >>>
>> >>> Best Regards,
>> >>> Shilun Fan.
>> >>>
>> >>>
>>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

[Ayush Saxena]

+1 (Binding)

* Built from source
* Verified Checksums
* Verified Signatures
* Verified no diff b/w the git tag & src tar
* Verified LICENSE & NOTICE files
* Compiled hadoop trunk with 1.2.0-RC1*

Thanx Shilun Fan & Xiaoqiao He for driving the release. Good Luck!!!

* You need to change that protobuf artifact name to compile, that ain't
fancy at all, the exclusions and all need to be updated after every
release, this thing we should change maybe from next release and let the
dependency stay without the version suffix. Incompatible for sure, but the
current approach isn't very compatible either, If someone is having a
dependency of any hadoop module, which pulls in this shaded jar
transitively, that guy needs to update his exclusions post every release
and that IMO isn't a very good experience, lets chase that in our next
release or have some discussions around it

-Ayush


On Tue, 6 Feb 2024 at 19:14, Takanobu Asanuma  wrote:

> +1 (binding).
>
> * Verified signatures and checksums
> * Reviewed the documents
> * Successfully built from source with `mvn clean install`
> * Successfully compiled Hadoop trunk and branch-3.4 with `mvn clean install
> -DskipTests` using the thirdparty 1.2.0-RC1
> * There are not any diffs between tag and src
>
> Regards,
> - Takanobu Asanuma
>
> 2024年2月6日(火) 11:03 Xiaoqiao He :
>
> > cc @PJ Fanning  @Ayush Saxena 
> @Steve
> > Loughran  @Takanobu Asanuma 
> @Shuyan
> > Zhang  @inigo...@apache.org  >
> >
> > On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He 
> wrote:
> >
> >> +1(binding).
> >>
> >> I checked the following items:
> >> - [X] Download links are valid.
> >> - [X] Checksums and PGP signatures are valid.
> >> - [X] LICENSE and NOTICE files are correct for the repository.
> >> - [X] Source code artifacts have correct names matching the current
> >> release.
> >> - [X] All files have license headers if necessary.
> >> - [X] Building is OK using `mvn clean install` on JDK_1.8.0_202.
> >> - [X] Built Hadoop trunk successfully with updated thirdparty (include
> >> update protobuf shaded path).
> >> - [X] No difference between tag and release src tar.
> >>
> >> Good Luck!
> >>
> >> Best Regards,
> >> - He Xiaoqiao
> >>
> >>
> >> On Sun, Feb 4, 2024 at 10:29 PM slfan1989  wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Xiaoqiao He and I have put together a release candidate (RC1) for
> Hadoop
> >>> Thirdparty 1.2.0.
> >>>
> >>> The RC is available at:
> >>>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1
> >>>
> >>> The RC tag is
> >>>
> >>>
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC1
> >>>
> >>> The maven artifacts are staged at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1401
> >>>
> >>> Comparing to 1.1.1, there are three additional fixes:
> >>>
> >>> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> >>> https://github.com/apache/hadoop-thirdparty/pull/26
> >>>
> >>> HADOOP-18921. Upgrade to avro 1.11.3
> >>> https://github.com/apache/hadoop-thirdparty/pull/24
> >>>
> >>> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 (#23)
> >>> https://github.com/apache/hadoop-thirdparty/pull/23
> >>>
> >>> You can find my public key at :
> >>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >>>
> >>> Best Regards,
> >>> Shilun Fan.
> >>>
> >>>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

[Takanobu Asanuma]

+1 (binding).

* Verified signatures and checksums
* Reviewed the documents
* Successfully built from source with `mvn clean install`
* Successfully compiled Hadoop trunk and branch-3.4 with `mvn clean install
-DskipTests` using the thirdparty 1.2.0-RC1
* There are not any diffs between tag and src

Regards,
- Takanobu Asanuma

2024年2月6日(火) 11:03 Xiaoqiao He :

> cc @PJ Fanning  @Ayush Saxena  
> @Steve
> Loughran  @Takanobu Asanuma  @Shuyan
> Zhang  @inigo...@apache.org 
>
> On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He  wrote:
>
>> +1(binding).
>>
>> I checked the following items:
>> - [X] Download links are valid.
>> - [X] Checksums and PGP signatures are valid.
>> - [X] LICENSE and NOTICE files are correct for the repository.
>> - [X] Source code artifacts have correct names matching the current
>> release.
>> - [X] All files have license headers if necessary.
>> - [X] Building is OK using `mvn clean install` on JDK_1.8.0_202.
>> - [X] Built Hadoop trunk successfully with updated thirdparty (include
>> update protobuf shaded path).
>> - [X] No difference between tag and release src tar.
>>
>> Good Luck!
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>>
>> On Sun, Feb 4, 2024 at 10:29 PM slfan1989  wrote:
>>
>>> Hi folks,
>>>
>>> Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop
>>> Thirdparty 1.2.0.
>>>
>>> The RC is available at:
>>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1
>>>
>>> The RC tag is
>>>
>>> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC1
>>>
>>> The maven artifacts are staged at
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1401
>>>
>>> Comparing to 1.1.1, there are three additional fixes:
>>>
>>> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
>>> https://github.com/apache/hadoop-thirdparty/pull/26
>>>
>>> HADOOP-18921. Upgrade to avro 1.11.3
>>> https://github.com/apache/hadoop-thirdparty/pull/24
>>>
>>> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 (#23)
>>> https://github.com/apache/hadoop-thirdparty/pull/23
>>>
>>> You can find my public key at :
>>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>>
>>> Best Regards,
>>> Shilun Fan.
>>>
>>>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

[Xiaoqiao He]

cc @PJ Fanning  @Ayush Saxena  @Steve
Loughran  @Takanobu Asanuma  @Shuyan
Zhang  @inigo...@apache.org 

On Mon, Feb 5, 2024 at 11:30 AM Xiaoqiao He  wrote:

> +1(binding).
>
> I checked the following items:
> - [X] Download links are valid.
> - [X] Checksums and PGP signatures are valid.
> - [X] LICENSE and NOTICE files are correct for the repository.
> - [X] Source code artifacts have correct names matching the current
> release.
> - [X] All files have license headers if necessary.
> - [X] Building is OK using `mvn clean install` on JDK_1.8.0_202.
> - [X] Built Hadoop trunk successfully with updated thirdparty (include
> update protobuf shaded path).
> - [X] No difference between tag and release src tar.
>
> Good Luck!
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Sun, Feb 4, 2024 at 10:29 PM slfan1989  wrote:
>
>> Hi folks,
>>
>> Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop
>> Thirdparty 1.2.0.
>>
>> The RC is available at:
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1
>>
>> The RC tag is
>> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC1
>>
>> The maven artifacts are staged at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1401
>>
>> Comparing to 1.1.1, there are three additional fixes:
>>
>> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
>> https://github.com/apache/hadoop-thirdparty/pull/26
>>
>> HADOOP-18921. Upgrade to avro 1.11.3
>> https://github.com/apache/hadoop-thirdparty/pull/24
>>
>> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 (#23)
>> https://github.com/apache/hadoop-thirdparty/pull/23
>>
>> You can find my public key at :
>> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>>
>> Best Regards,
>> Shilun Fan.
>>
>>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 (RC1)

[Xiaoqiao He]

+1(binding).

I checked the following items:
- [X] Download links are valid.
- [X] Checksums and PGP signatures are valid.
- [X] LICENSE and NOTICE files are correct for the repository.
- [X] Source code artifacts have correct names matching the current release.
- [X] All files have license headers if necessary.
- [X] Building is OK using `mvn clean install` on JDK_1.8.0_202.
- [X] Built Hadoop trunk successfully with updated thirdparty (include
update protobuf shaded path).
- [X] No difference between tag and release src tar.

Good Luck!

Best Regards,
- He Xiaoqiao


On Sun, Feb 4, 2024 at 10:29 PM slfan1989  wrote:

> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC1) for Hadoop
> Thirdparty 1.2.0.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC1
>
> The RC tag is
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC1
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1401
>
> Comparing to 1.1.1, there are three additional fixes:
>
> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> https://github.com/apache/hadoop-thirdparty/pull/26
>
> HADOOP-18921. Upgrade to avro 1.11.3
> https://github.com/apache/hadoop-thirdparty/pull/24
>
> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976 (#23)
> https://github.com/apache/hadoop-thirdparty/pull/23
>
> You can find my public key at :
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Best Regards,
> Shilun Fan.
>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[slfan1989]

Thank you all very much for helping with the review and vote!

I will conclude the voting for Hadoop-thirdparty-1.2.0-RC0 and open the
voting for Hadoop-thirdparty-1.2.0-RC1.

Thank you for the feedback on the code diff, Ayush. I have already
submitted the changes to the Dockerfile and create-release files to the
Hadoop-thirdparty trunk branch and backported them to branch-1.2 and
branch-1.2.0. The issues you provided feedback on will be addressed in
Hadoop-thirdparty-1.2.0-RC1.

Once again, thank you Xiaoqiao He, Ayush Saxena, Takanobu Asanuma, PJ
Fanning, Shuyan Zhang for the review and vote.

Best Regards,
Shilun Fan.

On Sun, Feb 4, 2024 at 4:18 PM Shuyan Zhang  wrote:

> +1 (non-binding)
>
> - Verified hashes
> - LICENSE and NOTICE are included.
> - Rat check is ok. `mvn clean apache-rat:check`
> - `mvn clean install` works well
>
>
> slfan1989  于2024年2月2日周五 11:11写道：
>
> > Thank you very much for the review! I will avoid the diff.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Fri, Feb 2, 2024 at 9:59 AM Takanobu Asanuma 
> > wrote:
> >
> > > It also looks good to me, except for the diff.
> > >
> > > * Verified signatures and hashes
> > > * Reviewed the documents
> > > * Successfully built from source with `mvn clean install`
> > > * Successfully compiled Hadoop trunk and branch-3.4 using the Hadoop
> > > thirdparty 1.2.0
> > >
> > > Anyway, since hadoop-thirdparty-1.1.1 has some high vulnerabilities,
> > > hadoop-thirdparty-1.2.0 would be required for Hadoop-3.4.0.
> > >
> > > Thanks,
> > > - Takanobu
> > >
> > > 2024年2月2日(金) 4:45 slfan1989 :
> > >
> > > > Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and
> > providing
> > > > feedback!
> > > >
> > > > I followed the "how to release" documentation and tried to package it
> > > using
> > > > create-release and Dockerfile, but I couldn't successfully package it
> > > > directly. Some modifications are required before compilation. I
> should
> > > > submit a pull request to fix this issue before
> > > > Hadoop-Thirdparty-1.2.0-RC0 compile.
> > > >
> > > > This is an area that needs improvement. We should ensure that the
> code
> > of
> > > > src is consistent with the tag.
> > > >
> > > > On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena 
> > wrote:
> > > >
> > > > >
> > > > > There is some diff b/w the git tag & the src tar, the Dockerfile &
> > the
> > > > > create-release are different, Why?
> > > > >
> > > > > Files hadoop-thirdparty/dev-support/bin/create-release and
> > > > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
> > > > >
> > > > > Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> > > > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
> > > > >
> > > > >
> > > > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > > > hadoop-thirdparty/dev-support/bin/create-release
> > > > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
> > > > >
> > > > > 444,446c444,446
> > > > >
> > > > > < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
> > > > >
> > > > > < echo "RUN useradd -g ${group_id} -u ${user_id} -m
> ${user_name}"
> > > > >
> > > > > < echo "RUN chown -R ${user_name} /home/${user_name}"
> > > > >
> > > > > ---
> > > > >
> > > > > > echo "RUN groupadd --non-unique -g ${group_id} ${user_name};
> > exit
> > > > > 0;"
> > > > >
> > > > > > echo "RUN useradd -g ${group_id} -u ${user_id} -m
> ${user_name};
> > > > > exit 0;"
> > > > >
> > > > > > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
> > > > >
> > > > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > > > hadoop-thirdparty/dev-support/docker/Dockerfile
> > > > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
> > > > >
> > > > > 103a104,105
> > > > >
> > > > > > RUN rm -f /etc/maven/settings.xml && ln -s
> > > /home/root/.m2/settings.xml
> > > > > /etc/maven/settings.xml
> > > > >
> > > > > >
> > > > >
> > > > > 126a129,130
> > > > >
> > > > > > RUN pip2 install setuptools-scm==5.0.2
> > > > >
> > > > > > RUN pip2 install lazy-object-proxy==1.5.0
> > > > >
> > > > > 159d162
> > > > >
> > > > > <
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Other things look Ok,
> > > > > * Built from source
> > > > > * Verified Checksums
> > > > > * Verified Signatures
> > > > > * Validated files have ASF header
> > > > >
> > > > > Not sure if having diff b/w the git tag & src tar is ok, this
> doesn't
> > > > look
> > > > > like core code change though, can anybody check & confirm?
> > > > >
> > > > > -Ayush
> > > > >
> > > > >
> > > > > On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He 
> > > wrote:
> > > > >
> > > > >> Gentle ping. @Ayush Saxena  @Steve Loughran
> > > > >>  @inigo...@apache.org 
> > > > >> @Masatake
> > > > >> Iwasaki  and some other folks.
> > > > >>
> > > > >> On Wed, Jan 31, 2024 at 10:17 AM slfan1989 
> > > > wrote:
> > > > >>
> > > > >> > Thank you for the review and vote! Looking forward to other
> forks
> > > > >> helping
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[Shuyan Zhang]

+1 (non-binding)

- Verified hashes
- LICENSE and NOTICE are included.
- Rat check is ok. `mvn clean apache-rat:check`
- `mvn clean install` works well


slfan1989  于2024年2月2日周五 11:11写道：

> Thank you very much for the review! I will avoid the diff.
>
> Best Regards,
> Shilun Fan.
>
> On Fri, Feb 2, 2024 at 9:59 AM Takanobu Asanuma 
> wrote:
>
> > It also looks good to me, except for the diff.
> >
> > * Verified signatures and hashes
> > * Reviewed the documents
> > * Successfully built from source with `mvn clean install`
> > * Successfully compiled Hadoop trunk and branch-3.4 using the Hadoop
> > thirdparty 1.2.0
> >
> > Anyway, since hadoop-thirdparty-1.1.1 has some high vulnerabilities,
> > hadoop-thirdparty-1.2.0 would be required for Hadoop-3.4.0.
> >
> > Thanks,
> > - Takanobu
> >
> > 2024年2月2日(金) 4:45 slfan1989 :
> >
> > > Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and
> providing
> > > feedback!
> > >
> > > I followed the "how to release" documentation and tried to package it
> > using
> > > create-release and Dockerfile, but I couldn't successfully package it
> > > directly. Some modifications are required before compilation. I should
> > > submit a pull request to fix this issue before
> > > Hadoop-Thirdparty-1.2.0-RC0 compile.
> > >
> > > This is an area that needs improvement. We should ensure that the code
> of
> > > src is consistent with the tag.
> > >
> > > On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena 
> wrote:
> > >
> > > >
> > > > There is some diff b/w the git tag & the src tar, the Dockerfile &
> the
> > > > create-release are different, Why?
> > > >
> > > > Files hadoop-thirdparty/dev-support/bin/create-release and
> > > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
> > > >
> > > > Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> > > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
> > > >
> > > >
> > > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > > hadoop-thirdparty/dev-support/bin/create-release
> > > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
> > > >
> > > > 444,446c444,446
> > > >
> > > > < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
> > > >
> > > > < echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
> > > >
> > > > < echo "RUN chown -R ${user_name} /home/${user_name}"
> > > >
> > > > ---
> > > >
> > > > > echo "RUN groupadd --non-unique -g ${group_id} ${user_name};
> exit
> > > > 0;"
> > > >
> > > > > echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name};
> > > > exit 0;"
> > > >
> > > > > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
> > > >
> > > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > > hadoop-thirdparty/dev-support/docker/Dockerfile
> > > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
> > > >
> > > > 103a104,105
> > > >
> > > > > RUN rm -f /etc/maven/settings.xml && ln -s
> > /home/root/.m2/settings.xml
> > > > /etc/maven/settings.xml
> > > >
> > > > >
> > > >
> > > > 126a129,130
> > > >
> > > > > RUN pip2 install setuptools-scm==5.0.2
> > > >
> > > > > RUN pip2 install lazy-object-proxy==1.5.0
> > > >
> > > > 159d162
> > > >
> > > > <
> > > >
> > > >
> > > >
> > > >
> > > > Other things look Ok,
> > > > * Built from source
> > > > * Verified Checksums
> > > > * Verified Signatures
> > > > * Validated files have ASF header
> > > >
> > > > Not sure if having diff b/w the git tag & src tar is ok, this doesn't
> > > look
> > > > like core code change though, can anybody check & confirm?
> > > >
> > > > -Ayush
> > > >
> > > >
> > > > On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He 
> > wrote:
> > > >
> > > >> Gentle ping. @Ayush Saxena  @Steve Loughran
> > > >>  @inigo...@apache.org 
> > > >> @Masatake
> > > >> Iwasaki  and some other folks.
> > > >>
> > > >> On Wed, Jan 31, 2024 at 10:17 AM slfan1989 
> > > wrote:
> > > >>
> > > >> > Thank you for the review and vote! Looking forward to other forks
> > > >> helping
> > > >> > with voting and verification.
> > > >> >
> > > >> > Best Regards,
> > > >> > Shilun Fan.
> > > >> >
> > > >> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He <
> hexiaoq...@apache.org>
> > > >> wrote:
> > > >> >
> > > >> > > Thanks Shilun for driving it and making it happen.
> > > >> > >
> > > >> > > +1(binding).
> > > >> > >
> > > >> > > [x] Checksums and PGP signatures are valid.
> > > >> > > [x] LICENSE files exist.
> > > >> > > [x] NOTICE is included.
> > > >> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> > > >> > > [x] Built from source works well: `mvn clean install`
> > > >> > > [x] Built Hadoop trunk with updated thirdparty successfully
> > (include
> > > >> > update
> > > >> > > protobuf shaded path).
> > > >> > >
> > > >> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0,
> > hope
> > > >> we
> > > >> > > could finish this vote before 2024/02/06(UTC) if there are no
> > > >> concerns.
> > > >> > > Thanks all.

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[slfan1989]

Thank you very much for the review! I will avoid the diff.

Best Regards,
Shilun Fan.

On Fri, Feb 2, 2024 at 9:59 AM Takanobu Asanuma  wrote:

> It also looks good to me, except for the diff.
>
> * Verified signatures and hashes
> * Reviewed the documents
> * Successfully built from source with `mvn clean install`
> * Successfully compiled Hadoop trunk and branch-3.4 using the Hadoop
> thirdparty 1.2.0
>
> Anyway, since hadoop-thirdparty-1.1.1 has some high vulnerabilities,
> hadoop-thirdparty-1.2.0 would be required for Hadoop-3.4.0.
>
> Thanks,
> - Takanobu
>
> 2024年2月2日(金) 4:45 slfan1989 :
>
> > Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and providing
> > feedback!
> >
> > I followed the "how to release" documentation and tried to package it
> using
> > create-release and Dockerfile, but I couldn't successfully package it
> > directly. Some modifications are required before compilation. I should
> > submit a pull request to fix this issue before
> > Hadoop-Thirdparty-1.2.0-RC0 compile.
> >
> > This is an area that needs improvement. We should ensure that the code of
> > src is consistent with the tag.
> >
> > On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena  wrote:
> >
> > >
> > > There is some diff b/w the git tag & the src tar, the Dockerfile & the
> > > create-release are different, Why?
> > >
> > > Files hadoop-thirdparty/dev-support/bin/create-release and
> > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
> > >
> > > Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
> > >
> > >
> > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > hadoop-thirdparty/dev-support/bin/create-release
> > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
> > >
> > > 444,446c444,446
> > >
> > > < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
> > >
> > > < echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
> > >
> > > < echo "RUN chown -R ${user_name} /home/${user_name}"
> > >
> > > ---
> > >
> > > > echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit
> > > 0;"
> > >
> > > > echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name};
> > > exit 0;"
> > >
> > > > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
> > >
> > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > hadoop-thirdparty/dev-support/docker/Dockerfile
> > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
> > >
> > > 103a104,105
> > >
> > > > RUN rm -f /etc/maven/settings.xml && ln -s
> /home/root/.m2/settings.xml
> > > /etc/maven/settings.xml
> > >
> > > >
> > >
> > > 126a129,130
> > >
> > > > RUN pip2 install setuptools-scm==5.0.2
> > >
> > > > RUN pip2 install lazy-object-proxy==1.5.0
> > >
> > > 159d162
> > >
> > > <
> > >
> > >
> > >
> > >
> > > Other things look Ok,
> > > * Built from source
> > > * Verified Checksums
> > > * Verified Signatures
> > > * Validated files have ASF header
> > >
> > > Not sure if having diff b/w the git tag & src tar is ok, this doesn't
> > look
> > > like core code change though, can anybody check & confirm?
> > >
> > > -Ayush
> > >
> > >
> > > On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He 
> wrote:
> > >
> > >> Gentle ping. @Ayush Saxena  @Steve Loughran
> > >>  @inigo...@apache.org 
> > >> @Masatake
> > >> Iwasaki  and some other folks.
> > >>
> > >> On Wed, Jan 31, 2024 at 10:17 AM slfan1989 
> > wrote:
> > >>
> > >> > Thank you for the review and vote! Looking forward to other forks
> > >> helping
> > >> > with voting and verification.
> > >> >
> > >> > Best Regards,
> > >> > Shilun Fan.
> > >> >
> > >> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
> > >> wrote:
> > >> >
> > >> > > Thanks Shilun for driving it and making it happen.
> > >> > >
> > >> > > +1(binding).
> > >> > >
> > >> > > [x] Checksums and PGP signatures are valid.
> > >> > > [x] LICENSE files exist.
> > >> > > [x] NOTICE is included.
> > >> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> > >> > > [x] Built from source works well: `mvn clean install`
> > >> > > [x] Built Hadoop trunk with updated thirdparty successfully
> (include
> > >> > update
> > >> > > protobuf shaded path).
> > >> > >
> > >> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0,
> hope
> > >> we
> > >> > > could finish this vote before 2024/02/06(UTC) if there are no
> > >> concerns.
> > >> > > Thanks all.
> > >> > >
> > >> > > Best Regards,
> > >> > > - He Xiaoqiao
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
> > >> wrote:
> > >> > >
> > >> > > > Hi folks,
> > >> > > >
> > >> > > > Xiaoqiao He and I have put together a release candidate (RC0)
> for
> > >> > Hadoop
> > >> > > > Thirdparty 1.2.0.
> > >> > > >
> > >> > > > The RC is available at:
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> > >> > > >
> 

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[Takanobu Asanuma]

It also looks good to me, except for the diff.

* Verified signatures and hashes
* Reviewed the documents
* Successfully built from source with `mvn clean install`
* Successfully compiled Hadoop trunk and branch-3.4 using the Hadoop
thirdparty 1.2.0

Anyway, since hadoop-thirdparty-1.1.1 has some high vulnerabilities,
hadoop-thirdparty-1.2.0 would be required for Hadoop-3.4.0.

Thanks,
- Takanobu

2024年2月2日(金) 4:45 slfan1989 :

> Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and providing
> feedback!
>
> I followed the "how to release" documentation and tried to package it using
> create-release and Dockerfile, but I couldn't successfully package it
> directly. Some modifications are required before compilation. I should
> submit a pull request to fix this issue before
> Hadoop-Thirdparty-1.2.0-RC0 compile.
>
> This is an area that needs improvement. We should ensure that the code of
> src is consistent with the tag.
>
> On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena  wrote:
>
> >
> > There is some diff b/w the git tag & the src tar, the Dockerfile & the
> > create-release are different, Why?
> >
> > Files hadoop-thirdparty/dev-support/bin/create-release and
> > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
> >
> > Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
> >
> >
> > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > hadoop-thirdparty/dev-support/bin/create-release
> > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
> >
> > 444,446c444,446
> >
> > < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
> >
> > < echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
> >
> > < echo "RUN chown -R ${user_name} /home/${user_name}"
> >
> > ---
> >
> > > echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit
> > 0;"
> >
> > > echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name};
> > exit 0;"
> >
> > > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
> >
> > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > hadoop-thirdparty/dev-support/docker/Dockerfile
> > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
> >
> > 103a104,105
> >
> > > RUN rm -f /etc/maven/settings.xml && ln -s /home/root/.m2/settings.xml
> > /etc/maven/settings.xml
> >
> > >
> >
> > 126a129,130
> >
> > > RUN pip2 install setuptools-scm==5.0.2
> >
> > > RUN pip2 install lazy-object-proxy==1.5.0
> >
> > 159d162
> >
> > <
> >
> >
> >
> >
> > Other things look Ok,
> > * Built from source
> > * Verified Checksums
> > * Verified Signatures
> > * Validated files have ASF header
> >
> > Not sure if having diff b/w the git tag & src tar is ok, this doesn't
> look
> > like core code change though, can anybody check & confirm?
> >
> > -Ayush
> >
> >
> > On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He  wrote:
> >
> >> Gentle ping. @Ayush Saxena  @Steve Loughran
> >>  @inigo...@apache.org 
> >> @Masatake
> >> Iwasaki  and some other folks.
> >>
> >> On Wed, Jan 31, 2024 at 10:17 AM slfan1989 
> wrote:
> >>
> >> > Thank you for the review and vote! Looking forward to other forks
> >> helping
> >> > with voting and verification.
> >> >
> >> > Best Regards,
> >> > Shilun Fan.
> >> >
> >> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
> >> wrote:
> >> >
> >> > > Thanks Shilun for driving it and making it happen.
> >> > >
> >> > > +1(binding).
> >> > >
> >> > > [x] Checksums and PGP signatures are valid.
> >> > > [x] LICENSE files exist.
> >> > > [x] NOTICE is included.
> >> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> >> > > [x] Built from source works well: `mvn clean install`
> >> > > [x] Built Hadoop trunk with updated thirdparty successfully (include
> >> > update
> >> > > protobuf shaded path).
> >> > >
> >> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope
> >> we
> >> > > could finish this vote before 2024/02/06(UTC) if there are no
> >> concerns.
> >> > > Thanks all.
> >> > >
> >> > > Best Regards,
> >> > > - He Xiaoqiao
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
> >> wrote:
> >> > >
> >> > > > Hi folks,
> >> > > >
> >> > > > Xiaoqiao He and I have put together a release candidate (RC0) for
> >> > Hadoop
> >> > > > Thirdparty 1.2.0.
> >> > > >
> >> > > > The RC is available at:
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> >> > > >
> >> > > > The RC tag is
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> >> > > >
> >> > > > The maven artifacts are staged at
> >> > > >
> >> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1398
> >> > > >
> >> > > > Comparing to 1.1.1, there are three additional fixes:
> >> > > >
> >> > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> >> > > > https://github.com/apache/hadoop-thir

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[slfan1989]

Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and providing
feedback!

I followed the "how to release" documentation and tried to package it using
create-release and Dockerfile, but I couldn't successfully package it
directly. Some modifications are required before compilation. I should
submit a pull request to fix this issue before
Hadoop-Thirdparty-1.2.0-RC0 compile.

This is an area that needs improvement. We should ensure that the code of
src is consistent with the tag.

On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena  wrote:

>
> There is some diff b/w the git tag & the src tar, the Dockerfile & the
> create-release are different, Why?
>
> Files hadoop-thirdparty/dev-support/bin/create-release and
> hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
>
> Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
>
>
> ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> hadoop-thirdparty/dev-support/bin/create-release
> hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
>
> 444,446c444,446
>
> < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
>
> < echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
>
> < echo "RUN chown -R ${user_name} /home/${user_name}"
>
> ---
>
> > echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit
> 0;"
>
> > echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name};
> exit 0;"
>
> > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
>
> ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> hadoop-thirdparty/dev-support/docker/Dockerfile
> hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
>
> 103a104,105
>
> > RUN rm -f /etc/maven/settings.xml && ln -s /home/root/.m2/settings.xml
> /etc/maven/settings.xml
>
> >
>
> 126a129,130
>
> > RUN pip2 install setuptools-scm==5.0.2
>
> > RUN pip2 install lazy-object-proxy==1.5.0
>
> 159d162
>
> <
>
>
>
>
> Other things look Ok,
> * Built from source
> * Verified Checksums
> * Verified Signatures
> * Validated files have ASF header
>
> Not sure if having diff b/w the git tag & src tar is ok, this doesn't look
> like core code change though, can anybody check & confirm?
>
> -Ayush
>
>
> On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He  wrote:
>
>> Gentle ping. @Ayush Saxena  @Steve Loughran
>>  @inigo...@apache.org 
>> @Masatake
>> Iwasaki  and some other folks.
>>
>> On Wed, Jan 31, 2024 at 10:17 AM slfan1989  wrote:
>>
>> > Thank you for the review and vote! Looking forward to other forks
>> helping
>> > with voting and verification.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
>> wrote:
>> >
>> > > Thanks Shilun for driving it and making it happen.
>> > >
>> > > +1(binding).
>> > >
>> > > [x] Checksums and PGP signatures are valid.
>> > > [x] LICENSE files exist.
>> > > [x] NOTICE is included.
>> > > [x] Rat check is ok. `mvn clean apache-rat:check`
>> > > [x] Built from source works well: `mvn clean install`
>> > > [x] Built Hadoop trunk with updated thirdparty successfully (include
>> > update
>> > > protobuf shaded path).
>> > >
>> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope
>> we
>> > > could finish this vote before 2024/02/06(UTC) if there are no
>> concerns.
>> > > Thanks all.
>> > >
>> > > Best Regards,
>> > > - He Xiaoqiao
>> > >
>> > >
>> > >
>> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
>> wrote:
>> > >
>> > > > Hi folks,
>> > > >
>> > > > Xiaoqiao He and I have put together a release candidate (RC0) for
>> > Hadoop
>> > > > Thirdparty 1.2.0.
>> > > >
>> > > > The RC is available at:
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
>> > > >
>> > > > The RC tag is
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
>> > > >
>> > > > The maven artifacts are staged at
>> > > >
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1398
>> > > >
>> > > > Comparing to 1.1.1, there are three additional fixes:
>> > > >
>> > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
>> > > > https://github.com/apache/hadoop-thirdparty/pull/26
>> > > >
>> > > > HADOOP-18921. Upgrade to avro 1.11.3
>> > > > https://github.com/apache/hadoop-thirdparty/pull/24
>> > > >
>> > > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
>> > > > https://github.com/apache/hadoop-thirdparty/pull/23
>> > > >
>> > > > You can find my public key at :
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > > Best Regards,
>> > > > Shilun Fan.
>> > > >
>> > >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[Ayush Saxena]

There is some diff b/w the git tag & the src tar, the Dockerfile & the
create-release are different, Why?

Files hadoop-thirdparty/dev-support/bin/create-release and
hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ

Files hadoop-thirdparty/dev-support/docker/Dockerfile and
hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ


ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
hadoop-thirdparty/dev-support/bin/create-release
hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release

444,446c444,446

< echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"

< echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"

< echo "RUN chown -R ${user_name} /home/${user_name}"

---

> echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit 0;"

> echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}; exit
0;"

> echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"

ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
hadoop-thirdparty/dev-support/docker/Dockerfile
hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile

103a104,105

> RUN rm -f /etc/maven/settings.xml && ln -s /home/root/.m2/settings.xml
/etc/maven/settings.xml

>

126a129,130

> RUN pip2 install setuptools-scm==5.0.2

> RUN pip2 install lazy-object-proxy==1.5.0

159d162

<




Other things look Ok,
* Built from source
* Verified Checksums
* Verified Signatures
* Validated files have ASF header

Not sure if having diff b/w the git tag & src tar is ok, this doesn't look
like core code change though, can anybody check & confirm?

-Ayush


On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He  wrote:

> Gentle ping. @Ayush Saxena  @Steve Loughran
>  @inigo...@apache.org  @Masatake
> Iwasaki  and some other folks.
>
> On Wed, Jan 31, 2024 at 10:17 AM slfan1989  wrote:
>
> > Thank you for the review and vote! Looking forward to other forks helping
> > with voting and verification.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
> wrote:
> >
> > > Thanks Shilun for driving it and making it happen.
> > >
> > > +1(binding).
> > >
> > > [x] Checksums and PGP signatures are valid.
> > > [x] LICENSE files exist.
> > > [x] NOTICE is included.
> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> > > [x] Built from source works well: `mvn clean install`
> > > [x] Built Hadoop trunk with updated thirdparty successfully (include
> > update
> > > protobuf shaded path).
> > >
> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we
> > > could finish this vote before 2024/02/06(UTC) if there are no concerns.
> > > Thanks all.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > >
> > >
> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC0) for
> > Hadoop
> > > > Thirdparty 1.2.0.
> > > >
> > > > The RC is available at:
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> > > >
> > > > The RC tag is
> > > >
> > >
> >
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1398
> > > >
> > > > Comparing to 1.1.1, there are three additional fixes:
> > > >
> > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> > > > https://github.com/apache/hadoop-thirdparty/pull/26
> > > >
> > > > HADOOP-18921. Upgrade to avro 1.11.3
> > > > https://github.com/apache/hadoop-thirdparty/pull/24
> > > >
> > > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
> > > > https://github.com/apache/hadoop-thirdparty/pull/23
> > > >
> > > > You can find my public key at :
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > >
> >
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[Xiaoqiao He]

Gentle ping. @Ayush Saxena  @Steve Loughran
 @inigo...@apache.org  @Masatake
Iwasaki  and some other folks.

On Wed, Jan 31, 2024 at 10:17 AM slfan1989  wrote:

> Thank you for the review and vote! Looking forward to other forks helping
> with voting and verification.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He  wrote:
>
> > Thanks Shilun for driving it and making it happen.
> >
> > +1(binding).
> >
> > [x] Checksums and PGP signatures are valid.
> > [x] LICENSE files exist.
> > [x] NOTICE is included.
> > [x] Rat check is ok. `mvn clean apache-rat:check`
> > [x] Built from source works well: `mvn clean install`
> > [x] Built Hadoop trunk with updated thirdparty successfully (include
> update
> > protobuf shaded path).
> >
> > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we
> > could finish this vote before 2024/02/06(UTC) if there are no concerns.
> > Thanks all.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> >
> >
> > On Mon, Jan 29, 2024 at 10:42 PM slfan1989  wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC0) for
> Hadoop
> > > Thirdparty 1.2.0.
> > >
> > > The RC is available at:
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> > >
> > > The RC tag is
> > >
> >
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1398
> > >
> > > Comparing to 1.1.1, there are three additional fixes:
> > >
> > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> > > https://github.com/apache/hadoop-thirdparty/pull/26
> > >
> > > HADOOP-18921. Upgrade to avro 1.11.3
> > > https://github.com/apache/hadoop-thirdparty/pull/24
> > >
> > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
> > > https://github.com/apache/hadoop-thirdparty/pull/23
> > >
> > > You can find my public key at :
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> >
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[slfan1989]

Thank you for the review and vote! Looking forward to other forks helping
with voting and verification.

Best Regards,
Shilun Fan.

On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He  wrote:

> Thanks Shilun for driving it and making it happen.
>
> +1(binding).
>
> [x] Checksums and PGP signatures are valid.
> [x] LICENSE files exist.
> [x] NOTICE is included.
> [x] Rat check is ok. `mvn clean apache-rat:check`
> [x] Built from source works well: `mvn clean install`
> [x] Built Hadoop trunk with updated thirdparty successfully (include update
> protobuf shaded path).
>
> BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we
> could finish this vote before 2024/02/06(UTC) if there are no concerns.
> Thanks all.
>
> Best Regards,
> - He Xiaoqiao
>
>
>
> On Mon, Jan 29, 2024 at 10:42 PM slfan1989  wrote:
>
> > Hi folks,
> >
> > Xiaoqiao He and I have put together a release candidate (RC0) for Hadoop
> > Thirdparty 1.2.0.
> >
> > The RC is available at:
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> >
> > The RC tag is
> >
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> >
> > The maven artifacts are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1398
> >
> > Comparing to 1.1.1, there are three additional fixes:
> >
> > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> > https://github.com/apache/hadoop-thirdparty/pull/26
> >
> > HADOOP-18921. Upgrade to avro 1.11.3
> > https://github.com/apache/hadoop-thirdparty/pull/24
> >
> > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
> > https://github.com/apache/hadoop-thirdparty/pull/23
> >
> > You can find my public key at :
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Best Regards,
> > Shilun Fan.
> >
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[Xiaoqiao He]

Thanks Shilun for driving it and making it happen.

+1(binding).

[x] Checksums and PGP signatures are valid.
[x] LICENSE files exist.
[x] NOTICE is included.
[x] Rat check is ok. `mvn clean apache-rat:check`
[x] Built from source works well: `mvn clean install`
[x] Built Hadoop trunk with updated thirdparty successfully (include update
protobuf shaded path).

BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we
could finish this vote before 2024/02/06(UTC) if there are no concerns.
Thanks all.

Best Regards,
- He Xiaoqiao



On Mon, Jan 29, 2024 at 10:42 PM slfan1989  wrote:

> Hi folks,
>
> Xiaoqiao He and I have put together a release candidate (RC0) for Hadoop
> Thirdparty 1.2.0.
>
> The RC is available at:
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
>
> The RC tag is
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
>
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1398
>
> Comparing to 1.1.1, there are three additional fixes:
>
> HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> https://github.com/apache/hadoop-thirdparty/pull/26
>
> HADOOP-18921. Upgrade to avro 1.11.3
> https://github.com/apache/hadoop-thirdparty/pull/24
>
> HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
> https://github.com/apache/hadoop-thirdparty/pull/23
>
> You can find my public key at :
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Best Regards,
> Shilun Fan.
>

Re: [INFO] Hadoop-3.4 Release Update

[slfan1989]

Thanks for driving this, I will be ready to release hadoop-3.4.0-RC1 in 1-2
days.

Best Regards,
Shilun Fan.

On Thu, Jan 25, 2024 at 10:22 AM Xiaoqiao He  wrote:

> Hi all, the branch-3.4.0 codebase has been frozen. Hadoop-3.4.0-RC1 will
> be ready for a while.
> Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Tue, Jan 23, 2024 at 7:58 AM slfan1989  wrote:
>
>> branch-3.4/branch-3.4.0 was created from the trunk branch, including the
>> changes from branch-3.3.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Tue, Jan 23, 2024 at 12:58 AM Brahma Reddy Battula 
>> wrote:
>>
>> > Thanks for driving this.
>> >
>> > One query:
>> > Does all the jiras from 3.3 are part of the 3.4?
>> >
>> >
>> >
>> >
>> > On Mon, 22 Jan 2024 at 11:50 AM, slfan1989 
>> wrote:
>> >
>> >> Thank you very much for specifying the frozen time for the
>> >> Branch-3.4.0.  @Xiaoqiao
>> >> He 
>> >>
>> >> If there is a need to backport any PRs to branch-3.4/branch-3.4.0,
>> please
>> >> feel free to contact me.
>> >>
>> >> Best Regards,
>> >> Shilun Fan.
>> >>
>> >> On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He 
>> >> wrote:
>> >>
>> >> > Hi All,
>> >> >
>> >> > Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
>> >> > If there is any blocker/critical PR please backport to branch-3.4 and
>> >> > branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.
>> >> >
>> >> > If any reported issues are not ready in time please wait for the next
>> >> > release.
>> >> >
>> >> > Thanks all.
>> >> >
>> >> > Best Regards,
>> >> > - He Xiaoqiao
>> >> >
>> >> >
>> >> >
>> >> > On Mon, Jan 22, 2024 at 10:43 AM slfan1989 
>> >> wrote:
>> >> >
>> >> >> Thank you for the feedback!
>> >> >>
>> >> >> I apologize for updating partial information in the Hadoop module.
>> >> >>
>> >> >> Today, I will complete the rollback of the modified JIRA and add
>> back
>> >> the
>> >> >> 3.4.0 version.
>> >> >>
>> >> >> Best
>> >> >>
>> >> >> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He > >
>> >> >> wrote:
>> >> >>
>> >> >>> Thanks all for your input. Connected to Shilun offline yesterday,
>> and
>> >> he
>> >> >>> will
>> >> >>> update or recover the fix version tag before 3.4.0-RC1.
>> >> >>>
>> >> >>> Best Regards,
>> >> >>> - He Xiaoqiao
>> >> >>>
>> >> >>>
>> >> >>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki <
>> >> iwasak...@apache.org>
>> >> >>> wrote:
>> >> >>>
>> >> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0
>> should
>> >> >>> still
>> >> >>> > be
>> >> >>> > > included in the Fix versions,
>> >> >>> > > even if it was already released in 3.3.x. If I understand
>> >> correctly,
>> >> >>> this
>> >> >>> > > is usually our practice.
>> >> >>> >
>> >> >>> > +1.
>> >> >>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
>> >> >>> > maintaining both branch-3.4 and branch-3.3.
>> >> >>> >
>> >> >>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma <
>> >> tasan...@apache.org>
>> >> >>> > wrote:
>> >> >>> > >
>> >> >>> > > Hi Shilun,
>> >> >>> > >
>> >> >>> > > Thank you for leading the 3.4.0 release.
>> >> >>> > >
>> >> >>> > > > 2. In JIRA issues where other release versions have been
>> >> released,
>> >> >>> > remove
>> >> >>> > > > the 3.4.0 entry from the fix version.
>> >> >>> > >
>> >> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0
>> should
>> >> >>> still
>> >> >>> > be
>> >> >>> > > included in the Fix versions,
>> >> >>> > > even if it was already released in 3.3.x. If I understand
>> >> correctly,
>> >> >>> this
>> >> >>> > > is usually our practice.
>> >> >>> > >
>> >> >>> > > Thanks,
>> >> >>> > > - Takanobu
>> >> >>> > >
>> >> >>> > > 2024年1月21日(日) 13:56 slfan1989 :
>> >> >>> > >
>> >> >>> > > > Hi All,
>> >> >>> > > >
>> >> >>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and
>> need to
>> >> >>> > perform
>> >> >>> > > > some operations on the JIRA issues related to HADOOP, HDFS,
>> >> YARN,
>> >> >>> and
>> >> >>> > > > MAPREDUCE. Specifically, the tasks include:
>> >> >>> > > >
>> >> >>> > > > 1. For JIRA issues with target version set to 3.4.0 and
>> marked
>> >> as
>> >> >>> > > > non-blocker, update the target version to 3.5.0.
>> >> >>> > > >
>> >> >>> > > > 2. In JIRA issues where other release versions have been
>> >> released,
>> >> >>> > remove
>> >> >>> > > > the 3.4.0 entry from the fix version.
>> >> >>> > > >
>> >> >>> > > > 3. For JIRA issues with target version set to 3.4.0, fix
>> version
>> >> >>> set to
>> >> >>> > > > 3.4.0, and status set to
>> >> >>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags
>> >> are
>> >> >>> not
>> >> >>> > set,
>> >> >>> > > > complete the information.
>> >> >>> > > >
>> >> >>> > > > Best Regards,
>> >> >>> > > > - Shilun Fan.
>> >> >>> > > >
>> >> >>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He <
>> >> xq.he2...@gmail.com>
>> >> >>> > wrote:
>> >> >>> > > >
>> >> >>> > > > > Hi All,
>> >> >>> > > > >
>> >> >>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4
>>

Re: [INFO] Hadoop-3.4 Release Update

[Xiaoqiao He]

Hi all, the branch-3.4.0 codebase has been frozen. Hadoop-3.4.0-RC1 will be
ready for a while.
Thanks.

Best Regards,
- He Xiaoqiao


On Tue, Jan 23, 2024 at 7:58 AM slfan1989  wrote:

> branch-3.4/branch-3.4.0 was created from the trunk branch, including the
> changes from branch-3.3.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Jan 23, 2024 at 12:58 AM Brahma Reddy Battula 
> wrote:
>
> > Thanks for driving this.
> >
> > One query:
> > Does all the jiras from 3.3 are part of the 3.4?
> >
> >
> >
> >
> > On Mon, 22 Jan 2024 at 11:50 AM, slfan1989  wrote:
> >
> >> Thank you very much for specifying the frozen time for the
> >> Branch-3.4.0.  @Xiaoqiao
> >> He 
> >>
> >> If there is a need to backport any PRs to branch-3.4/branch-3.4.0,
> please
> >> feel free to contact me.
> >>
> >> Best Regards,
> >> Shilun Fan.
> >>
> >> On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He 
> >> wrote:
> >>
> >> > Hi All,
> >> >
> >> > Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
> >> > If there is any blocker/critical PR please backport to branch-3.4 and
> >> > branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.
> >> >
> >> > If any reported issues are not ready in time please wait for the next
> >> > release.
> >> >
> >> > Thanks all.
> >> >
> >> > Best Regards,
> >> > - He Xiaoqiao
> >> >
> >> >
> >> >
> >> > On Mon, Jan 22, 2024 at 10:43 AM slfan1989 
> >> wrote:
> >> >
> >> >> Thank you for the feedback!
> >> >>
> >> >> I apologize for updating partial information in the Hadoop module.
> >> >>
> >> >> Today, I will complete the rollback of the modified JIRA and add back
> >> the
> >> >> 3.4.0 version.
> >> >>
> >> >> Best
> >> >>
> >> >> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He 
> >> >> wrote:
> >> >>
> >> >>> Thanks all for your input. Connected to Shilun offline yesterday,
> and
> >> he
> >> >>> will
> >> >>> update or recover the fix version tag before 3.4.0-RC1.
> >> >>>
> >> >>> Best Regards,
> >> >>> - He Xiaoqiao
> >> >>>
> >> >>>
> >> >>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki <
> >> iwasak...@apache.org>
> >> >>> wrote:
> >> >>>
> >> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0
> should
> >> >>> still
> >> >>> > be
> >> >>> > > included in the Fix versions,
> >> >>> > > even if it was already released in 3.3.x. If I understand
> >> correctly,
> >> >>> this
> >> >>> > > is usually our practice.
> >> >>> >
> >> >>> > +1.
> >> >>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
> >> >>> > maintaining both branch-3.4 and branch-3.3.
> >> >>> >
> >> >>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma <
> >> tasan...@apache.org>
> >> >>> > wrote:
> >> >>> > >
> >> >>> > > Hi Shilun,
> >> >>> > >
> >> >>> > > Thank you for leading the 3.4.0 release.
> >> >>> > >
> >> >>> > > > 2. In JIRA issues where other release versions have been
> >> released,
> >> >>> > remove
> >> >>> > > > the 3.4.0 entry from the fix version.
> >> >>> > >
> >> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0
> should
> >> >>> still
> >> >>> > be
> >> >>> > > included in the Fix versions,
> >> >>> > > even if it was already released in 3.3.x. If I understand
> >> correctly,
> >> >>> this
> >> >>> > > is usually our practice.
> >> >>> > >
> >> >>> > > Thanks,
> >> >>> > > - Takanobu
> >> >>> > >
> >> >>> > > 2024年1月21日(日) 13:56 slfan1989 :
> >> >>> > >
> >> >>> > > > Hi All,
> >> >>> > > >
> >> >>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need
> to
> >> >>> > perform
> >> >>> > > > some operations on the JIRA issues related to HADOOP, HDFS,
> >> YARN,
> >> >>> and
> >> >>> > > > MAPREDUCE. Specifically, the tasks include:
> >> >>> > > >
> >> >>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked
> >> as
> >> >>> > > > non-blocker, update the target version to 3.5.0.
> >> >>> > > >
> >> >>> > > > 2. In JIRA issues where other release versions have been
> >> released,
> >> >>> > remove
> >> >>> > > > the 3.4.0 entry from the fix version.
> >> >>> > > >
> >> >>> > > > 3. For JIRA issues with target version set to 3.4.0, fix
> version
> >> >>> set to
> >> >>> > > > 3.4.0, and status set to
> >> >>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags
> >> are
> >> >>> not
> >> >>> > set,
> >> >>> > > > complete the information.
> >> >>> > > >
> >> >>> > > > Best Regards,
> >> >>> > > > - Shilun Fan.
> >> >>> > > >
> >> >>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He <
> >> xq.he2...@gmail.com>
> >> >>> > wrote:
> >> >>> > > >
> >> >>> > > > > Hi All,
> >> >>> > > > >
> >> >>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4
> is
> >> >>> > created[1].
> >> >>> > > > >
> >> >>> > > > > Please set the proper fix version while committing jira,
> 3.5.0
> >> >>> should
> >> >>> > > > > be priority. If there is any blocker/critical please also
> >> >>> backport to
> >> >>> > > > > branch-3.4 or sync to me or RM Shilun Fan.
> >> >>> > > > >
> >> >>> > > > > Shilun will start 3.4

Re: [INFO] Hadoop-3.4 Release Update

[slfan1989]

branch-3.4/branch-3.4.0 was created from the trunk branch, including the
changes from branch-3.3.

Best Regards,
Shilun Fan.

On Tue, Jan 23, 2024 at 12:58 AM Brahma Reddy Battula 
wrote:

> Thanks for driving this.
>
> One query:
> Does all the jiras from 3.3 are part of the 3.4?
>
>
>
>
> On Mon, 22 Jan 2024 at 11:50 AM, slfan1989  wrote:
>
>> Thank you very much for specifying the frozen time for the
>> Branch-3.4.0.  @Xiaoqiao
>> He 
>>
>> If there is a need to backport any PRs to branch-3.4/branch-3.4.0, please
>> feel free to contact me.
>>
>> Best Regards,
>> Shilun Fan.
>>
>> On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He 
>> wrote:
>>
>> > Hi All,
>> >
>> > Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
>> > If there is any blocker/critical PR please backport to branch-3.4 and
>> > branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.
>> >
>> > If any reported issues are not ready in time please wait for the next
>> > release.
>> >
>> > Thanks all.
>> >
>> > Best Regards,
>> > - He Xiaoqiao
>> >
>> >
>> >
>> > On Mon, Jan 22, 2024 at 10:43 AM slfan1989 
>> wrote:
>> >
>> >> Thank you for the feedback!
>> >>
>> >> I apologize for updating partial information in the Hadoop module.
>> >>
>> >> Today, I will complete the rollback of the modified JIRA and add back
>> the
>> >> 3.4.0 version.
>> >>
>> >> Best
>> >>
>> >> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He 
>> >> wrote:
>> >>
>> >>> Thanks all for your input. Connected to Shilun offline yesterday, and
>> he
>> >>> will
>> >>> update or recover the fix version tag before 3.4.0-RC1.
>> >>>
>> >>> Best Regards,
>> >>> - He Xiaoqiao
>> >>>
>> >>>
>> >>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki <
>> iwasak...@apache.org>
>> >>> wrote:
>> >>>
>> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
>> >>> still
>> >>> > be
>> >>> > > included in the Fix versions,
>> >>> > > even if it was already released in 3.3.x. If I understand
>> correctly,
>> >>> this
>> >>> > > is usually our practice.
>> >>> >
>> >>> > +1.
>> >>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
>> >>> > maintaining both branch-3.4 and branch-3.3.
>> >>> >
>> >>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma <
>> tasan...@apache.org>
>> >>> > wrote:
>> >>> > >
>> >>> > > Hi Shilun,
>> >>> > >
>> >>> > > Thank you for leading the 3.4.0 release.
>> >>> > >
>> >>> > > > 2. In JIRA issues where other release versions have been
>> released,
>> >>> > remove
>> >>> > > > the 3.4.0 entry from the fix version.
>> >>> > >
>> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
>> >>> still
>> >>> > be
>> >>> > > included in the Fix versions,
>> >>> > > even if it was already released in 3.3.x. If I understand
>> correctly,
>> >>> this
>> >>> > > is usually our practice.
>> >>> > >
>> >>> > > Thanks,
>> >>> > > - Takanobu
>> >>> > >
>> >>> > > 2024年1月21日(日) 13:56 slfan1989 :
>> >>> > >
>> >>> > > > Hi All,
>> >>> > > >
>> >>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
>> >>> > perform
>> >>> > > > some operations on the JIRA issues related to HADOOP, HDFS,
>> YARN,
>> >>> and
>> >>> > > > MAPREDUCE. Specifically, the tasks include:
>> >>> > > >
>> >>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked
>> as
>> >>> > > > non-blocker, update the target version to 3.5.0.
>> >>> > > >
>> >>> > > > 2. In JIRA issues where other release versions have been
>> released,
>> >>> > remove
>> >>> > > > the 3.4.0 entry from the fix version.
>> >>> > > >
>> >>> > > > 3. For JIRA issues with target version set to 3.4.0, fix version
>> >>> set to
>> >>> > > > 3.4.0, and status set to
>> >>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags
>> are
>> >>> not
>> >>> > set,
>> >>> > > > complete the information.
>> >>> > > >
>> >>> > > > Best Regards,
>> >>> > > > - Shilun Fan.
>> >>> > > >
>> >>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He <
>> xq.he2...@gmail.com>
>> >>> > wrote:
>> >>> > > >
>> >>> > > > > Hi All,
>> >>> > > > >
>> >>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
>> >>> > created[1].
>> >>> > > > >
>> >>> > > > > Please set the proper fix version while committing jira, 3.5.0
>> >>> should
>> >>> > > > > be priority. If there is any blocker/critical please also
>> >>> backport to
>> >>> > > > > branch-3.4 or sync to me or RM Shilun Fan.
>> >>> > > > >
>> >>> > > > > Shilun will start 3.4.0 RC1 voting in the next few days.
>> >>> > > > >
>> >>> > > > > Thanks.
>> >>> > > > >
>> >>> > > > > Best Regards,
>> >>> > > > > - He Xiaoqiao
>> >>> > > > >
>> >>> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
>> >>> > > > >
>> >>> > > >
>> >>> >
>> >>> >
>> -
>> >>> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
>> >>> > For additional commands, e-mail: private-h...@hadoop.apache.org
>> >>> >
>> >>> >
>> >>>
>> 

Re: [INFO] Hadoop-3.4 Release Update

[Brahma Reddy Battula]

Thanks for driving this.

One query:
Does all the jiras from 3.3 are part of the 3.4?




On Mon, 22 Jan 2024 at 11:50 AM, slfan1989  wrote:

> Thank you very much for specifying the frozen time for the
> Branch-3.4.0.  @Xiaoqiao
> He 
>
> If there is a need to backport any PRs to branch-3.4/branch-3.4.0, please
> feel free to contact me.
>
> Best Regards,
> Shilun Fan.
>
> On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He  wrote:
>
> > Hi All,
> >
> > Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
> > If there is any blocker/critical PR please backport to branch-3.4 and
> > branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.
> >
> > If any reported issues are not ready in time please wait for the next
> > release.
> >
> > Thanks all.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> >
> >
> > On Mon, Jan 22, 2024 at 10:43 AM slfan1989  wrote:
> >
> >> Thank you for the feedback!
> >>
> >> I apologize for updating partial information in the Hadoop module.
> >>
> >> Today, I will complete the rollback of the modified JIRA and add back
> the
> >> 3.4.0 version.
> >>
> >> Best
> >>
> >> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He 
> >> wrote:
> >>
> >>> Thanks all for your input. Connected to Shilun offline yesterday, and
> he
> >>> will
> >>> update or recover the fix version tag before 3.4.0-RC1.
> >>>
> >>> Best Regards,
> >>> - He Xiaoqiao
> >>>
> >>>
> >>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki  >
> >>> wrote:
> >>>
> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> >>> still
> >>> > be
> >>> > > included in the Fix versions,
> >>> > > even if it was already released in 3.3.x. If I understand
> correctly,
> >>> this
> >>> > > is usually our practice.
> >>> >
> >>> > +1.
> >>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
> >>> > maintaining both branch-3.4 and branch-3.3.
> >>> >
> >>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma <
> tasan...@apache.org>
> >>> > wrote:
> >>> > >
> >>> > > Hi Shilun,
> >>> > >
> >>> > > Thank you for leading the 3.4.0 release.
> >>> > >
> >>> > > > 2. In JIRA issues where other release versions have been
> released,
> >>> > remove
> >>> > > > the 3.4.0 entry from the fix version.
> >>> > >
> >>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> >>> still
> >>> > be
> >>> > > included in the Fix versions,
> >>> > > even if it was already released in 3.3.x. If I understand
> correctly,
> >>> this
> >>> > > is usually our practice.
> >>> > >
> >>> > > Thanks,
> >>> > > - Takanobu
> >>> > >
> >>> > > 2024年1月21日(日) 13:56 slfan1989 :
> >>> > >
> >>> > > > Hi All,
> >>> > > >
> >>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
> >>> > perform
> >>> > > > some operations on the JIRA issues related to HADOOP, HDFS, YARN,
> >>> and
> >>> > > > MAPREDUCE. Specifically, the tasks include:
> >>> > > >
> >>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked as
> >>> > > > non-blocker, update the target version to 3.5.0.
> >>> > > >
> >>> > > > 2. In JIRA issues where other release versions have been
> released,
> >>> > remove
> >>> > > > the 3.4.0 entry from the fix version.
> >>> > > >
> >>> > > > 3. For JIRA issues with target version set to 3.4.0, fix version
> >>> set to
> >>> > > > 3.4.0, and status set to
> >>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are
> >>> not
> >>> > set,
> >>> > > > complete the information.
> >>> > > >
> >>> > > > Best Regards,
> >>> > > > - Shilun Fan.
> >>> > > >
> >>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He  >
> >>> > wrote:
> >>> > > >
> >>> > > > > Hi All,
> >>> > > > >
> >>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
> >>> > created[1].
> >>> > > > >
> >>> > > > > Please set the proper fix version while committing jira, 3.5.0
> >>> should
> >>> > > > > be priority. If there is any blocker/critical please also
> >>> backport to
> >>> > > > > branch-3.4 or sync to me or RM Shilun Fan.
> >>> > > > >
> >>> > > > > Shilun will start 3.4.0 RC1 voting in the next few days.
> >>> > > > >
> >>> > > > > Thanks.
> >>> > > > >
> >>> > > > > Best Regards,
> >>> > > > > - He Xiaoqiao
> >>> > > > >
> >>> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
> >>> > > > >
> >>> > > >
> >>> >
> >>> > -
> >>> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
> >>> > For additional commands, e-mail: private-h...@hadoop.apache.org
> >>> >
> >>> >
> >>>
> >>
>

Re: [INFO] Hadoop-3.4 Release Update

[slfan1989]

Thank you very much for specifying the frozen time for the
Branch-3.4.0.  @Xiaoqiao
He 

If there is a need to backport any PRs to branch-3.4/branch-3.4.0, please
feel free to contact me.

Best Regards,
Shilun Fan.

On Mon, Jan 22, 2024 at 1:58 PM Xiaoqiao He  wrote:

> Hi All,
>
> Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
> If there is any blocker/critical PR please backport to branch-3.4 and
> branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.
>
> If any reported issues are not ready in time please wait for the next
> release.
>
> Thanks all.
>
> Best Regards,
> - He Xiaoqiao
>
>
>
> On Mon, Jan 22, 2024 at 10:43 AM slfan1989  wrote:
>
>> Thank you for the feedback!
>>
>> I apologize for updating partial information in the Hadoop module.
>>
>> Today, I will complete the rollback of the modified JIRA and add back the
>> 3.4.0 version.
>>
>> Best
>>
>> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He 
>> wrote:
>>
>>> Thanks all for your input. Connected to Shilun offline yesterday, and he
>>> will
>>> update or recover the fix version tag before 3.4.0-RC1.
>>>
>>> Best Regards,
>>> - He Xiaoqiao
>>>
>>>
>>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki 
>>> wrote:
>>>
>>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
>>> still
>>> > be
>>> > > included in the Fix versions,
>>> > > even if it was already released in 3.3.x. If I understand correctly,
>>> this
>>> > > is usually our practice.
>>> >
>>> > +1.
>>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
>>> > maintaining both branch-3.4 and branch-3.3.
>>> >
>>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma 
>>> > wrote:
>>> > >
>>> > > Hi Shilun,
>>> > >
>>> > > Thank you for leading the 3.4.0 release.
>>> > >
>>> > > > 2. In JIRA issues where other release versions have been released,
>>> > remove
>>> > > > the 3.4.0 entry from the fix version.
>>> > >
>>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
>>> still
>>> > be
>>> > > included in the Fix versions,
>>> > > even if it was already released in 3.3.x. If I understand correctly,
>>> this
>>> > > is usually our practice.
>>> > >
>>> > > Thanks,
>>> > > - Takanobu
>>> > >
>>> > > 2024年1月21日(日) 13:56 slfan1989 :
>>> > >
>>> > > > Hi All,
>>> > > >
>>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
>>> > perform
>>> > > > some operations on the JIRA issues related to HADOOP, HDFS, YARN,
>>> and
>>> > > > MAPREDUCE. Specifically, the tasks include:
>>> > > >
>>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked as
>>> > > > non-blocker, update the target version to 3.5.0.
>>> > > >
>>> > > > 2. In JIRA issues where other release versions have been released,
>>> > remove
>>> > > > the 3.4.0 entry from the fix version.
>>> > > >
>>> > > > 3. For JIRA issues with target version set to 3.4.0, fix version
>>> set to
>>> > > > 3.4.0, and status set to
>>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are
>>> not
>>> > set,
>>> > > > complete the information.
>>> > > >
>>> > > > Best Regards,
>>> > > > - Shilun Fan.
>>> > > >
>>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He 
>>> > wrote:
>>> > > >
>>> > > > > Hi All,
>>> > > > >
>>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
>>> > created[1].
>>> > > > >
>>> > > > > Please set the proper fix version while committing jira, 3.5.0
>>> should
>>> > > > > be priority. If there is any blocker/critical please also
>>> backport to
>>> > > > > branch-3.4 or sync to me or RM Shilun Fan.
>>> > > > >
>>> > > > > Shilun will start 3.4.0 RC1 voting in the next few days.
>>> > > > >
>>> > > > > Thanks.
>>> > > > >
>>> > > > > Best Regards,
>>> > > > > - He Xiaoqiao
>>> > > > >
>>> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
>>> > > > >
>>> > > >
>>> >
>>> > -
>>> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
>>> > For additional commands, e-mail: private-h...@hadoop.apache.org
>>> >
>>> >
>>>
>>

Re: [INFO] Hadoop-3.4 Release Update

[Xiaoqiao He]

Hi All,

Branch-3.4.0 will be frozen this Thursday (UTC 00:00 Jan 25, 2024).
If there is any blocker/critical PR please backport to branch-3.4 and
branch-3.4.0 or sync to me or hadoop-3.4.0 RM Shilun Fan.

If any reported issues are not ready in time please wait for the next
release.

Thanks all.

Best Regards,
- He Xiaoqiao



On Mon, Jan 22, 2024 at 10:43 AM slfan1989  wrote:

> Thank you for the feedback!
>
> I apologize for updating partial information in the Hadoop module.
>
> Today, I will complete the rollback of the modified JIRA and add back the
> 3.4.0 version.
>
> Best
>
> On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He 
> wrote:
>
>> Thanks all for your input. Connected to Shilun offline yesterday, and he
>> will
>> update or recover the fix version tag before 3.4.0-RC1.
>>
>> Best Regards,
>> - He Xiaoqiao
>>
>>
>> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki 
>> wrote:
>>
>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
>> still
>> > be
>> > > included in the Fix versions,
>> > > even if it was already released in 3.3.x. If I understand correctly,
>> this
>> > > is usually our practice.
>> >
>> > +1.
>> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
>> > maintaining both branch-3.4 and branch-3.3.
>> >
>> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma 
>> > wrote:
>> > >
>> > > Hi Shilun,
>> > >
>> > > Thank you for leading the 3.4.0 release.
>> > >
>> > > > 2. In JIRA issues where other release versions have been released,
>> > remove
>> > > > the 3.4.0 entry from the fix version.
>> > >
>> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
>> still
>> > be
>> > > included in the Fix versions,
>> > > even if it was already released in 3.3.x. If I understand correctly,
>> this
>> > > is usually our practice.
>> > >
>> > > Thanks,
>> > > - Takanobu
>> > >
>> > > 2024年1月21日(日) 13:56 slfan1989 :
>> > >
>> > > > Hi All,
>> > > >
>> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
>> > perform
>> > > > some operations on the JIRA issues related to HADOOP, HDFS, YARN,
>> and
>> > > > MAPREDUCE. Specifically, the tasks include:
>> > > >
>> > > > 1. For JIRA issues with target version set to 3.4.0 and marked as
>> > > > non-blocker, update the target version to 3.5.0.
>> > > >
>> > > > 2. In JIRA issues where other release versions have been released,
>> > remove
>> > > > the 3.4.0 entry from the fix version.
>> > > >
>> > > > 3. For JIRA issues with target version set to 3.4.0, fix version
>> set to
>> > > > 3.4.0, and status set to
>> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are
>> not
>> > set,
>> > > > complete the information.
>> > > >
>> > > > Best Regards,
>> > > > - Shilun Fan.
>> > > >
>> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He 
>> > wrote:
>> > > >
>> > > > > Hi All,
>> > > > >
>> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
>> > created[1].
>> > > > >
>> > > > > Please set the proper fix version while committing jira, 3.5.0
>> should
>> > > > > be priority. If there is any blocker/critical please also
>> backport to
>> > > > > branch-3.4 or sync to me or RM Shilun Fan.
>> > > > >
>> > > > > Shilun will start 3.4.0 RC1 voting in the next few days.
>> > > > >
>> > > > > Thanks.
>> > > > >
>> > > > > Best Regards,
>> > > > > - He Xiaoqiao
>> > > > >
>> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
>> > > > >
>> > > >
>> >
>> > -
>> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
>> > For additional commands, e-mail: private-h...@hadoop.apache.org
>> >
>> >
>>
>

Re: [INFO] Hadoop-3.4 Release Update

[slfan1989]

Thank you for the feedback!

I apologize for updating partial information in the Hadoop module.

Today, I will complete the rollback of the modified JIRA and add back the
3.4.0 version.

Best

On Mon, Jan 22, 2024 at 10:38 AM Xiaoqiao He  wrote:

> Thanks all for your input. Connected to Shilun offline yesterday, and he
> will
> update or recover the fix version tag before 3.4.0-RC1.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki 
> wrote:
>
> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> still
> > be
> > > included in the Fix versions,
> > > even if it was already released in 3.3.x. If I understand correctly,
> this
> > > is usually our practice.
> >
> > +1.
> > We can not assume 3.4.x contains all fixes of 3.3.y since we are
> > maintaining both branch-3.4 and branch-3.3.
> >
> > On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma 
> > wrote:
> > >
> > > Hi Shilun,
> > >
> > > Thank you for leading the 3.4.0 release.
> > >
> > > > 2. In JIRA issues where other release versions have been released,
> > remove
> > > > the 3.4.0 entry from the fix version.
> > >
> > > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should
> still
> > be
> > > included in the Fix versions,
> > > even if it was already released in 3.3.x. If I understand correctly,
> this
> > > is usually our practice.
> > >
> > > Thanks,
> > > - Takanobu
> > >
> > > 2024年1月21日(日) 13:56 slfan1989 :
> > >
> > > > Hi All,
> > > >
> > > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
> > perform
> > > > some operations on the JIRA issues related to HADOOP, HDFS, YARN, and
> > > > MAPREDUCE. Specifically, the tasks include:
> > > >
> > > > 1. For JIRA issues with target version set to 3.4.0 and marked as
> > > > non-blocker, update the target version to 3.5.0.
> > > >
> > > > 2. In JIRA issues where other release versions have been released,
> > remove
> > > > the 3.4.0 entry from the fix version.
> > > >
> > > > 3. For JIRA issues with target version set to 3.4.0, fix version set
> to
> > > > 3.4.0, and status set to
> > > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are not
> > set,
> > > > complete the information.
> > > >
> > > > Best Regards,
> > > > - Shilun Fan.
> > > >
> > > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He 
> > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
> > created[1].
> > > > >
> > > > > Please set the proper fix version while committing jira, 3.5.0
> should
> > > > > be priority. If there is any blocker/critical please also backport
> to
> > > > > branch-3.4 or sync to me or RM Shilun Fan.
> > > > >
> > > > > Shilun will start 3.4.0 RC1 voting in the next few days.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Best Regards,
> > > > > - He Xiaoqiao
> > > > >
> > > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
> > > > >
> > > >
> >
> > -
> > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: private-h...@hadoop.apache.org
> >
> >
>

Re: [INFO] Hadoop-3.4 Release Update

[Xiaoqiao He]

Thanks all for your input. Connected to Shilun offline yesterday, and he
will
update or recover the fix version tag before 3.4.0-RC1.

Best Regards,
- He Xiaoqiao


On Mon, Jan 22, 2024 at 9:53 AM Masatake Iwasaki 
wrote:

> > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should still
> be
> > included in the Fix versions,
> > even if it was already released in 3.3.x. If I understand correctly, this
> > is usually our practice.
>
> +1.
> We can not assume 3.4.x contains all fixes of 3.3.y since we are
> maintaining both branch-3.4 and branch-3.3.
>
> On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma 
> wrote:
> >
> > Hi Shilun,
> >
> > Thank you for leading the 3.4.0 release.
> >
> > > 2. In JIRA issues where other release versions have been released,
> remove
> > > the 3.4.0 entry from the fix version.
> >
> > As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should still
> be
> > included in the Fix versions,
> > even if it was already released in 3.3.x. If I understand correctly, this
> > is usually our practice.
> >
> > Thanks,
> > - Takanobu
> >
> > 2024年1月21日(日) 13:56 slfan1989 :
> >
> > > Hi All,
> > >
> > > I am currently preparing to release hadoop-3.4.0-RC1 and need to
> perform
> > > some operations on the JIRA issues related to HADOOP, HDFS, YARN, and
> > > MAPREDUCE. Specifically, the tasks include:
> > >
> > > 1. For JIRA issues with target version set to 3.4.0 and marked as
> > > non-blocker, update the target version to 3.5.0.
> > >
> > > 2. In JIRA issues where other release versions have been released,
> remove
> > > the 3.4.0 entry from the fix version.
> > >
> > > 3. For JIRA issues with target version set to 3.4.0, fix version set to
> > > 3.4.0, and status set to
> > > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are not
> set,
> > > complete the information.
> > >
> > > Best Regards,
> > > - Shilun Fan.
> > >
> > > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He 
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is
> created[1].
> > > >
> > > > Please set the proper fix version while committing jira, 3.5.0 should
> > > > be priority. If there is any blocker/critical please also backport to
> > > > branch-3.4 or sync to me or RM Shilun Fan.
> > > >
> > > > Shilun will start 3.4.0 RC1 voting in the next few days.
> > > >
> > > > Thanks.
> > > >
> > > > Best Regards,
> > > > - He Xiaoqiao
> > > >
> > > > [1] https://github.com/apache/hadoop/commits/branch-3.4
> > > >
> > >
>
> -
> To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: private-h...@hadoop.apache.org
>
>

Re: [INFO] Hadoop-3.4 Release Update

[Masatake Iwasaki]

> As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should still be
> included in the Fix versions,
> even if it was already released in 3.3.x. If I understand correctly, this
> is usually our practice.

+1.
We can not assume 3.4.x contains all fixes of 3.3.y since we are
maintaining both branch-3.4 and branch-3.3.

On Sun, Jan 21, 2024 at 2:45 PM Takanobu Asanuma  wrote:
>
> Hi Shilun,
>
> Thank you for leading the 3.4.0 release.
>
> > 2. In JIRA issues where other release versions have been released, remove
> > the 3.4.0 entry from the fix version.
>
> As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should still be
> included in the Fix versions,
> even if it was already released in 3.3.x. If I understand correctly, this
> is usually our practice.
>
> Thanks,
> - Takanobu
>
> 2024年1月21日(日) 13:56 slfan1989 :
>
> > Hi All,
> >
> > I am currently preparing to release hadoop-3.4.0-RC1 and need to perform
> > some operations on the JIRA issues related to HADOOP, HDFS, YARN, and
> > MAPREDUCE. Specifically, the tasks include:
> >
> > 1. For JIRA issues with target version set to 3.4.0 and marked as
> > non-blocker, update the target version to 3.5.0.
> >
> > 2. In JIRA issues where other release versions have been released, remove
> > the 3.4.0 entry from the fix version.
> >
> > 3. For JIRA issues with target version set to 3.4.0, fix version set to
> > 3.4.0, and status set to
> > RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are not set,
> > complete the information.
> >
> > Best Regards,
> > - Shilun Fan.
> >
> > On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He  wrote:
> >
> > > Hi All,
> > >
> > > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is created[1].
> > >
> > > Please set the proper fix version while committing jira, 3.5.0 should
> > > be priority. If there is any blocker/critical please also backport to
> > > branch-3.4 or sync to me or RM Shilun Fan.
> > >
> > > Shilun will start 3.4.0 RC1 voting in the next few days.
> > >
> > > Thanks.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > > [1] https://github.com/apache/hadoop/commits/branch-3.4
> > >
> >

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [INFO] Hadoop-3.4 Release Update

[Takanobu Asanuma]

Hi Shilun,

Thank you for leading the 3.4.0 release.

> 2. In JIRA issues where other release versions have been released, remove
> the 3.4.0 entry from the fix version.

As I mentioned in my comment on HADOOP-18045, I think 3.4.0 should still be
included in the Fix versions,
even if it was already released in 3.3.x. If I understand correctly, this
is usually our practice.

Thanks,
- Takanobu

2024年1月21日(日) 13:56 slfan1989 :

> Hi All,
>
> I am currently preparing to release hadoop-3.4.0-RC1 and need to perform
> some operations on the JIRA issues related to HADOOP, HDFS, YARN, and
> MAPREDUCE. Specifically, the tasks include:
>
> 1. For JIRA issues with target version set to 3.4.0 and marked as
> non-blocker, update the target version to 3.5.0.
>
> 2. In JIRA issues where other release versions have been released, remove
> the 3.4.0 entry from the fix version.
>
> 3. For JIRA issues with target version set to 3.4.0, fix version set to
> 3.4.0, and status set to
> RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are not set,
> complete the information.
>
> Best Regards,
> - Shilun Fan.
>
> On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He  wrote:
>
> > Hi All,
> >
> > Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is created[1].
> >
> > Please set the proper fix version while committing jira, 3.5.0 should
> > be priority. If there is any blocker/critical please also backport to
> > branch-3.4 or sync to me or RM Shilun Fan.
> >
> > Shilun will start 3.4.0 RC1 voting in the next few days.
> >
> > Thanks.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > [1] https://github.com/apache/hadoop/commits/branch-3.4
> >
>

Re: [INFO] Hadoop-3.4 Release Update

[slfan1989]

Hi All,

I am currently preparing to release hadoop-3.4.0-RC1 and need to perform
some operations on the JIRA issues related to HADOOP, HDFS, YARN, and
MAPREDUCE. Specifically, the tasks include:

1. For JIRA issues with target version set to 3.4.0 and marked as
non-blocker, update the target version to 3.5.0.

2. In JIRA issues where other release versions have been released, remove
the 3.4.0 entry from the fix version.

3. For JIRA issues with target version set to 3.4.0, fix version set to
3.4.0, and status set to
RESOLVED, if Affects Version/s, Component/s, and Hadoop Flags are not set,
complete the information.

Best Regards,
- Shilun Fan.

On Fri, Jan 19, 2024 at 3:23 PM Xiaoqiao He  wrote:

> Hi All,
>
> Branch trunk has been set to 3.5.0-SNAPSHOT and branch-3.4 is created[1].
>
> Please set the proper fix version while committing jira, 3.5.0 should
> be priority. If there is any blocker/critical please also backport to
> branch-3.4 or sync to me or RM Shilun Fan.
>
> Shilun will start 3.4.0 RC1 voting in the next few days.
>
> Thanks.
>
> Best Regards,
> - He Xiaoqiao
>
> [1] https://github.com/apache/hadoop/commits/branch-3.4
>

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Xiaoqiao He]

Hi Shilun,

Thanks for your great work!

Compare branch-3.4 and branch-3.4.0, I found that commit [1]
is only checked in branch-3.4.0 but not branch-3.4. Please double
check if we need to backport branch-3.4.

Moreover, before 3.4.0 release we should cherry-pick from trunk
to branch-3.4 first if necessary, then to branch-3.4.0.

Thanks again.

Best Regards,
- He Xiaoqiao

[1]
https://github.com/apache/hadoop/commit/5e30d28d7e524dd3eed179d81c38e2eb82ae4673

On Wed, Jan 17, 2024 at 9:19 AM slfan1989  wrote:

> Hi, all
>
> Since the hadoop-3.4.0-RC0 vote, I have received valuable feedback. I
> encountered some issues during the preparation of hadoop-3.4.0-RC0, and I
> will address these issues in hadoop-3.4.0-RC1.
>
> The voting for RC0 will be closed. After the release of RC1, I will invite
> members of the community to review and vote once again.
>
> Thank you all once again for your support!
>
> Best Regards,
> Shilun Fan.
>
> On Wed, Jan 17, 2024 at 9:03 AM slfan1989  wrote:
>
> > Thank you very much for the response!
> >
> > The content is very comprehensive and valuable.
> >
> > I will prepare hadoop-3.4.0-RC1 according to the instructions provided by
> > you, and after RC1 is packaged, I will use
> validate-hadoop-client-artifacts
> > for validation.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Jan 16, 2024 at 12:34 AM Steve Loughran
> >  wrote:
> >
> >> -1 I'm afraid, just due to staging/packaging issues.
> >>
> >> This took me a few goes to get right myself, so nothing unusual.
> >>
> >> Note I used my validator project which is set to retrieve binaries,
> check
> >> signatures, run maven builds against staged artifacts *and clean up any
> >> local copies first*and more.
> >>
> >> This uses apache ant to manage all this:
> >>
> >> https://github.com/steveloughran/validate-hadoop-client-artifacts
> >>
> >> Here's the initial build.properties:file I used to try and manage this
> >>
> >> ## build.properties:
> >> hadoop.version=3.4.0
> >> rc=RC0
> >> amd.src.dir=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/
> >> http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64
> >> <
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64
> >
> >>
> >> release=hadoop-${hadoop.version}-RC0
> >> rc.dirname=${release}
> >> release.native.binaries=false
> >> git.commit.id=cdb8af4f22ec
> >> nexus.staging.url=
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1391/
> >> hadoop.source.dir=${local.dev.dir}/hadoop-trunk
> >> ##
> >>
> >> When I did my own builds, all the artifacts created were without the RC0
> >> suffix. It is critical this happens because the .sha512 checksums
> include
> >> that in their paths
> >>
> >> > cat hadoop-3.4.0-RC0.tar.gz.sha512
> >> SHA512 (hadoop-3.4.0-RC0.tar.gz) =
> >>
> >>
> e50e68aecb36867c610db8309ccd3aae812184da21354b50d2a461b29c73f21d097fb27372c73c150e1c035003bb99a61c64db26c090fe0fb9e7ed6041722eab
> >>
> >>
> >> Maven artifacts: staging problems
> >>
> >> Couldn't build with a -Pstaging profile as the staging repository wasn't
> >> yet closed -I tried to do that myself.
> >>
> >> This failed with some rule problem
> >>
> >> Event: Failed: Checksum Validation
> >> Monday, January 15, 2024 14:37:13 GMT (GMT+)
> >> typeId checksum-staging
> >> failureMessage INVALID SHA-1:
> >>
> >>
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1'
> >> failureMessage Requires one-of SHA-1:
> >>
> >>
> /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1,
> >> SHA-256:
> >>
> >>
> /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha256,
> >> SHA-512:
> >>
> >>
> /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha512
> >>
> >> I don't know precisely what this means...my guess is that the upload
> >> didn't
> >> include everything.
> >>
> >> Note my client-validator module can check this; just run its maven test
> >> commands
> >>
> >> mvn clean test -U -P3.4 -Pstaging
> >>
> >> GPG signing: all good.
> >>
> >> Picked your key up from the site ( ant gpg.keys ) ... first validation
> >> with
> >> ant gpg.verify was unhappy as your key wasn't trusted. I've signed it
> and
> >> pushed that signature up, so people who trust me get some reassurance
> >> about
> >> you.
> >>
> >> My build then failed as the gpg code couldn't find the
> >> hadoop-3.4.0-aarch64.tar.gz.asc
> >>
> >> The problem here is that although we want separate arm and x86 tar
> files,
> >> we don't really want separate binaries as it only creates different jars
> >> in
> >> the wild.
> >>
> >> The way I addressed that was after creating that x86 release on an ec2
> vm
> >> and downloading it, I then did a local arm64 build and then created an
> arm
> >

Re: Fw:Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[slfan1989]

Thank you very much for the response!

The content is very comprehensive and valuable.

I will prepare hadoop-3.4.0-RC1 according to the instructions provided by
you, and after RC1 is packaged, I will use validate-hadoop-client-artifacts
for validation.

Best Regards,
Shilun Fan.

On Tue, Jan 16, 2024 at 12:34 AM Steve Loughran 
wrote:

> -1 I'm afraid, just due to staging/packaging issues.
>
> This took me a few goes to get right myself, so nothing unusual.
>
> Note I used my validator project which is set to retrieve binaries, check
> signatures, run maven builds against staged artifacts *and clean up any
> local copies first*and more.
>
> This uses apache ant to manage all this:
>
> https://github.com/steveloughran/validate-hadoop-client-artifacts
>
> Here's the initial build.properties:file I used to try and manage this
>
> ## build.properties:
> hadoop.version=3.4.0
> rc=RC0
> amd.src.dir=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/
> http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64
> 
>
> release=hadoop-${hadoop.version}-RC0
> rc.dirname=${release}
> release.native.binaries=false
> git.commit.id=cdb8af4f22ec
> nexus.staging.url=
> https://repository.apache.org/content/repositories/orgapachehadoop-1391/
> hadoop.source.dir=${local.dev.dir}/hadoop-trunk
> ##
>
> When I did my own builds, all the artifacts created were without the RC0
> suffix. It is critical this happens because the .sha512 checksums include
> that in their paths
>
> > cat hadoop-3.4.0-RC0.tar.gz.sha512
> SHA512 (hadoop-3.4.0-RC0.tar.gz) =
>
> e50e68aecb36867c610db8309ccd3aae812184da21354b50d2a461b29c73f21d097fb27372c73c150e1c035003bb99a61c64db26c090fe0fb9e7ed6041722eab
>
>
> Maven artifacts: staging problems
>
> Couldn't build with a -Pstaging profile as the staging repository wasn't
> yet closed -I tried to do that myself.
>
> This failed with some rule problem
>
> Event: Failed: Checksum Validation
> Monday, January 15, 2024 14:37:13 GMT (GMT+)
> typeId checksum-staging
> failureMessage INVALID SHA-1:
>
> '/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1'
> failureMessage Requires one-of SHA-1:
>
> /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1,
> SHA-256:
>
> /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha256,
> SHA-512:
>
> /org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha512
>
> I don't know precisely what this means...my guess is that the upload didn't
> include everything.
>
> Note my client-validator module can check this; just run its maven test
> commands
>
> mvn clean test -U -P3.4 -Pstaging
>
> GPG signing: all good.
>
> Picked your key up from the site ( ant gpg.keys ) ... first validation with
> ant gpg.verify was unhappy as your key wasn't trusted. I've signed it and
> pushed that signature up, so people who trust me get some reassurance about
> you.
>
> My build then failed as the gpg code couldn't find the
> hadoop-3.4.0-aarch64.tar.gz.asc
>
> The problem here is that although we want separate arm and x86 tar files,
> we don't really want separate binaries as it only creates different jars in
> the wild.
>
> The way I addressed that was after creating that x86 release on an ec2 vm
> and downloading it, I then did a local arm64 build and then created an arm
> .tar.gz file, copied it into the same dir as the amd66 binaries but with
> the arm64 .tar.gz filename, .asc and .sha512 checksum files all renamed
> (checksum file patches to match the name).
>
>
> https://github.com/steveloughran/validate-hadoop-client-artifacts?tab=readme-ov-file#arm64-binaries
>

Re: Fw:Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Steve Loughran]

-1 I'm afraid, just due to staging/packaging issues.

This took me a few goes to get right myself, so nothing unusual.

Note I used my validator project which is set to retrieve binaries, check
signatures, run maven builds against staged artifacts *and clean up any
local copies first*and more.

This uses apache ant to manage all this:

https://github.com/steveloughran/validate-hadoop-client-artifacts

Here's the initial build.properties:file I used to try and manage this

## build.properties:
hadoop.version=3.4.0
rc=RC0
amd.src.dir=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/
http.source=https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64

release=hadoop-${hadoop.version}-RC0
rc.dirname=${release}
release.native.binaries=false
git.commit.id=cdb8af4f22ec
nexus.staging.url=
https://repository.apache.org/content/repositories/orgapachehadoop-1391/
hadoop.source.dir=${local.dev.dir}/hadoop-trunk
##

When I did my own builds, all the artifacts created were without the RC0
suffix. It is critical this happens because the .sha512 checksums include
that in their paths

> cat hadoop-3.4.0-RC0.tar.gz.sha512
SHA512 (hadoop-3.4.0-RC0.tar.gz) =
e50e68aecb36867c610db8309ccd3aae812184da21354b50d2a461b29c73f21d097fb27372c73c150e1c035003bb99a61c64db26c090fe0fb9e7ed6041722eab


Maven artifacts: staging problems

Couldn't build with a -Pstaging profile as the staging repository wasn't
yet closed -I tried to do that myself.

This failed with some rule problem

Event: Failed: Checksum Validation
Monday, January 15, 2024 14:37:13 GMT (GMT+)
typeId checksum-staging
failureMessage INVALID SHA-1:
'/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1'
failureMessage Requires one-of SHA-1:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha1,
SHA-256:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha256,
SHA-512:
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/3.4.0/hadoop-mapreduce-client-jobclient-3.4.0-tests.jar.sha512

I don't know precisely what this means...my guess is that the upload didn't
include everything.

Note my client-validator module can check this; just run its maven test
commands

mvn clean test -U -P3.4 -Pstaging

GPG signing: all good.

Picked your key up from the site ( ant gpg.keys ) ... first validation with
ant gpg.verify was unhappy as your key wasn't trusted. I've signed it and
pushed that signature up, so people who trust me get some reassurance about
you.

My build then failed as the gpg code couldn't find the
hadoop-3.4.0-aarch64.tar.gz.asc

The problem here is that although we want separate arm and x86 tar files,
we don't really want separate binaries as it only creates different jars in
the wild.

The way I addressed that was after creating that x86 release on an ec2 vm
and downloading it, I then did a local arm64 build and then created an arm
.tar.gz file, copied it into the same dir as the amd66 binaries but with
the arm64 .tar.gz filename, .asc and .sha512 checksum files all renamed
(checksum file patches to match the name).

https://github.com/steveloughran/validate-hadoop-client-artifacts?tab=readme-ov-file#arm64-binaries

Re: [DISCUSS] Release Hadoop 3.4.0

[slfan1989]

Dear Community Members,

While discussing release Hadoop 3.4.0, we have also decided to release
Hadoop Thirdparty version 1.2.0.

I'll be initiating some branch preparation work over the weekend.

Best Regards,
Shilun Fan.

-- Forwarded message -
发件人： slfan1989 
Date: 2024/01/13 08:08:36
Subject: Re: Re: [DISCUSS] Release Hadoop 3.4.0
To: , , <
ste...@cloudera.com>
Cc: , Ayush Saxena , <
hexiaoq...@apache.org>


Thank you for reporting this issue and helping resolve it！

I will try to release hadoop-thirdparty 1.2.0-RC0 version first.

Best Regards,
Shilun Fan.

Original

From:"PJ Fanning"< fannin...@apache.org >;

Date:2024/1/9 1:49

To:"common-dev"< common-...@hadoop.apache.org >;

Subject:Re: Re: [DISCUSS] Release Hadoop 3.4.0

It looks like the Jackson upgrade [1] needs more investigation which I will
need to come back to in a few weeks.

I would still like to get hadoop-thirdparty 1.2.0 released so that
protobuf-java, avro and guava can be upgraded for hadoop 3.4.0.


[1] https://github.com/apache/hadoop/pull/6370

On 2024/01/04 15:53:43 PJ Fanning wrote:
> I would like to get some dependencies upgraded for Hadoop 3.4.0. For me,
it would be good to upgrade protobuf-java and Jackson to more secure
versions.
>
> For protobuf-java, that would involve releasing hadoop-thirdparty 1.2.0
[1], including merging the protobuf upgrade [2].
>
> For Jackson, we are hampered by Jackson dropping support for
Jersey/JAX-RS 1.x. I have a workaround for this that I think is worth
further investigating [3].
>
> [1] https://github.com/apache/hadoop-thirdparty
> [2] https://github.com/apache/hadoop-thirdparty/pull/19
> [3] https://github.com/apache/hadoop/pull/6370
>
>
>
> On 2024/01/04 14:26:37 slfan1989 wrote:
> > Hey all,
> >
> > We are planning to release Hadoop 3.4.0 base on trunk. I made some
> > preparations and changed the target version of JIRA for non-blockers in
> > HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to
create
> > a new JIRA, the target version can directly select version 3.5.0.
> >
> > If you have any thoughts, suggestions, or concerns, please feel free to
> > share them.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > > +1 from me.
> > >> It will include the new AWS V2 SDK upgrade as well.
> >
> > > On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He wrote:
> >
> > > >
> > > > I think the release discussion can be in public ML?
> > >
> > > Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > > On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote:
> > >
> > > > +1 from me as well.
> > > >
> > > > We should definitely attempt to upgrade the thirdparty version for
> > > > 3.4.0 & check if there are any pending critical/blocker issues as
> > > > well.
> > > >
> > > > I think the release discussion can be in public ML?
> > > >
> > > > -Ayush
> > > >
> > > > On Mon, 1 Jan 2024 at 18:25, Steve Loughran
 > > >
> > > > wrote:
> > > > >
> > > > > +1 from me
> > > > >
> > > > > ant and maven repo to build and validate things, including making
arm
> > > > > binaries if you work from an arm macbook.
> > > > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > > > >
> > > > > do we need to publish an up to date thirdparty release for this?
> > > > >
> > > > >
> > > > >
> > > > > On Mon, 25 Dec 2023 at 16:06, slfan1989 wrote:
> > > > >
> > > > > > Dear PMC Members,
> > > > > >
> > > > > > First of all, Merry Christmas to everyone!
> > > > > >
> > > > > > In our community discussions, we collectively finalized the
plan to
> > > > release
> > > > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to
> > take
> > > > on
> > > > > > the responsibility for the initial release of version 3.4.0,
and the
> > > > entire
> > > > > > process is set to officially commence in January 2024.
> > > > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0.
> > > > > >
> > > > > > The specific work plan includes:
> > > > > >
> > > > > > 1. Following the guidance in the HowToRelease d

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[slfan1989]

Thank you for reporting this issue and I will follow up on HDFS-17129.

I acknowledge that I haven't checked for those marked as critical/blocker
in RC0. I intend to complete this check before the release of RC1.

Thank you again for your valuable suggestions!

Best Regards
Shilun Fan.

Ayush Saxena  于2024年1月13日周六 05:10写道：

> We should consider including
> https://issues.apache.org/jira/browse/HDFS-17129
>
> Which looks like inducing some misorder between IBR & FBR which can
> potentially lead to strange issues, if that can’t be merged, should revert
> the one which causes that.
>
> I think we should check for any ticket which has a target version or
> affect version & is marked critica/blockerl for 3.4.0 before spinning up a
> new RC, I think I mentioned that somewhere before.
>
> -1, in case HDFS-17129 is not a false alarm or we can prove it won't cause
> any issues. There is a comment which says a block was reported missing post
> the patch that induced it: [1]
>
> [1] https://github.com/apache/hadoop/pull/6244#issuecomment-1793981740
>
> -Ayush
>
>
> On Fri, 12 Jan 2024 at 07:37, slfan1989  wrote:
>
>> Thank you very much for your help in verifying this version! We will use
>> version 3.5.0 for fix jira in the future.
>>
>> Best Regards,
>> Shilun Fan.
>>
>>  > wonderful! I'll be testing over the weekend
>>
>>  > Meanwhile, new changes I'm putting in to trunk are tagged as fixed in
>> 3.5.0
>>  > -correct?
>>
>>  > steve
>>
>>
>> > On Thu, 11 Jan 2024 at 05:15, slfan1989 wrote:
>>
>> > Hello all,
>> >
>> > We plan to release hadoop 3.4.0 based on hadoop trunk, which is the
>> first
>> > hadoop 3.4.0-RC version.
>> >
>> > The RC is available at:
>> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64)
>> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64)
>> >
>> > Maven artifacts is built by x86 machine and are staged at
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1391/
>> >
>> > My public key:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > Changelog:
>> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md
>> >
>> > Release notes:
>> >
>> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md
>> >
>> > This is a relatively big release (by Hadoop standard) containing about
>> 2852
>> > commits.
>> >
>> > Please give it a try, this RC vote will run for 7 days.
>> >
>> > Feature highlights:
>> >
>> > DataNode FsDatasetImpl Fine-Grained Locking via BlockPool
>> > 
>> > [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split
>> > FsDatasetImpl datasetLock via blockpool to solve the issue of heavy
>> > FsDatasetImpl datasetLock
>> > When there are many namespaces in a large cluster.
>> >
>> > YARN Federation improvements
>> > 
>> > [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings
>> many
>> > improvements, including the following:
>> >
>> > 1. YARN Router now boasts a full implementation of all relevant
>> interfaces
>> > including the ApplicationClientProtocol,
>> > ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
>> > 2. Enhanced support for Application cleanup and automatic offline
>> > mechanisms for SubCluster are now facilitated by the YARN Router.
>> > 3. Code optimization for Router and AMRMProxy was undertaken, coupled
>> with
>> > improvements to previously pending functionalities.
>> > 4. Audit logs and Metrics for Router received upgrades.
>> > 5. A boost in cluster security features was achieved, with the inclusion
>> of
>> > Kerberos support.
>> > 6. The page function of the router has been enhanced.
>> >
>> > Upgrade AWS SDK to V2
>> > 
>> > [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073)
>> > The S3A connector now uses the V2 AWS SDK. This is a significant change
>> at
>> > the source code level.
>> > Any applications using the internal extension/override points in the
>> > filesystem connector are likely to break.
>> > Consult the document aws\_sdk\_upgrade for the full details.
>> >
>> > hadoop-thirdparty will also provide the new RC0 soon.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>>
>

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Ayush Saxena]

We should consider including
https://issues.apache.org/jira/browse/HDFS-17129

Which looks like inducing some misorder between IBR & FBR which can
potentially lead to strange issues, if that can’t be merged, should revert
the one which causes that.

I think we should check for any ticket which has a target version or
affect version & is marked critica/blockerl for 3.4.0 before spinning up a
new RC, I think I mentioned that somewhere before.

-1, in case HDFS-17129 is not a false alarm or we can prove it won't cause
any issues. There is a comment which says a block was reported missing post
the patch that induced it: [1]

[1] https://github.com/apache/hadoop/pull/6244#issuecomment-1793981740

-Ayush


On Fri, 12 Jan 2024 at 07:37, slfan1989  wrote:

> Thank you very much for your help in verifying this version! We will use
> version 3.5.0 for fix jira in the future.
>
> Best Regards,
> Shilun Fan.
>
>  > wonderful! I'll be testing over the weekend
>
>  > Meanwhile, new changes I'm putting in to trunk are tagged as fixed in
> 3.5.0
>  > -correct?
>
>  > steve
>
>
> > On Thu, 11 Jan 2024 at 05:15, slfan1989 wrote:
>
> > Hello all,
> >
> > We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first
> > hadoop 3.4.0-RC version.
> >
> > The RC is available at:
> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64)
> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64)
> >
> > Maven artifacts is built by x86 machine and are staged at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1391/
> >
> > My public key:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > Changelog:
> > https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md
> >
> > Release notes:
> >
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md
> >
> > This is a relatively big release (by Hadoop standard) containing about
> 2852
> > commits.
> >
> > Please give it a try, this RC vote will run for 7 days.
> >
> > Feature highlights:
> >
> > DataNode FsDatasetImpl Fine-Grained Locking via BlockPool
> > 
> > [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split
> > FsDatasetImpl datasetLock via blockpool to solve the issue of heavy
> > FsDatasetImpl datasetLock
> > When there are many namespaces in a large cluster.
> >
> > YARN Federation improvements
> > 
> > [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many
> > improvements, including the following:
> >
> > 1. YARN Router now boasts a full implementation of all relevant
> interfaces
> > including the ApplicationClientProtocol,
> > ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
> > 2. Enhanced support for Application cleanup and automatic offline
> > mechanisms for SubCluster are now facilitated by the YARN Router.
> > 3. Code optimization for Router and AMRMProxy was undertaken, coupled
> with
> > improvements to previously pending functionalities.
> > 4. Audit logs and Metrics for Router received upgrades.
> > 5. A boost in cluster security features was achieved, with the inclusion
> of
> > Kerberos support.
> > 6. The page function of the router has been enhanced.
> >
> > Upgrade AWS SDK to V2
> > 
> > [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073)
> > The S3A connector now uses the V2 AWS SDK. This is a significant change
> at
> > the source code level.
> > Any applications using the internal extension/override points in the
> > filesystem connector are likely to break.
> > Consult the document aws\_sdk\_upgrade for the full details.
> >
> > hadoop-thirdparty will also provide the new RC0 soon.
> >
> > Best Regards,
> > Shilun Fan.
> >
>

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[slfan1989]

Thank you very much for your help in verifying this version! We will use
version 3.5.0 for fix jira in the future.

Best Regards,
Shilun Fan.

 > wonderful! I'll be testing over the weekend

 > Meanwhile, new changes I'm putting in to trunk are tagged as fixed in
3.5.0
 > -correct?

 > steve


> On Thu, 11 Jan 2024 at 05:15, slfan1989 wrote:

> Hello all,
>
> We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first
> hadoop 3.4.0-RC version.
>
> The RC is available at:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64)
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64)
>
> Maven artifacts is built by x86 machine and are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1391/
>
> My public key:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Changelog:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md
>
> Release notes:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md
>
> This is a relatively big release (by Hadoop standard) containing about
2852
> commits.
>
> Please give it a try, this RC vote will run for 7 days.
>
> Feature highlights:
>
> DataNode FsDatasetImpl Fine-Grained Locking via BlockPool
> 
> [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split
> FsDatasetImpl datasetLock via blockpool to solve the issue of heavy
> FsDatasetImpl datasetLock
> When there are many namespaces in a large cluster.
>
> YARN Federation improvements
> 
> [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many
> improvements, including the following:
>
> 1. YARN Router now boasts a full implementation of all relevant interfaces
> including the ApplicationClientProtocol,
> ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
> 2. Enhanced support for Application cleanup and automatic offline
> mechanisms for SubCluster are now facilitated by the YARN Router.
> 3. Code optimization for Router and AMRMProxy was undertaken, coupled with
> improvements to previously pending functionalities.
> 4. Audit logs and Metrics for Router received upgrades.
> 5. A boost in cluster security features was achieved, with the inclusion
of
> Kerberos support.
> 6. The page function of the router has been enhanced.
>
> Upgrade AWS SDK to V2
> 
> [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073)
> The S3A connector now uses the V2 AWS SDK. This is a significant change at
> the source code level.
> Any applications using the internal extension/override points in the
> filesystem connector are likely to break.
> Consult the document aws\_sdk\_upgrade for the full details.
>
> hadoop-thirdparty will also provide the new RC0 soon.
>
> Best Regards,
> Shilun Fan.
>

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Steve Loughran]

wonderful! I'll be testing over the weekend

Meanwhile, new changes I'm putting in to trunk are tagged as fixed in 3.5.0
-correct?

steve


On Thu, 11 Jan 2024 at 05:15, slfan1989  wrote:

> Hello all,
>
> We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first
> hadoop 3.4.0-RC version.
>
> The RC is available at:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64)
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64)
>
> Maven artifacts is built by x86 machine and are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1391/
>
> My public key:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Changelog:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md
>
> Release notes:
> https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md
>
> This is a relatively big release (by Hadoop standard) containing about 2852
> commits.
>
> Please give it a try, this RC vote will run for 7 days.
>
> Feature highlights:
>
> DataNode FsDatasetImpl Fine-Grained Locking via BlockPool
> 
> [HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split
> FsDatasetImpl datasetLock via blockpool to solve the issue of heavy
> FsDatasetImpl datasetLock
> When there are many namespaces in a large cluster.
>
> YARN Federation improvements
> 
> [YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many
> improvements, including the following:
>
> 1. YARN Router now boasts a full implementation of all relevant interfaces
> including the ApplicationClientProtocol,
> ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
> 2. Enhanced support for Application cleanup and automatic offline
> mechanisms for SubCluster are now facilitated by the YARN Router.
> 3. Code optimization for Router and AMRMProxy was undertaken, coupled with
> improvements to previously pending functionalities.
> 4. Audit logs and Metrics for Router received upgrades.
> 5. A boost in cluster security features was achieved, with the inclusion of
> Kerberos support.
> 6. The page function of the router has been enhanced.
>
> Upgrade AWS SDK to V2
> 
> [HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073)
> The S3A connector now uses the V2 AWS SDK.  This is a significant change at
> the source code level.
> Any applications using the internal extension/override points in the
> filesystem connector are likely to break.
> Consult the document aws\_sdk\_upgrade for the full details.
>
> hadoop-thirdparty will also provide the new RC0 soon.
>
> Best Regards,
> Shilun Fan.
>

Re: [VOTE] Release Apache Hadoop 3.4.0 RC0

[Masatake Iwasaki]

Thanks for driving this release, Shilun Fan.

The top page of site documentation (in hadoop-3.4.0-RC0-site.tar.gz) 
looks the same as 3.3.5.


While the index.md.vm is updated in branch-3.4.0[1], it seems not to be 
reflected.

release-3.4.0-RC0 tag should be pushed to make checking easier.

In addition, the description about new features of previous release 
should be removed from the index.md.vm.


[1] 
https://github.com/apache/hadoop/blob/branch-3.4.0/hadoop-project/src/site/markdown/index.md.vm


Masatake Iwasaki

On 2024/01/11 14:15, slfan1989 wrote:

Hello all,

We plan to release hadoop 3.4.0 based on hadoop trunk, which is the first
hadoop 3.4.0-RC version.

The RC is available at:
https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/ (for amd64)
https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-arm64/ (for arm64)

Maven artifacts is built by x86 machine and are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1391/

My public key:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

Changelog:
https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/CHANGELOG.md

Release notes:
https://home.apache.org/~slfan1989/hadoop-3.4.0-RC0-amd64/RELEASENOTES.md

This is a relatively big release (by Hadoop standard) containing about 2852
commits.

Please give it a try, this RC vote will run for 7 days.

Feature highlights:

DataNode FsDatasetImpl Fine-Grained Locking via BlockPool

[HDFS-15180](https://issues.apache.org/jira/browse/HDFS-15180) Split
FsDatasetImpl datasetLock via blockpool to solve the issue of heavy
FsDatasetImpl datasetLock
When there are many namespaces in a large cluster.

YARN Federation improvements

[YARN-5597](https://issues.apache.org/jira/browse/YARN-5597) brings many
improvements, including the following:

1. YARN Router now boasts a full implementation of all relevant interfaces
including the ApplicationClientProtocol,
ResourceManagerAdministrationProtocol, and RMWebServiceProtocol.
2. Enhanced support for Application cleanup and automatic offline
mechanisms for SubCluster are now facilitated by the YARN Router.
3. Code optimization for Router and AMRMProxy was undertaken, coupled with
improvements to previously pending functionalities.
4. Audit logs and Metrics for Router received upgrades.
5. A boost in cluster security features was achieved, with the inclusion of
Kerberos support.
6. The page function of the router has been enhanced.

Upgrade AWS SDK to V2

[HADOOP-18073](https://issues.apache.org/jira/browse/HADOOP-18073)
The S3A connector now uses the V2 AWS SDK.  This is a significant change at
the source code level.
Any applications using the internal extension/override points in the
filesystem connector are likely to break.
Consult the document aws\_sdk\_upgrade for the full details.

hadoop-thirdparty will also provide the new RC0 soon.

Best Regards,
Shilun Fan.



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Re: [DISCUSS] Release Hadoop 3.4.0

[Ayush Saxena]

Thanx @slfan1989 for volunteering. Please remove this [1] from the new
branches-3.4 & 3.4.0 when you create them as part of preparing for the
release, else it would be putting up trunk labels for backport PRs to
those branches as well.

There are some tickets marked as Critical/Blocker for 3.4.0 [2], just
give a check to them if they are actually critical or not, if yes, we
should get them in. Most of them were not looking relevant to me at
first glance.

-Ayush


[1] https://github.com/apache/hadoop/blob/trunk/.github/labeler.yml
[2] 
https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20HADOOP%2C%20MAPREDUCE%2C%20YARN)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20resolution%20%3D%20Unresolved%20AND%20affectedVersion%20in%20(3.4.0%2C%203.4.1)


On Thu, 4 Jan 2024 at 19:57, slfan1989  wrote:
>
> Hey all,
>
> We are planning to release Hadoop 3.4.0 base on trunk. I made some 
> preparations and changed the target version of JIRA for non-blockers in 
> HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to create a 
> new JIRA, the target version can directly select version 3.5.0.
>
> If you have any thoughts, suggestions, or concerns, please feel free to share 
> them.
>
> Best Regards,
> Shilun Fan.
>
> > +1 from me.
> >> It will include the new AWS V2 SDK upgrade as well.
>
> > On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He wrote:
>
> > >
> > > I think the release discussion can be in public ML?
> >
> > Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> > On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote:
> >
> > > +1 from me as well.
> > >
> > > We should definitely attempt to upgrade the thirdparty version for
> > > 3.4.0 & check if there are any pending critical/blocker issues as
> > > well.
> > >
> > > I think the release discussion can be in public ML?
> > >
> > > -Ayush
> > >
> > > On Mon, 1 Jan 2024 at 18:25, Steve Loughran  > >
> > > wrote:
> > > >
> > > > +1 from me
> > > >
> > > > ant and maven repo to build and validate things, including making arm
> > > > binaries if you work from an arm macbook.
> > > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > > >
> > > > do we need to publish an up to date thirdparty release for this?
> > > >
> > > >
> > > >
> > > > On Mon, 25 Dec 2023 at 16:06, slfan1989 wrote:
> > > >
> > > > > Dear PMC Members,
> > > > >
> > > > > First of all, Merry Christmas to everyone!
> > > > >
> > > > > In our community discussions, we collectively finalized the plan to
> > > release
> > > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to take
> > > on
> > > > > the responsibility for the initial release of version 3.4.0, and the
> > > entire
> > > > > process is set to officially commence in January 2024.
> > > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0.
> > > > >
> > > > > The specific work plan includes:
> > > > >
> > > > > 1. Following the guidance in the HowToRelease document, completing
> > all
> > > the
> > > > > relevant tasks required for the release of version 3.4.0.
> > > > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT.
> > > > > 3. Currently, the Fix Versions of all tasks merged into trunk are set
> > > as
> > > > > 3.4.0; I will move them to 3.5.0.
> > > > >
> > > > > Confirmed features to be included in the release:
> > > > >
> > > > > 1. Enhanced functionality for YARN Federation.
> > > > > 2. Optimization of HDFS RBF.
> > > > > 3. Introduction of fine-grained global locks for DataNodes.
> > > > > 4. Improvements in the stability of HDFS EC, and more.
> > > > > 5. Fixes for important CVEs.
> > > > >
> > > > > If you have any thoughts, suggestions, or concerns, please feel free
> > to
> > > > > share them.
> > > > >
> > > > > Looking forward to a successful release!
> > > > >
> > > > > Best Regards,
> > > > > Shilun Fan.
> > > > >
> > >
> >

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Re: [DISCUSS] Release Hadoop 3.4.0

[slfan1989]

Hey all,

I plan to create a new branch of branch-3.4.0 this weekend and change the
version of the trunk branch to 3.5.0-SNAPSHOT.

If all goes well, I will release hadoop-3.4.0-RC0 early next week.

hadoop-3.4.0 contains many changes, we may release multiple RC versions.

Best Regards,
Shilun Fan.

> On Thu, 4 Jan 2024 at 15:53, PJ Fanning wrote:

> The new jar dependency just adds a class that Jackson JAX-RS support
> expects but that doesn't exist in the JAX-RS v1 jsr-311 jar [1].
> JAX-RS v2 uses a jar called rs-api instead of jsr-311. The package
> names are the same but there are differences (new classes and API
> changes in some other classes).

> Studying the Jackson change that broke support for JAX-RS v1 [2] and
> some basic testing seems to support the idea that having this one
> class available means that Hadoop and Tex should be able to use
> versions of Jackson later than the breaking change. Jackson JAX-RS
> code does not use the rs-api changes with the one exception of
> requiring the NoContentException. The Jackson change [2] appears in
> Jackson 2.13.0.

> [1]
https://github.com/pjfanning/jsr311-compat/blob/main/src/main/java/javax/ws/rs/core/

> NoContentException.java

> [2]
https://github.com/FasterXML/jackson-jaxrs-providers/issues/134#issuecomment-1180637522

> On Thu, 4 Jan 2024 at 19:30, Steve Loughran wrote:
>
>
>
> On Thu, 4 Jan 2024 at 15:53, PJ Fanning wrote:
>>
>> I would like to get some dependencies upgraded for Hadoop 3.4.0. For me,
it would be good to upgrade protobuf-java and Jackson to more secure
versions.
>>
>> For protobuf-java, that would involve releasing hadoop-thirdparty 1.2.0
[1], including merging the protobuf upgrade [2].
>
>
> I agree, we should update this. Anything else in there to update?
>>
>>
>> For Jackson, we are hampered by Jackson dropping support for
Jersey/JAX-RS 1.x. I have a workaround for this that I think is worth
further investigating [3].
>
>
> What does your new module do?
>>
>>
>> [1] https://github.com/apache/hadoop-thirdparty
>> [2] https://github.com/apache/hadoop-thirdparty/pull/19
>> [3] https://github.com/apache/hadoop/pull/6370
>>
>>
>>
>> On 2024/01/04 14:26:37 slfan1989 wrote:
>> > Hey all,
>> >
>> > We are planning to release Hadoop 3.4.0 base on trunk. I made some
>> > preparations and changed the target version of JIRA for non-blockers in
>> > HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to
create
>> > a new JIRA, the target version can directly select version 3.5.0.
>> >
>> > If you have any thoughts, suggestions, or concerns, please feel free to
>> > share them.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Re: [DISCUSS] Release Hadoop 3.4.0

[slfan1989]

Hey all,

We are planning to release Hadoop 3.4.0 base on trunk. I made some
preparations and changed the target version of JIRA for non-blockers in
HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to create
a new JIRA, the target version can directly select version 3.5.0.

If you have any thoughts, suggestions, or concerns, please feel free to
share them.

Best Regards,
Shilun Fan.

> +1 from me.
>> It will include the new AWS V2 SDK upgrade as well.

> On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He wrote:

> >
> > I think the release discussion can be in public ML?
>
> Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote:
>
> > +1 from me as well.
> >
> > We should definitely attempt to upgrade the thirdparty version for
> > 3.4.0 & check if there are any pending critical/blocker issues as
> > well.
> >
> > I think the release discussion can be in public ML?
> >
> > -Ayush
> >
> > On Mon, 1 Jan 2024 at 18:25, Steve Loughran  >
> > wrote:
> > >
> > > +1 from me
> > >
> > > ant and maven repo to build and validate things, including making arm
> > > binaries if you work from an arm macbook.
> > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > >
> > > do we need to publish an up to date thirdparty release for this?
> > >
> > >
> > >
> > > On Mon, 25 Dec 2023 at 16:06, slfan1989 wrote:
> > >
> > > > Dear PMC Members,
> > > >
> > > > First of all, Merry Christmas to everyone!
> > > >
> > > > In our community discussions, we collectively finalized the plan to
> > release
> > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to
take
> > on
> > > > the responsibility for the initial release of version 3.4.0, and the
> > entire
> > > > process is set to officially commence in January 2024.
> > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0.
> > > >
> > > > The specific work plan includes:
> > > >
> > > > 1. Following the guidance in the HowToRelease document, completing
> all
> > the
> > > > relevant tasks required for the release of version 3.4.0.
> > > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT.
> > > > 3. Currently, the Fix Versions of all tasks merged into trunk are
set
> > as
> > > > 3.4.0; I will move them to 3.5.0.
> > > >
> > > > Confirmed features to be included in the release:
> > > >
> > > > 1. Enhanced functionality for YARN Federation.
> > > > 2. Optimization of HDFS RBF.
> > > > 3. Introduction of fine-grained global locks for DataNodes.
> > > > 4. Improvements in the stability of HDFS EC, and more.
> > > > 5. Fixes for important CVEs.
> > > >
> > > > If you have any thoughts, suggestions, or concerns, please feel free
> to
> > > > share them.
> > > >
> > > > Looking forward to a successful release!
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> >
>

Re: [DISCUSS] Release Hadoop 3.4.0

[Mukund Madhav Thakur]

+1 from me.
It will include the new AWS V2 SDK upgrade as well.

On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He  wrote:

> >
> >  I think the release discussion can be in public ML?
>
> Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena  wrote:
>
> > +1 from me as well.
> >
> > We should definitely attempt to upgrade the thirdparty version for
> > 3.4.0 & check if there are any pending critical/blocker issues as
> > well.
> >
> > I think the release discussion can be in public ML?
> >
> > -Ayush
> >
> > On Mon, 1 Jan 2024 at 18:25, Steve Loughran  >
> > wrote:
> > >
> > > +1 from me
> > >
> > > ant and maven repo to build and validate things, including making arm
> > > binaries if you work from an arm macbook.
> > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > >
> > > do we need to publish an up to date thirdparty release for this?
> > >
> > >
> > >
> > > On Mon, 25 Dec 2023 at 16:06, slfan1989  wrote:
> > >
> > > > Dear PMC Members,
> > > >
> > > > First of all, Merry Christmas to everyone!
> > > >
> > > > In our community discussions, we collectively finalized the plan to
> > release
> > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to take
> > on
> > > > the responsibility for the initial release of version 3.4.0, and the
> > entire
> > > > process is set to officially commence in January 2024.
> > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0.
> > > >
> > > > The specific work plan includes:
> > > >
> > > > 1. Following the guidance in the HowToRelease document, completing
> all
> > the
> > > > relevant tasks required for the release of version 3.4.0.
> > > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT.
> > > > 3. Currently, the Fix Versions of all tasks merged into trunk are set
> > as
> > > > 3.4.0; I will move them to 3.5.0.
> > > >
> > > > Confirmed features to be included in the release:
> > > >
> > > > 1. Enhanced functionality for YARN Federation.
> > > > 2. Optimization of HDFS RBF.
> > > > 3. Introduction of fine-grained global locks for DataNodes.
> > > > 4. Improvements in the stability of HDFS EC, and more.
> > > > 5. Fixes for important CVEs.
> > > >
> > > > If you have any thoughts, suggestions, or concerns, please feel free
> to
> > > > share them.
> > > >
> > > > Looking forward to a successful release!
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> >
>

Re: [DISCUSS] Release Hadoop 3.4.0

[Xiaoqiao He]

>
>  I think the release discussion can be in public ML?

Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML.

Best Regards,
- He Xiaoqiao

On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena  wrote:

> +1 from me as well.
>
> We should definitely attempt to upgrade the thirdparty version for
> 3.4.0 & check if there are any pending critical/blocker issues as
> well.
>
> I think the release discussion can be in public ML?
>
> -Ayush
>
> On Mon, 1 Jan 2024 at 18:25, Steve Loughran 
> wrote:
> >
> > +1 from me
> >
> > ant and maven repo to build and validate things, including making arm
> > binaries if you work from an arm macbook.
> > https://github.com/steveloughran/validate-hadoop-client-artifacts
> >
> > do we need to publish an up to date thirdparty release for this?
> >
> >
> >
> > On Mon, 25 Dec 2023 at 16:06, slfan1989  wrote:
> >
> > > Dear PMC Members,
> > >
> > > First of all, Merry Christmas to everyone!
> > >
> > > In our community discussions, we collectively finalized the plan to
> release
> > > Hadoop 3.4.0 based on the current trunk branch. I am applying to take
> on
> > > the responsibility for the initial release of version 3.4.0, and the
> entire
> > > process is set to officially commence in January 2024.
> > > I have created a new JIRA: HADOOP-19018. Release 3.4.0.
> > >
> > > The specific work plan includes:
> > >
> > > 1. Following the guidance in the HowToRelease document, completing all
> the
> > > relevant tasks required for the release of version 3.4.0.
> > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT.
> > > 3. Currently, the Fix Versions of all tasks merged into trunk are set
> as
> > > 3.4.0; I will move them to 3.5.0.
> > >
> > > Confirmed features to be included in the release:
> > >
> > > 1. Enhanced functionality for YARN Federation.
> > > 2. Optimization of HDFS RBF.
> > > 3. Introduction of fine-grained global locks for DataNodes.
> > > 4. Improvements in the stability of HDFS EC, and more.
> > > 5. Fixes for important CVEs.
> > >
> > > If you have any thoughts, suggestions, or concerns, please feel free to
> > > share them.
> > >
> > > Looking forward to a successful release!
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
>

Re: [VOTE] Hadoop 3.2.x EOL

[Xiaoqiao He]

Hi All, JIRA/WIKI/Website has been updated. Release line 3.2 is EOL now.

Thanks all.

Best Regards,
- He Xiaoqiao

On Tue, Dec 19, 2023 at 11:58 AM Xiaoqiao He  wrote:

> This vote passed with 10 +1. I'll update the JIRA, WIKI and website for a
> while.
>
> Thanks all for your participation.
>
> Best Regards,
> - He Xiaoqiao
>
>
> On Thu, Dec 7, 2023 at 6:14 AM Iñigo Goiri  wrote:
>
>> +1
>>
>> On Wed, Dec 6, 2023 at 9:03 AM Chao Sun  wrote:
>>
>>> +1
>>>
>>> On Wed, Dec 6, 2023 at 8:39 AM Akira Ajisaka 
>>> wrote:
>>> >
>>> > +1
>>> >
>>> >
>>> >
>>> > On Wed, Dec 6, 2023 at 1:10 PM Xiaoqiao He 
>>> wrote:
>>> >
>>> > > Dear Hadoop devs,
>>> > >
>>> > > Given the feedback from the discussion thread [1], I'd like to start
>>> > > an official thread for the community to vote on release line 3.2 EOL.
>>> > >
>>> > > It will include,
>>> > > a. An official announcement informs no further regular Hadoop 3.2.x
>>> > > releases.
>>> > > b. Issues which target 3.2.5 will not be fixed.
>>> > >
>>> > > This vote will run for 7 days and conclude by Dec 13, 2023.
>>> > >
>>> > > I’ll start with my +1.
>>> > >
>>> > > Best Regards,
>>> > > - He Xiaoqiao
>>> > >
>>> > > [1] https://lists.apache.org/thread/bbf546c6jz0og3xcl9l3qfjo93b65szr
>>> > >
>>>
>>> -
>>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>>>
>>>