date:20240201

Re: Discussion about NameNode Fine-grained locking

2024-02-01 Thread Hui Fei

Thanks for driving this. It's very meaningful. The performance improvement
looks very good.

Many users are facing the write performance issue. As far as I know, some
companies already implemented the similar idea on their internal branches.
But the internal branch is very different from the community one. So it's
very hard to be in sync with the community. If this improvement can be
involved in the community, that would be great to both end-user and the
community.

It is very worth doing.

Zengqiang XU  于2024年2月2日周五 11:07写道：

> Hi everyone
>
> I have started a discussion about NameNode Fine-grained Locking to improve
> performance of write operations in NameNode.
>
> I started this discussion again for serval main reasons:
> 1. We have implemented it and gained nearly 7x performance improvement in
> our prod environment
> 2. Many other companies made similar improvements based on their internal
> branch.
> 3. This topic has been discussed for a long time, but still without any
> results.
>
> I hope we can push this important improvement in the community so that all
> end-users can enjoy this significant improvement.
>
> I'd really appreciate you can join in and work with me to push this feature
> forward.
>
> Thanks very much.
>
> Ticket: HDFS-17366 
> Design: NameNode Fine-grained locking based on directory tree
> <
> https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing
> >
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-02-01 Thread slfan1989

Thank you very much for the review! I will avoid the diff.

Best Regards,
Shilun Fan.

On Fri, Feb 2, 2024 at 9:59 AM Takanobu Asanuma  wrote:

> It also looks good to me, except for the diff.
>
> * Verified signatures and hashes
> * Reviewed the documents
> * Successfully built from source with `mvn clean install`
> * Successfully compiled Hadoop trunk and branch-3.4 using the Hadoop
> thirdparty 1.2.0
>
> Anyway, since hadoop-thirdparty-1.1.1 has some high vulnerabilities,
> hadoop-thirdparty-1.2.0 would be required for Hadoop-3.4.0.
>
> Thanks,
> - Takanobu
>
> 2024年2月2日(金) 4:45 slfan1989 :
>
> > Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and providing
> > feedback!
> >
> > I followed the "how to release" documentation and tried to package it
> using
> > create-release and Dockerfile, but I couldn't successfully package it
> > directly. Some modifications are required before compilation. I should
> > submit a pull request to fix this issue before
> > Hadoop-Thirdparty-1.2.0-RC0 compile.
> >
> > This is an area that needs improvement. We should ensure that the code of
> > src is consistent with the tag.
> >
> > On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena  wrote:
> >
> > >
> > > There is some diff b/w the git tag & the src tar, the Dockerfile & the
> > > create-release are different, Why?
> > >
> > > Files hadoop-thirdparty/dev-support/bin/create-release and
> > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
> > >
> > > Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
> > >
> > >
> > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > hadoop-thirdparty/dev-support/bin/create-release
> > > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
> > >
> > > 444,446c444,446
> > >
> > > < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
> > >
> > > < echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
> > >
> > > < echo "RUN chown -R ${user_name} /home/${user_name}"
> > >
> > > ---
> > >
> > > > echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit
> > > 0;"
> > >
> > > > echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name};
> > > exit 0;"
> > >
> > > > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
> > >
> > > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > > hadoop-thirdparty/dev-support/docker/Dockerfile
> > > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
> > >
> > > 103a104,105
> > >
> > > > RUN rm -f /etc/maven/settings.xml && ln -s
> /home/root/.m2/settings.xml
> > > /etc/maven/settings.xml
> > >
> > > >
> > >
> > > 126a129,130
> > >
> > > > RUN pip2 install setuptools-scm==5.0.2
> > >
> > > > RUN pip2 install lazy-object-proxy==1.5.0
> > >
> > > 159d162
> > >
> > > <
> > >
> > >
> > >
> > >
> > > Other things look Ok,
> > > * Built from source
> > > * Verified Checksums
> > > * Verified Signatures
> > > * Validated files have ASF header
> > >
> > > Not sure if having diff b/w the git tag & src tar is ok, this doesn't
> > look
> > > like core code change though, can anybody check & confirm?
> > >
> > > -Ayush
> > >
> > >
> > > On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He 
> wrote:
> > >
> > >> Gentle ping. @Ayush Saxena  @Steve Loughran
> > >>  @inigo...@apache.org 
> > >> @Masatake
> > >> Iwasaki  and some other folks.
> > >>
> > >> On Wed, Jan 31, 2024 at 10:17 AM slfan1989 
> > wrote:
> > >>
> > >> > Thank you for the review and vote! Looking forward to other forks
> > >> helping
> > >> > with voting and verification.
> > >> >
> > >> > Best Regards,
> > >> > Shilun Fan.
> > >> >
> > >> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
> > >> wrote:
> > >> >
> > >> > > Thanks Shilun for driving it and making it happen.
> > >> > >
> > >> > > +1(binding).
> > >> > >
> > >> > > [x] Checksums and PGP signatures are valid.
> > >> > > [x] LICENSE files exist.
> > >> > > [x] NOTICE is included.
> > >> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> > >> > > [x] Built from source works well: `mvn clean install`
> > >> > > [x] Built Hadoop trunk with updated thirdparty successfully
> (include
> > >> > update
> > >> > > protobuf shaded path).
> > >> > >
> > >> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0,
> hope
> > >> we
> > >> > > could finish this vote before 2024/02/06(UTC) if there are no
> > >> concerns.
> > >> > > Thanks all.
> > >> > >
> > >> > > Best Regards,
> > >> > > - He Xiaoqiao
> > >> > >
> > >> > >
> > >> > >
> > >> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
> > >> wrote:
> > >> > >
> > >> > > > Hi folks,
> > >> > > >
> > >> > > > Xiaoqiao He and I have put together a release candidate (RC0)
> for
> > >> > Hadoop
> > >> > > > Thirdparty 1.2.0.
> > >> > > >
> > >> > > > The RC is available at:
> > >> > > >
> > >> > >
> > >> >
> > >>
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> > >> > > >
>

Discussion about NameNode Fine-grained locking

2024-02-01 Thread Zengqiang XU

Hi everyone

I have started a discussion about NameNode Fine-grained Locking to improve
performance of write operations in NameNode.

I started this discussion again for serval main reasons:
1. We have implemented it and gained nearly 7x performance improvement in
our prod environment
2. Many other companies made similar improvements based on their internal
branch.
3. This topic has been discussed for a long time, but still without any
results.

I hope we can push this important improvement in the community so that all
end-users can enjoy this significant improvement.

I'd really appreciate you can join in and work with me to push this feature
forward.

Thanks very much.

Ticket: HDFS-17366 
Design: NameNode Fine-grained locking based on directory tree

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-02-01 Thread Takanobu Asanuma

It also looks good to me, except for the diff.

* Verified signatures and hashes
* Reviewed the documents
* Successfully built from source with `mvn clean install`
* Successfully compiled Hadoop trunk and branch-3.4 using the Hadoop
thirdparty 1.2.0

Anyway, since hadoop-thirdparty-1.1.1 has some high vulnerabilities,
hadoop-thirdparty-1.2.0 would be required for Hadoop-3.4.0.

Thanks,
- Takanobu

2024年2月2日(金) 4:45 slfan1989 :

> Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and providing
> feedback!
>
> I followed the "how to release" documentation and tried to package it using
> create-release and Dockerfile, but I couldn't successfully package it
> directly. Some modifications are required before compilation. I should
> submit a pull request to fix this issue before
> Hadoop-Thirdparty-1.2.0-RC0 compile.
>
> This is an area that needs improvement. We should ensure that the code of
> src is consistent with the tag.
>
> On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena  wrote:
>
> >
> > There is some diff b/w the git tag & the src tar, the Dockerfile & the
> > create-release are different, Why?
> >
> > Files hadoop-thirdparty/dev-support/bin/create-release and
> > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
> >
> > Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
> >
> >
> > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > hadoop-thirdparty/dev-support/bin/create-release
> > hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
> >
> > 444,446c444,446
> >
> > < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
> >
> > < echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
> >
> > < echo "RUN chown -R ${user_name} /home/${user_name}"
> >
> > ---
> >
> > > echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit
> > 0;"
> >
> > > echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name};
> > exit 0;"
> >
> > > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
> >
> > ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> > hadoop-thirdparty/dev-support/docker/Dockerfile
> > hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
> >
> > 103a104,105
> >
> > > RUN rm -f /etc/maven/settings.xml && ln -s /home/root/.m2/settings.xml
> > /etc/maven/settings.xml
> >
> > >
> >
> > 126a129,130
> >
> > > RUN pip2 install setuptools-scm==5.0.2
> >
> > > RUN pip2 install lazy-object-proxy==1.5.0
> >
> > 159d162
> >
> > <
> >
> >
> >
> >
> > Other things look Ok,
> > * Built from source
> > * Verified Checksums
> > * Verified Signatures
> > * Validated files have ASF header
> >
> > Not sure if having diff b/w the git tag & src tar is ok, this doesn't
> look
> > like core code change though, can anybody check & confirm?
> >
> > -Ayush
> >
> >
> > On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He  wrote:
> >
> >> Gentle ping. @Ayush Saxena  @Steve Loughran
> >>  @inigo...@apache.org 
> >> @Masatake
> >> Iwasaki  and some other folks.
> >>
> >> On Wed, Jan 31, 2024 at 10:17 AM slfan1989 
> wrote:
> >>
> >> > Thank you for the review and vote! Looking forward to other forks
> >> helping
> >> > with voting and verification.
> >> >
> >> > Best Regards,
> >> > Shilun Fan.
> >> >
> >> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
> >> wrote:
> >> >
> >> > > Thanks Shilun for driving it and making it happen.
> >> > >
> >> > > +1(binding).
> >> > >
> >> > > [x] Checksums and PGP signatures are valid.
> >> > > [x] LICENSE files exist.
> >> > > [x] NOTICE is included.
> >> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> >> > > [x] Built from source works well: `mvn clean install`
> >> > > [x] Built Hadoop trunk with updated thirdparty successfully (include
> >> > update
> >> > > protobuf shaded path).
> >> > >
> >> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope
> >> we
> >> > > could finish this vote before 2024/02/06(UTC) if there are no
> >> concerns.
> >> > > Thanks all.
> >> > >
> >> > > Best Regards,
> >> > > - He Xiaoqiao
> >> > >
> >> > >
> >> > >
> >> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
> >> wrote:
> >> > >
> >> > > > Hi folks,
> >> > > >
> >> > > > Xiaoqiao He and I have put together a release candidate (RC0) for
> >> > Hadoop
> >> > > > Thirdparty 1.2.0.
> >> > > >
> >> > > > The RC is available at:
> >> > > >
> >> > >
> >> >
> >>
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> >> > > >
> >> > > > The RC tag is
> >> > > >
> >> > >
> >> >
> >>
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> >> > > >
> >> > > > The maven artifacts are staged at
> >> > > >
> >> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1398
> >> > > >
> >> > > > Comparing to 1.1.1, there are three additional fixes:
> >> > > >
> >> > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> >> > > >

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2024-02-01 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/623/

[Jan 31, 2024, 5:30:35 AM] (github) HADOOP-19056. Highlight RBF features and 
improvements targeting version 3.4. (#6512) Contributed by Takanobu Asanuma.




-1 overall


The following subsystems voted -1:
blanks hadolint mvnsite pathlen spotbugs unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:[line 343] 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:[line 356] 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:[line 333] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:[line 343] 
   Redundant nullcheck of it, which is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:is known to be non-null in 
org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.recoverTrackerResources(LocalResourcesTracker,
 NMStateStoreService$LocalResourceTrackerState) Redundant null check at 
ResourceLocalizationService.java:[line 356] 
   Boxed value is unboxed and then immediately reboxed in 
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnRWHelper.readResultsWithTimestamps(Result,
 byte[], byte[], KeyConverter, ValueConverter, boolean) At 
ColumnRWHelper.java:then immediately reboxed in

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-02-01 Thread slfan1989

Thank you for helping to review Hadoop-Thirdparty-1.2.0-RC0 and providing
feedback!

I followed the "how to release" documentation and tried to package it using
create-release and Dockerfile, but I couldn't successfully package it
directly. Some modifications are required before compilation. I should
submit a pull request to fix this issue before
Hadoop-Thirdparty-1.2.0-RC0 compile.

This is an area that needs improvement. We should ensure that the code of
src is consistent with the tag.

On Fri, Feb 2, 2024 at 2:25 AM Ayush Saxena  wrote:

>
> There is some diff b/w the git tag & the src tar, the Dockerfile & the
> create-release are different, Why?
>
> Files hadoop-thirdparty/dev-support/bin/create-release and
> hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ
>
> Files hadoop-thirdparty/dev-support/docker/Dockerfile and
> hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ
>
>
> ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> hadoop-thirdparty/dev-support/bin/create-release
> hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release
>
> 444,446c444,446
>
> < echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"
>
> < echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"
>
> < echo "RUN chown -R ${user_name} /home/${user_name}"
>
> ---
>
> > echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit
> 0;"
>
> > echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name};
> exit 0;"
>
> > echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"
>
> ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
> hadoop-thirdparty/dev-support/docker/Dockerfile
> hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile
>
> 103a104,105
>
> > RUN rm -f /etc/maven/settings.xml && ln -s /home/root/.m2/settings.xml
> /etc/maven/settings.xml
>
> >
>
> 126a129,130
>
> > RUN pip2 install setuptools-scm==5.0.2
>
> > RUN pip2 install lazy-object-proxy==1.5.0
>
> 159d162
>
> <
>
>
>
>
> Other things look Ok,
> * Built from source
> * Verified Checksums
> * Verified Signatures
> * Validated files have ASF header
>
> Not sure if having diff b/w the git tag & src tar is ok, this doesn't look
> like core code change though, can anybody check & confirm?
>
> -Ayush
>
>
> On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He  wrote:
>
>> Gentle ping. @Ayush Saxena  @Steve Loughran
>>  @inigo...@apache.org 
>> @Masatake
>> Iwasaki  and some other folks.
>>
>> On Wed, Jan 31, 2024 at 10:17 AM slfan1989  wrote:
>>
>> > Thank you for the review and vote! Looking forward to other forks
>> helping
>> > with voting and verification.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> >
>> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
>> wrote:
>> >
>> > > Thanks Shilun for driving it and making it happen.
>> > >
>> > > +1(binding).
>> > >
>> > > [x] Checksums and PGP signatures are valid.
>> > > [x] LICENSE files exist.
>> > > [x] NOTICE is included.
>> > > [x] Rat check is ok. `mvn clean apache-rat:check`
>> > > [x] Built from source works well: `mvn clean install`
>> > > [x] Built Hadoop trunk with updated thirdparty successfully (include
>> > update
>> > > protobuf shaded path).
>> > >
>> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope
>> we
>> > > could finish this vote before 2024/02/06(UTC) if there are no
>> concerns.
>> > > Thanks all.
>> > >
>> > > Best Regards,
>> > > - He Xiaoqiao
>> > >
>> > >
>> > >
>> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
>> wrote:
>> > >
>> > > > Hi folks,
>> > > >
>> > > > Xiaoqiao He and I have put together a release candidate (RC0) for
>> > Hadoop
>> > > > Thirdparty 1.2.0.
>> > > >
>> > > > The RC is available at:
>> > > >
>> > >
>> >
>> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
>> > > >
>> > > > The RC tag is
>> > > >
>> > >
>> >
>> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
>> > > >
>> > > > The maven artifacts are staged at
>> > > >
>> > https://repository.apache.org/content/repositories/orgapachehadoop-1398
>> > > >
>> > > > Comparing to 1.1.1, there are three additional fixes:
>> > > >
>> > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
>> > > > https://github.com/apache/hadoop-thirdparty/pull/26
>> > > >
>> > > > HADOOP-18921. Upgrade to avro 1.11.3
>> > > > https://github.com/apache/hadoop-thirdparty/pull/24
>> > > >
>> > > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
>> > > > https://github.com/apache/hadoop-thirdparty/pull/23
>> > > >
>> > > > You can find my public key at :
>> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> > > >
>> > > > Best Regards,
>> > > > Shilun Fan.
>> > > >
>> > >
>> >
>>
>

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-02-01 Thread Ayush Saxena

There is some diff b/w the git tag & the src tar, the Dockerfile & the
create-release are different, Why?

Files hadoop-thirdparty/dev-support/bin/create-release and
hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release differ

Files hadoop-thirdparty/dev-support/docker/Dockerfile and
hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile differ


ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
hadoop-thirdparty/dev-support/bin/create-release
hadoop-thirdparty-1.2.0-src/dev-support/bin/create-release

444,446c444,446

< echo "RUN groupadd --non-unique -g ${group_id} ${user_name}"

< echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}"

< echo "RUN chown -R ${user_name} /home/${user_name}"

---

> echo "RUN groupadd --non-unique -g ${group_id} ${user_name}; exit 0;"

> echo "RUN useradd -g ${group_id} -u ${user_id} -m ${user_name}; exit
0;"

> echo "RUN chown -R ${user_name} /home/${user_name}; exit 0;"

ayushsaxena@ayushsaxena hadoop-thirdparty-1.2.0-RC0 % diff
hadoop-thirdparty/dev-support/docker/Dockerfile
hadoop-thirdparty-1.2.0-src/dev-support/docker/Dockerfile

103a104,105

> RUN rm -f /etc/maven/settings.xml && ln -s /home/root/.m2/settings.xml
/etc/maven/settings.xml

>

126a129,130

> RUN pip2 install setuptools-scm==5.0.2

> RUN pip2 install lazy-object-proxy==1.5.0

159d162

<




Other things look Ok,
* Built from source
* Verified Checksums
* Verified Signatures
* Validated files have ASF header

Not sure if having diff b/w the git tag & src tar is ok, this doesn't look
like core code change though, can anybody check & confirm?

-Ayush


On Thu, 1 Feb 2024 at 13:39, Xiaoqiao He  wrote:

> Gentle ping. @Ayush Saxena  @Steve Loughran
>  @inigo...@apache.org  @Masatake
> Iwasaki  and some other folks.
>
> On Wed, Jan 31, 2024 at 10:17 AM slfan1989  wrote:
>
> > Thank you for the review and vote! Looking forward to other forks helping
> > with voting and verification.
> >
> > Best Regards,
> > Shilun Fan.
> >
> > On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He 
> wrote:
> >
> > > Thanks Shilun for driving it and making it happen.
> > >
> > > +1(binding).
> > >
> > > [x] Checksums and PGP signatures are valid.
> > > [x] LICENSE files exist.
> > > [x] NOTICE is included.
> > > [x] Rat check is ok. `mvn clean apache-rat:check`
> > > [x] Built from source works well: `mvn clean install`
> > > [x] Built Hadoop trunk with updated thirdparty successfully (include
> > update
> > > protobuf shaded path).
> > >
> > > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we
> > > could finish this vote before 2024/02/06(UTC) if there are no concerns.
> > > Thanks all.
> > >
> > > Best Regards,
> > > - He Xiaoqiao
> > >
> > >
> > >
> > > On Mon, Jan 29, 2024 at 10:42 PM slfan1989 
> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Xiaoqiao He and I have put together a release candidate (RC0) for
> > Hadoop
> > > > Thirdparty 1.2.0.
> > > >
> > > > The RC is available at:
> > > >
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> > > >
> > > > The RC tag is
> > > >
> > >
> >
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> > > >
> > > > The maven artifacts are staged at
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1398
> > > >
> > > > Comparing to 1.1.1, there are three additional fixes:
> > > >
> > > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> > > > https://github.com/apache/hadoop-thirdparty/pull/26
> > > >
> > > > HADOOP-18921. Upgrade to avro 1.11.3
> > > > https://github.com/apache/hadoop-thirdparty/pull/24
> > > >
> > > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
> > > > https://github.com/apache/hadoop-thirdparty/pull/23
> > > >
> > > > You can find my public key at :
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> > >
> >
>

[jira] [Resolved] (HDFS-17359) EC: recheck failed streamers should only after flushing all packets.

2024-02-01 Thread Takanobu Asanuma (Jira)



 [ 
https://issues.apache.org/jira/browse/HDFS-17359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Takanobu Asanuma resolved HDFS-17359.
-
Fix Version/s: 3.3.9
   3.4.1
   3.5.0
   Resolution: Fixed

> EC: recheck failed streamers should only after flushing all packets.
> 
>
> Key: HDFS-17359
> URL: https://issues.apache.org/jira/browse/HDFS-17359
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: ec
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.3.9, 3.4.1, 3.5.0
>
>
> In method DFSStripedOutputStream#checkStreamerFailures, we have below codes:
> {code:java}
>     Set newFailed = checkStreamers();
>     if (newFailed.size() == 0) {
>       return;
>     }    if (isNeedFlushAllPackets) {
>       // for healthy streamers, wait till all of them have fetched the new 
> block
>       // and flushed out all the enqueued packets.
>       flushAllInternals();
>     }
>     // recheck failed streamers again after the flush
>     newFailed = checkStreamers(); {code}
> We should better move the re-check logic into if condition to reduce useless 
> invocation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17368) HA: Standy should exit safemode when resources are from low available

2024-02-01 Thread Zilong Zhu (Jira)

Zilong Zhu created HDFS-17368:
-

 Summary: HA: Standy should exit safemode when resources are from 
low available
 Key: HDFS-17368
 URL: https://issues.apache.org/jira/browse/HDFS-17368
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Zilong Zhu


The NameNodeResourceMonitor automatically enters safemode when it detects that 
the resources are not suffcient. NNRM is only in ANN. If both ANN and SNN enter 
SM due to low resources, and later SNN's disk space is restored, SNN willl 
become ANN and ANN will become SNN. However, at this point, SNN will not exit 
the SM, even if the disk is recovered.

Consider the following scenario:
 * Initially, nn-1 is active and nn-2 is standby. The insufficient resources of 
both nn-1 and nn-2 in dfs.namenode.name.dir, the NameNodeResourceMonitor 
detects the resource issue and puts nn01 into safemode.
 * At this point, nn-1 is in safemode (ON) and active, while nn-2 is in 
safemode (OFF) and standby.
 * After a period of time, the resources in nn-2's dfs.namenode.name.dir 
recover, triggering failover.
 * Now, nn-1 is in safe mode (ON) and standby, while nn-2 is in safe mode (OFF) 
and active.
 * Afterward, the resources in nn-1's dfs.namenode.name.dir recover.
 * However, since nn-1 is standby but in safemode (ON), it unable to exit safe 
mode automatically.

There are two possible ways fix this issues:
 # If SNN is detected to be in SM(because low resource), it will exit.
 # Or we already have HDFS-17231, we can revert HDFS-2914. Bringing NNRM back 
to SNN.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX

2024-02-01 Thread Hualong Zhang (Jira)

Hualong Zhang created HDFS-17367:


 Summary: Add PercentUsed for Different StorageTypes in JMX
 Key: HDFS-17367
 URL: https://issues.apache.org/jira/browse/HDFS-17367
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: namenode
Affects Versions: 3.5.0
Reporter: Hualong Zhang
Assignee: Hualong Zhang


Currently, the NameNode only displays PercentUsed for the entire cluster. We 
plan to add corresponding PercentUsed metrics for different StorageTypes.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

2024-02-01 Thread Xiaoqiao He

Gentle ping. @Ayush Saxena  @Steve Loughran
 @inigo...@apache.org  @Masatake
Iwasaki  and some other folks.

On Wed, Jan 31, 2024 at 10:17 AM slfan1989  wrote:

> Thank you for the review and vote! Looking forward to other forks helping
> with voting and verification.
>
> Best Regards,
> Shilun Fan.
>
> On Tue, Jan 30, 2024 at 6:20 PM Xiaoqiao He  wrote:
>
> > Thanks Shilun for driving it and making it happen.
> >
> > +1(binding).
> >
> > [x] Checksums and PGP signatures are valid.
> > [x] LICENSE files exist.
> > [x] NOTICE is included.
> > [x] Rat check is ok. `mvn clean apache-rat:check`
> > [x] Built from source works well: `mvn clean install`
> > [x] Built Hadoop trunk with updated thirdparty successfully (include
> update
> > protobuf shaded path).
> >
> > BTW, hadoop-thirdparty-1.2.0 will be included in release-3.4.0, hope we
> > could finish this vote before 2024/02/06(UTC) if there are no concerns.
> > Thanks all.
> >
> > Best Regards,
> > - He Xiaoqiao
> >
> >
> >
> > On Mon, Jan 29, 2024 at 10:42 PM slfan1989  wrote:
> >
> > > Hi folks,
> > >
> > > Xiaoqiao He and I have put together a release candidate (RC0) for
> Hadoop
> > > Thirdparty 1.2.0.
> > >
> > > The RC is available at:
> > >
> >
> https://dist.apache.org/repos/dist/dev/hadoop/hadoop-thirdparty-1.2.0-RC0
> > >
> > > The RC tag is
> > >
> >
> https://github.com/apache/hadoop-thirdparty/releases/tag/release-1.2.0-RC0
> > >
> > > The maven artifacts are staged at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1398
> > >
> > > Comparing to 1.1.1, there are three additional fixes:
> > >
> > > HADOOP-18197. Upgrade Protobuf-Java to 3.21.12
> > > https://github.com/apache/hadoop-thirdparty/pull/26
> > >
> > > HADOOP-18921. Upgrade to avro 1.11.3
> > > https://github.com/apache/hadoop-thirdparty/pull/24
> > >
> > > HADOOP-18843. Guava version 32.0.1 bump to fix CVE-2023-2976
> > > https://github.com/apache/hadoop-thirdparty/pull/23
> > >
> > > You can find my public key at :
> > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > >
> > > Best Regards,
> > > Shilun Fan.
> > >
> >
>

Re: Discussion about NameNode Fine-grained locking

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

Discussion about NameNode Fine-grained locking

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

[jira] [Resolved] (HDFS-17359) EC: recheck failed streamers should only after flushing all packets.

[jira] [Created] (HDFS-17368) HA: Standy should exit safemode when resources are from low available

[jira] [Created] (HDFS-17367) Add PercentUsed for Different StorageTypes in JMX

Re: [VOTE] Release Apache Hadoop Thirdparty 1.2.0 RC0

11 matches

Site Navigation

Mail list logo

Footer information