date:20240104

Re: Re: [DISCUSS] Release Hadoop 3.4.0

2024-01-04 Thread slfan1989

Hey all,

I plan to create a new branch of branch-3.4.0 this weekend and change the
version of the trunk branch to 3.5.0-SNAPSHOT.

If all goes well, I will release hadoop-3.4.0-RC0 early next week.

hadoop-3.4.0 contains many changes, we may release multiple RC versions.

Best Regards,
Shilun Fan.

> On Thu, 4 Jan 2024 at 15:53, PJ Fanning wrote:

> The new jar dependency just adds a class that Jackson JAX-RS support
> expects but that doesn't exist in the JAX-RS v1 jsr-311 jar [1].
> JAX-RS v2 uses a jar called rs-api instead of jsr-311. The package
> names are the same but there are differences (new classes and API
> changes in some other classes).

> Studying the Jackson change that broke support for JAX-RS v1 [2] and
> some basic testing seems to support the idea that having this one
> class available means that Hadoop and Tex should be able to use
> versions of Jackson later than the breaking change. Jackson JAX-RS
> code does not use the rs-api changes with the one exception of
> requiring the NoContentException. The Jackson change [2] appears in
> Jackson 2.13.0.

> [1]
https://github.com/pjfanning/jsr311-compat/blob/main/src/main/java/javax/ws/rs/core/

> NoContentException.java

> [2]
https://github.com/FasterXML/jackson-jaxrs-providers/issues/134#issuecomment-1180637522

> On Thu, 4 Jan 2024 at 19:30, Steve Loughran wrote:
>
>
>
> On Thu, 4 Jan 2024 at 15:53, PJ Fanning wrote:
>>
>> I would like to get some dependencies upgraded for Hadoop 3.4.0. For me,
it would be good to upgrade protobuf-java and Jackson to more secure
versions.
>>
>> For protobuf-java, that would involve releasing hadoop-thirdparty 1.2.0
[1], including merging the protobuf upgrade [2].
>
>
> I agree, we should update this. Anything else in there to update?
>>
>>
>> For Jackson, we are hampered by Jackson dropping support for
Jersey/JAX-RS 1.x. I have a workaround for this that I think is worth
further investigating [3].
>
>
> What does your new module do?
>>
>>
>> [1] https://github.com/apache/hadoop-thirdparty
>> [2] https://github.com/apache/hadoop-thirdparty/pull/19
>> [3] https://github.com/apache/hadoop/pull/6370
>>
>>
>>
>> On 2024/01/04 14:26:37 slfan1989 wrote:
>> > Hey all,
>> >
>> > We are planning to release Hadoop 3.4.0 base on trunk. I made some
>> > preparations and changed the target version of JIRA for non-blockers in
>> > HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to
create
>> > a new JIRA, the target version can directly select version 3.5.0.
>> >
>> > If you have any thoughts, suggestions, or concerns, please feel free to
>> > share them.
>> >
>> > Best Regards,
>> > Shilun Fan.
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2024-01-04 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/

[Jan 3, 2024, 12:49:52 PM] (github) HADOOP-18971. [ABFS] Read and cache file 
footer with fs.azure.footer.read.request.size (#6270)
[Jan 3, 2024, 1:12:38 PM] (github) Add synchronized on lockLeakCheck() because 
threadCountMap is not thread safe. (#6029)
[Jan 3, 2024, 4:07:51 PM] (github) HDFS-17310. DiskBalancer: Enhance the log 
message for submitPlan (#6391) Contributed by Haiyang Hu.




-1 overall


The following subsystems voted -1:
blanks hadolint pathlen spotbugs xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-common-project/hadoop-common/src/test/resources/xml/external-dtd.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

spotbugs :

   module:hadoop-hdfs-project/hadoop-hdfs 
   Dead store to sharedDirs in 
org.apache.hadoop.hdfs.server.namenode.NameNode.format(Configuration, boolean, 
boolean) At 
NameNode.java:org.apache.hadoop.hdfs.server.namenode.NameNode.format(Configuration,
 boolean, boolean) At NameNode.java:[line 1383] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   
org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT 
isn't final but should be At TimelineConnector.java:be At 
TimelineConnector.java:[line 82] 

spotbugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common 
   
org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT 
isn't final but should be At TimelineConnector.java:be At 
TimelineConnector.java:[line 82] 

spotbugs :

   module:hadoop-hdfs-project 
   Dead store to sharedDirs in 
org.apache.hadoop.hdfs.server.namenode.NameNode.format(Configuration, boolean, 
boolean) At 
NameNode.java:org.apache.hadoop.hdfs.server.namenode.NameNode.format(Configuration,
 boolean, boolean) At NameNode.java:[line 1383] 

spotbugs :

   module:hadoop-yarn-project 
   
org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT 
isn't final but should be At TimelineConnector.java:be At 
TimelineConnector.java:[line 82] 

spotbugs :

   module:root 
   Dead store to sharedDirs in 
org.apache.hadoop.hdfs.server.namenode.NameNode.format(Configuration, boolean, 
boolean) At 
NameNode.java:org.apache.hadoop.hdfs.server.namenode.NameNode.format(Configuration,
 boolean, boolean) At NameNode.java:[line 1383] 
   
org.apache.hadoop.yarn.client.api.impl.TimelineConnector.DEFAULT_SOCKET_TIMEOUT 
isn't final but should be At TimelineConnector.java:be At 
TimelineConnector.java:[line 82] 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/results-compile-cc-root.txt
 [96K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/results-compile-javac-root.txt
 [12K]

   blanks:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/blanks-eol.txt
 [15M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/blanks-tabs.txt
 [2.0M]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/results-checkstyle-root.txt
 [13M]

   hadolint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/results-hadolint.txt
 [24K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/results-pathlen.txt
 [16K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/results-pylint.txt
 [20K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/1461/artifact/out/results-shellcheck.txt
 [24K]

   xml:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-li

Re: Re: [DISCUSS] Release Hadoop 3.4.0

2024-01-04 Thread slfan1989

Hey all,

We are planning to release Hadoop 3.4.0 base on trunk. I made some
preparations and changed the target version of JIRA for non-blockers in
HADOOP, HDFS, YARN, and MAPREDUCE from 3.4.0 to 3.5.0. If we want to create
a new JIRA, the target version can directly select version 3.5.0.

If you have any thoughts, suggestions, or concerns, please feel free to
share them.

Best Regards,
Shilun Fan.

> +1 from me.
>> It will include the new AWS V2 SDK upgrade as well.

> On Wed, Jan 3, 2024 at 6:35 AM Xiaoqiao He wrote:

> >
> > I think the release discussion can be in public ML?
>
> Good idea. cc common-dev/hdfs-dev/yarn-dev/mapreduce-dev ML.
>
> Best Regards,
> - He Xiaoqiao
>
> On Tue, Jan 2, 2024 at 6:18 AM Ayush Saxena wrote:
>
> > +1 from me as well.
> >
> > We should definitely attempt to upgrade the thirdparty version for
> > 3.4.0 & check if there are any pending critical/blocker issues as
> > well.
> >
> > I think the release discussion can be in public ML?
> >
> > -Ayush
> >
> > On Mon, 1 Jan 2024 at 18:25, Steve Loughran  >
> > wrote:
> > >
> > > +1 from me
> > >
> > > ant and maven repo to build and validate things, including making arm
> > > binaries if you work from an arm macbook.
> > > https://github.com/steveloughran/validate-hadoop-client-artifacts
> > >
> > > do we need to publish an up to date thirdparty release for this?
> > >
> > >
> > >
> > > On Mon, 25 Dec 2023 at 16:06, slfan1989 wrote:
> > >
> > > > Dear PMC Members,
> > > >
> > > > First of all, Merry Christmas to everyone!
> > > >
> > > > In our community discussions, we collectively finalized the plan to
> > release
> > > > Hadoop 3.4.0 based on the current trunk branch. I am applying to
take
> > on
> > > > the responsibility for the initial release of version 3.4.0, and the
> > entire
> > > > process is set to officially commence in January 2024.
> > > > I have created a new JIRA: HADOOP-19018. Release 3.4.0.
> > > >
> > > > The specific work plan includes:
> > > >
> > > > 1. Following the guidance in the HowToRelease document, completing
> all
> > the
> > > > relevant tasks required for the release of version 3.4.0.
> > > > 2. Pointing the trunk branch to 3.5.0-SNAPSHOT.
> > > > 3. Currently, the Fix Versions of all tasks merged into trunk are
set
> > as
> > > > 3.4.0; I will move them to 3.5.0.
> > > >
> > > > Confirmed features to be included in the release:
> > > >
> > > > 1. Enhanced functionality for YARN Federation.
> > > > 2. Optimization of HDFS RBF.
> > > > 3. Introduction of fine-grained global locks for DataNodes.
> > > > 4. Improvements in the stability of HDFS EC, and more.
> > > > 5. Fixes for important CVEs.
> > > >
> > > > If you have any thoughts, suggestions, or concerns, please feel free
> to
> > > > share them.
> > > >
> > > > Looking forward to a successful release!
> > > >
> > > > Best Regards,
> > > > Shilun Fan.
> > > >
> >
>

[jira] [Created] (YARN-11640) capacity scheduler supports application priority with FairOrderingPolicy

2024-01-04 Thread Ming Chen (Jira)

Ming Chen created YARN-11640:


 Summary: capacity scheduler supports application priority with 
FairOrderingPolicy
 Key: YARN-11640
 URL: https://issues.apache.org/jira/browse/YARN-11640
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: yarn
Reporter: Ming Chen






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-2098) App priority support in Fair Scheduler

2024-01-04 Thread Shilun Fan (Jira)



 [ 
https://issues.apache.org/jira/browse/YARN-2098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shilun Fan resolved YARN-2098.
--
Target Version/s:   (was: 3.5.0)
  Resolution: Done

> App priority support in Fair Scheduler
> --
>
> Key: YARN-2098
> URL: https://issues.apache.org/jira/browse/YARN-2098
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Reporter: Ashwin Shankar
>Priority: Major
>  Labels: pull-request-available
> Attachments: YARN-2098.patch, YARN-2098.patch
>
>
> This jira is created for supporting app priorities in fair scheduler. 
> AppSchedulable hard codes priority of apps to 1, we should change this to get 
> priority from ApplicationSubmissionContext.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2024-01-04 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/

No changes




-1 overall


The following subsystems voted -1:
asflicense hadolint mvnsite pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.fs.TestFileUtil 
   hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.TestLeaseRecovery2 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotDeletion 
   hadoop.hdfs.TestFileLengthOnClusterRestart 
   hadoop.hdfs.server.namenode.ha.TestPipelinesFailover 
   hadoop.hdfs.TestDFSInotifyEventInputStream 
   hadoop.hdfs.server.namenode.snapshot.TestSnapshotBlocksMap 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.fs.viewfs.TestViewFileSystemHdfs 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.contrib.bkjournal.TestBootstrapStandbyWithBKJM 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.mapreduce.lib.input.TestLineRecordReader 
   hadoop.mapred.TestLineRecordReader 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.tools.util.TestProducerConsumer 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
   hadoop.yarn.sls.appmaster.TestAMSimulator 
   hadoop.yarn.sls.TestSLSRunner 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceAllocator
 
   
hadoop.yarn.server.nodemanager.containermanager.linux.resources.TestNumaResourceHandlerImpl
 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore 
   
hadoop.yarn.server.resourcemanager.monitor.invariants.TestMetricsInvariantChecker
 
  

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/diff-compile-javac-root.txt
  [488K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/diff-checkstyle-root.txt
  [14M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   mvnsite:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/patch-mvnsite-root.txt
  [572K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/diff-patch-shellcheck.txt
  [72K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/patch-javadoc-root.txt
  [36K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [220K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [1.8M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [36K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [24K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/1261/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
  [104K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64

[jira] [Resolved] (YARN-11529) Add metrics for ContainerMonitorImpl.

2024-01-04 Thread Shilun Fan (Jira)



 [ 
https://issues.apache.org/jira/browse/YARN-11529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shilun Fan resolved YARN-11529.
---
Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Add metrics for ContainerMonitorImpl.
> -
>
> Key: YARN-11529
> URL: https://issues.apache.org/jira/browse/YARN-11529
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: nodemanager
>Affects Versions: 3.4.0
>Reporter: Xianming Lei
>Assignee: Xianming Lei
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> In our production environment, we have ample machine resources and a 
> significant number of active Containers. However, the MonitoringThread in 
> ContainerMonitorImpl experiences significant latency during each execution. 
> To address this, it is highly recommended to incorporate metrics for 
> monitoring the duration of this time-consuming process.
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Re: [DISCUSS] Release Hadoop 3.4.0

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

Re: Re: [DISCUSS] Release Hadoop 3.4.0

[jira] [Created] (YARN-11640) capacity scheduler supports application priority with FairOrderingPolicy

[jira] [Resolved] (YARN-2098) App priority support in Fair Scheduler

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

[jira] [Resolved] (YARN-11529) Add metrics for ContainerMonitorImpl.

7 matches

Site Navigation

Mail list logo

Footer information