date:20210203

Re: [ANNOUNCE] Mukund Thakur is a new Apache Hadoop Committer

2021-02-03 Thread Mukund Madhav Thakur

Thanks everyone :)

On Wed, Feb 3, 2021 at 8:14 PM Steve Loughran 
wrote:

> Many congratulations Mukund!
> I look forward to you vetoing my PRs on the basis of quality and test
> coverage.
>
> -steve
>
> On Wed, 3 Feb 2021 at 11:39, hemanth boyina 
> wrote:
>
> > Congratulations Mukund !!
> >
> >
> >
> > Thanks
> > HemanthBoyina
> >
> > On Wed, 3 Feb 2021, 11:11 Mingliang Liu,  wrote:
> >
> > > Hi all,
> > >
> > > It's my pleasure to announce that Mukund Thakur has been elected as a
> > > committer on the Apache Hadoop project recognizing his continued
> > > contributions to the project. He has accepted the invitation.
> > >
> > >
> > > His involvement in the hadoop object store modules, hadoop common and
> > > distcp tool is a great addition to the Hadoop project. His reviews are
> > > comprehensive with high standards to "approve a pull request". The
> > granting
> > > of committership to him will enable better productivity.
> > >
> > >
> > > Please join me in congratulating him. Welcome aboard, Mukund!
> > >
> > >
> > >
> > > Mingliang Liu
> > >
> > > (on behalf of the Apache Hadoop PMC)
> > >
> >
>

[jira] [Resolved] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement

2021-02-03 Thread Ahmed Hussein (Jira)



 [ 
https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved YARN-10352.
--
Resolution: Fixed

> Skip schedule on not heartbeated nodes in Multi Node Placement
> --
>
> Key: YARN-10352
> URL: https://issues.apache.org/jira/browse/YARN-10352
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Prabhu Joseph
>Assignee: Prabhu Joseph
>Priority: Major
>  Labels: capacityscheduler, multi-node-placement
> Fix For: 3.4.0
>
> Attachments: YARN-10352-001.patch, YARN-10352-002.patch, 
> YARN-10352-003.patch, YARN-10352-004.patch, YARN-10352-005.patch, 
> YARN-10352-006.patch, YARN-10352-007.patch, YARN-10352-008.patch, 
> YARN-10352-010.patch, YARN-10352.009.patch
>
>
> When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM 
> Active Nodes will be still having those stopped nodes until NM Liveliness 
> Monitor Expires after configured timeout 
> (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, 
> Multi Node Placement assigns the containers on those nodes. They need to 
> exclude the nodes which has not heartbeated for configured heartbeat interval 
> (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to 
> Asynchronous Capacity Scheduler Threads. 
> (CapacityScheduler#shouldSkipNodeSchedule)
> *Repro:*
> 1. Enable Multi Node Placement 
> (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery 
> Enabled  (yarn.node.recovery.enabled)
> 2. Have only one NM running say worker0
> 3. Stop worker0 and start any other NM say worker1
> 4. Submit a sleep job. The containers will timeout as assigned to stopped NM 
> worker0.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format

2021-02-03 Thread Ahmed Hussein (Jira)



 [ 
https://issues.apache.org/jira/browse/YARN-10585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ahmed Hussein resolved YARN-10585.
--
Resolution: Fixed

> Create a class which can convert from legacy mapping rule format to the new 
> JSON format
> ---
>
> Key: YARN-10585
> URL: https://issues.apache.org/jira/browse/YARN-10585
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10585.001.patch, YARN-10585.002.patch, 
> YARN-10585.003.patch
>
>
> To make transition easier we need to create tooling to support the migration 
> effort. The first step is to create a class which can migrate from legacy to 
> the new JSON format.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Resolved] (YARN-10601) The Yarn client should use the UGI who created the Yarn client for obtaining a delegation token for the remote log dir

2021-02-03 Thread Daniel Fritsi (Jira)



 [ 
https://issues.apache.org/jira/browse/YARN-10601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Fritsi resolved YARN-10601.
--
Resolution: Invalid

See my previous comment

> The Yarn client should use the UGI who created the Yarn client for obtaining 
> a delegation token for the remote log dir
> --
>
> Key: YARN-10601
> URL: https://issues.apache.org/jira/browse/YARN-10601
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: log-aggregation
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Daniel Fritsi
>Priority: Critical
>
> It seems there was a bug introduced in YARN-10333 in this section of 
> *{color:#0747A6}{{addLogAggregationDelegationToken}}{color}*:
> {code:java}
> Path remoteRootLogDir = fileController.getRemoteRootLogDir();
> FileSystem fs = remoteRootLogDir.getFileSystem(conf);
> final org.apache.hadoop.security.token.Token[] finalTokens =
> fs.addDelegationTokens(masterPrincipal, credentials);
> {code}
> *{color:#0747A6}{{remoteRootLogDir.getFileSystem}}{color}* simply does this:
> {code:java}
> public FileSystem getFileSystem(Configuration conf) throws IOException {
>   return FileSystem.get(this.toUri(), conf);
> }
> {code}
> As far as I know it's customary to create a YarnClient instance via 
> *{color:#0747A6}{{YarnClient.createYarnClient()}}{color}* in a 
> UserGroupInformation.doAs block if you would like to use it with a different 
> user then the current one. E.g.:
> {code:java}
> YarnClient yarnClient = ugi.doAs(new PrivilegedExceptionAction() {
> @Override
> public YarnClient run() throws Exception {
> YarnClient yarnClient = YarnClient.createYarnClient();
> yarnClient.init(conf);
> yarnClient.start();
> return yarnClient;
> }
> });
> {code}
> If this statement is correct then I think YarnClient should save the 
> *{color:#0747A6}{{UserGroupInformation.getCurrentUser()}}{color}* when the 
> YarnClient is being created and the 
> *{color:#0747A6}{{remoteRootLogDir.getFileSystem(conf)}}{color}* call should 
> be made inside an ugi.doAs block with that saved user.
> A more concrete example:
> {code:java}
> public YarnClient createYarnClient(UserGroupInformation ugi, Configuration 
> conf) throws Exception {
> return ugi.doAs((PrivilegedExceptionAction) () -> {
> // Her I am the submitterUser (see below)
> YarnClient yarnClient = YarnClient.createYarnClient();
> yarnClient.init(conf);
> yarnClient.start();
> return yarnClient;
> });
> }
> public void run() {
> // Here I am the serviceUser
> // ...
> Configuration conf = ...
> // ...
> UserGroupInformation ugi = getSubmitterUser();
> // ...
> YarnClient yarnClient = createYarnClient(ugi);
> // ...
> ApplicationSubmissionContext context = ...
> // ...
> yarnClient.submitApplication(context);
> }
> {code}
> As you can see *{color:#0747A6}{{submitApplication}}{color}* is not invoked 
> inside an ugi.doAs block and submitApplication is the one who will eventually 
> invoke *{color:#0747A6}{{addLogAggregationDelegationToken}}{color}*. That's 
> why we need to save the UGI during the YarnClient creation and create the 
> FileSystem instance inside an ugi.doAs with that saved user. Otherwise Yarn 
> will try to get a delegation token with an incorrect user (serviceUser) 
> instead of the submitterUser.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10613) Config to allow Intra-queue preemption to enable/disable conservativeDRF

2021-02-03 Thread Eric Payne (Jira)

Eric Payne created YARN-10613:
-

 Summary: Config to allow Intra-queue preemption to  enable/disable 
conservativeDRF
 Key: YARN-10613
 URL: https://issues.apache.org/jira/browse/YARN-10613
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: capacity scheduler, scheduler preemption
Affects Versions: 2.10.1, 3.1.4, 3.2.2, 3.3.0
Reporter: Eric Payne
Assignee: Eric Payne


YARN-8292 added code that prevents CS intra-queue preemption from preempting 
containers from an app unless all of the major resources used by the app are 
greater than the user limit for that user.

Ex:
| Used | User Limit |
| <58GB, 58> | <30GB, 300> |

In this example, only used memory is above the user limit, not used vcores. So, 
intra-queue preemption will not occur.

YARN-8292 added the {{conservativeDRF}} flag to 
{{CapacitySchedulerPreemptionUtils#tryPreemptContainerAndDeductResToObtain}}. 
If {{conservativeDRF}} is false, containers will be preempted from apps in the 
example state. If true, containers will not be preempted.

This flag is hard-coded to false for Inter-queue (cross-queue) preemption and 
true for intra-queue (in-queue) preemption.

I propose that in some cases, we want intra-queue preemption to be more 
aggressive and preempt in the example case. To accommodate that, I propose the 
addition of the following config property:
{code:xml}
  

yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.conservative-drf
true
  
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

2021-02-03 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/

[Feb 1, 2021 3:28:29 PM] (Xiaoqiao He) HDFS-15792. ClasscastException while 
loading FSImage. Contributed by Renukaprasad C.
[Feb 1, 2021 5:40:01 PM] (noreply) HADOOP-17482: Remove Commons Logger from 
FileSystem Class (#2633)
[Feb 2, 2021 5:33:17 AM] (noreply) HADOOP-17508. Simplify deps install 
instructions (#2664)
[Feb 2, 2021 5:50:54 AM] (Xiaoqiao He) HDFS-15792. Addendum: ClasscastException 
while loading FSImage. Contributed by Renukaprasad C.
[Feb 2, 2021 8:07:25 AM] (Szilard Nemeth) YARN-10604. Support auto queue 
creation without mapping rules. Contributed by Andras Gyori
[Feb 2, 2021 8:13:46 AM] (Szilard Nemeth) YARN-10605. Add 
queue-mappings-override.enable property in FS2CS conversions. Contributed by 
Andras Gyori
[Feb 2, 2021 9:02:09 AM] (noreply) HDFS-15795. EC: Wrong checksum when 
reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657)
[Feb 2, 2021 3:09:30 PM] (noreply) HADOOP-17509. Parallelize building of 
dependencies (#2669)
[Feb 2, 2021 6:13:41 PM] (noreply) HADOOP-15710. ABFS checkException to map 403 
to AccessDeniedException. (#2648)
[Feb 3, 2021 3:20:45 AM] (Bibin Chundatt) YARN-10352 Skip schedule on not 
heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi 
Zhu




-1 overall


The following subsystems voted -1:
mvnsite pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

Failed junit tests :

   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.client.TestRMFailoverProxyProvider 
   hadoop.yarn.client.TestNoHaRMFailoverProxyProvider 
   
hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage
 
   hadoop.yarn.server.router.clientrm.TestFederationClientInterceptor 
   
hadoop.yarn.server.timelineservice.documentstore.TestDocumentStoreCollectionCreator
 
   
hadoop.yarn.server.timelineservice.documentstore.TestDocumentStoreTimelineReaderImpl
 
   
hadoop.yarn.server.timelineservice.documentstore.TestDocumentStoreTimelineWriterImpl
 
   
hadoop.yarn.server.timelineservice.documentstore.writer.cosmosdb.TestCosmosDBDocumentStoreWriter
 
   
hadoop.yarn.server.timelineservice.documentstore.reader.cosmosdb.TestCosmosDBDocumentStoreReader
 
   hadoop.tools.dynamometer.TestDynamometerInfra 
   hadoop.tools.dynamometer.TestDynamometerInfra 
  

   cc:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/diff-compile-cc-root.txt
 [116K]

   javac:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/diff-compile-javac-root.txt
 [596K]

   checkstyle:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/diff-checkstyle-root.txt
 [16M]

   mvnsite:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/patch-mvnsite-root.txt
 [500K]

   pathlen:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/pathlen.txt
 [12K]

   pylint:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/diff-patch-pylint.txt
 [60K]

   shellcheck:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/diff-patch-shellcheck.txt
 [28K]

   shelldocs:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/diff-patch-shelldocs.txt
 [96K]

   whitespace:

  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/whitespace-eol.txt
 [13M]
  
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java11-linux-x86_64/116/artifact/out/whitespace-tabs.txt
 [2.0M]

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

2021-02-03 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/406/

[Feb 2, 2021 5:33:17 AM] (noreply) HADOOP-17508. Simplify deps install 
instructions (#2664)
[Feb 2, 2021 5:50:54 AM] (Xiaoqiao He) HDFS-15792. Addendum: ClasscastException 
while loading FSImage. Contributed by Renukaprasad C.
[Feb 2, 2021 8:07:25 AM] (Szilard Nemeth) YARN-10604. Support auto queue 
creation without mapping rules. Contributed by Andras Gyori
[Feb 2, 2021 8:13:46 AM] (Szilard Nemeth) YARN-10605. Add 
queue-mappings-override.enable property in FS2CS conversions. Contributed by 
Andras Gyori
[Feb 2, 2021 9:02:09 AM] (noreply) HDFS-15795. EC: Wrong checksum when 
reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657)
[Feb 2, 2021 3:09:30 PM] (noreply) HADOOP-17509. Parallelize building of 
dependencies (#2669)
[Feb 2, 2021 6:13:41 PM] (noreply) HADOOP-15710. ABFS checkException to map 403 
to AccessDeniedException. (#2648)




-1 overall


The following subsystems voted -1:
findbugs pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-excerpt.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-output-missing-tags2.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/resources/nvidia-smi-sample-output.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/fair-scheduler-invalid.xml
 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/resources/yarn-site-with-invalid-allocation-file-ref.xml
 

findbugs :

   module:hadoop-yarn-project/hadoop-yarn 
   Possible null pointer dereference of 
LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at 
LegacyMappingRuleToJson.java:LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at LegacyMappingRuleToJson.java:[line 151] 

findbugs :

   module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server 
   Possible null pointer dereference of 
LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at 
LegacyMappingRuleToJson.java:LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at LegacyMappingRuleToJson.java:[line 151] 

findbugs :

   
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 
   Possible null pointer dereference of 
LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at 
LegacyMappingRuleToJson.java:LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at LegacyMappingRuleToJson.java:[line 151] 

findbugs :

   module:hadoop-yarn-project 
   Possible null pointer dereference of 
LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at 
LegacyMappingRuleToJson.java:LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at LegacyMappingRuleToJson.java:[line 151] 

findbugs :

   module:root 
   Possible null pointer dereference of 
LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at 
LegacyMappingRuleToJson.java:LegacyMappingRuleToJson.userGroupMappingRules in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.placement.converter.LegacyMappingRuleToJson.convert()
 Dereferenced at

[jira] [Created] (YARN-10612) Fix find bugs issue introduced in YARN-10585

2021-02-03 Thread Gergely Pollak (Jira)

Gergely Pollak created YARN-10612:
-

 Summary: Fix find bugs issue introduced in YARN-10585
 Key: YARN-10612
 URL: https://issues.apache.org/jira/browse/YARN-10612
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Gergely Pollak






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-10611) Fix that shaded should be used for google guava imports in YARN-10352.

2021-02-03 Thread Qi Zhu (Jira)

Qi Zhu created YARN-10611:
-

 Summary: Fix that shaded should be used for google guava imports 
in YARN-10352.
 Key: YARN-10611
 URL: https://issues.apache.org/jira/browse/YARN-10611
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Qi Zhu
Assignee: Qi Zhu






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [ANNOUNCE] Mukund Thakur is a new Apache Hadoop Committer

2021-02-03 Thread Steve Loughran

Many congratulations Mukund!
I look forward to you vetoing my PRs on the basis of quality and test
coverage.

-steve

On Wed, 3 Feb 2021 at 11:39, hemanth boyina 
wrote:

> Congratulations Mukund !!
>
>
>
> Thanks
> HemanthBoyina
>
> On Wed, 3 Feb 2021, 11:11 Mingliang Liu,  wrote:
>
> > Hi all,
> >
> > It's my pleasure to announce that Mukund Thakur has been elected as a
> > committer on the Apache Hadoop project recognizing his continued
> > contributions to the project. He has accepted the invitation.
> >
> >
> > His involvement in the hadoop object store modules, hadoop common and
> > distcp tool is a great addition to the Hadoop project. His reviews are
> > comprehensive with high standards to "approve a pull request". The
> granting
> > of committership to him will enable better productivity.
> >
> >
> > Please join me in congratulating him. Welcome aboard, Mukund!
> >
> >
> >
> > Mingliang Liu
> >
> > (on behalf of the Apache Hadoop PMC)
> >
>

Re: [DISCUSS] Hadoop 3.3.1 release

2021-02-03 Thread Steve Loughran

yeah, it's time.I've been trying to get things lined up for this
object-store wise, a few things tagged as blockers where I really want them
in

Wei Chiu -if you want to be the release manager I'll share the workload,

Regarding blockers : how about we have a little hackathon where we try
and get things in. This means a promise of review time from the people with
commit rights and other people who understand the code (Stack?)

-steve

On Thu, 28 Jan 2021 at 06:48, Ayush Saxena  wrote:

> +1
> Just to mention we would need to release hadoop-thirdparty too before.
> Presently we are using the snapshot version of it.
>
> -Ayush
>
> > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang  wrote:
> >
> > Hi all,
> >
> > Hadoop 3.3.0 was released half a year ago, and as of now we've
> accumulated
> > more than 400 changes in the branch-3.3. A number of downstreamers are
> > eagerly waiting for 3.3.1 which addresses the guava version conflict
> issue.
> >
> >
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20and%20fixVersion%20in%20(3.3.1)%20and%20status%20%3D%20Resolved%20
> >
> > We should start the release work for 3.3.1 before the diff becomes even
> > larger.
> >
> > I believe there are  currently only two real blockers for a 3.3.1 (using
> > this filter
> >
> https://issues.apache.org/jira/issues/?filter=-1=project%20in%20(HDFS%2C%20HADOOP%2C%20YARN%2C%20MAPREDUCE)%20AND%20cf%5B12310320%5D%20in%20(3.3.1)%20AND%20status%20not%20in%20(Resolved)%20ORDER%20BY%20priority%20DESC
> > )
> >
> >
> >   1. HDFS-15566 
> >   2.
> >  1. HADOOP-17112  >
> > 2.
> >
> >
> >
> > Is there anyone who would volunteer to be the 3.3.1 RM?
> >
> > Also, the HowToRelease wiki does not describe the ARM build process.
> That's
> > going to be important for future releases.
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

2021-02-03 Thread Apache Jenkins Server

For more details, see 
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/

[Feb 1, 2021 11:43:44 PM] (noreply) HDFS-15801. Backport HDFS-14582 to 
branch-2.10 (Failed to start DN with ArithmeticException when NULL checksum 
used) (#2659)




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint jshint pathlen unit xml


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

XML :

   Parsing Error(s): 
   hadoop-build-tools/src/main/resources/checkstyle/checkstyle.xml 
   hadoop-build-tools/src/main/resources/checkstyle/suppressions.xml 
   
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/empty-configuration.xml
 
   hadoop-tools/hadoop-azure/src/config/checkstyle-suppressions.xml 
   hadoop-tools/hadoop-azure/src/config/checkstyle.xml 
   hadoop-tools/hadoop-resourceestimator/src/config/checkstyle.xml 
   hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/public/crossdomain.xml 
   
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/src/main/webapp/public/crossdomain.xml
 

Failed junit tests :

   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   
hadoop.hdfs.server.blockmanagement.TestReplicationPolicyWithUpgradeDomain 
   hadoop.hdfs.qjournal.server.TestJournalNodeRespectsBindHostKeys 
   hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints 
   hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat 
   hadoop.hdfs.server.federation.resolver.order.TestLocalResolver 
   hadoop.hdfs.server.federation.router.TestRouterQuota 
   hadoop.hdfs.server.federation.resolver.TestMultipleDestinationResolver 
   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer 
   hadoop.yarn.server.resourcemanager.TestClientRMService 
   hadoop.mapreduce.jobhistory.TestHistoryViewerPrinter 
   hadoop.tools.TestDistCpSystem 
   hadoop.resourceestimator.service.TestResourceEstimatorService 
   hadoop.resourceestimator.solver.impl.TestLpSolver 
  

   jshint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-patch-jshint.txt
  [208K]

   cc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-compile-javac-root.txt
  [456K]

   checkstyle:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-checkstyle-root.txt
  [16M]

   hadolint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-patch-hadolint.txt
  [4.0K]

   pathlen:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-patch-pylint.txt
  [60K]

   shellcheck:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-patch-shellcheck.txt
  [56K]

   shelldocs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-patch-shelldocs.txt
  [8.0K]

   whitespace:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/whitespace-eol.txt
  [12M]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/whitespace-tabs.txt
  [1.3M]

   xml:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/xml.txt
  [4.0K]

   javadoc:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/diff-javadoc-javadoc-root.txt
  [20K]

   findbugs:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/branch-findbugs-root.txt
  [48K]

   unit:

   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [272K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs_src_contrib_bkjournal.txt
  [12K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
  [44K]
   
https://ci-hadoop.apache.org/job/hadoop-qbt-branch-2.10-java7-linux-x86_64/197/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt

Re: [ANNOUNCE] Mukund Thakur is a new Apache Hadoop Committer

2021-02-03 Thread hemanth boyina

Congratulations Mukund !!



Thanks
HemanthBoyina

On Wed, 3 Feb 2021, 11:11 Mingliang Liu,  wrote:

> Hi all,
>
> It's my pleasure to announce that Mukund Thakur has been elected as a
> committer on the Apache Hadoop project recognizing his continued
> contributions to the project. He has accepted the invitation.
>
>
> His involvement in the hadoop object store modules, hadoop common and
> distcp tool is a great addition to the Hadoop project. His reviews are
> comprehensive with high standards to "approve a pull request". The granting
> of committership to him will enable better productivity.
>
>
> Please join me in congratulating him. Welcome aboard, Mukund!
>
>
>
> Mingliang Liu
>
> (on behalf of the Apache Hadoop PMC)
>

Re: [ANNOUNCE] Mukund Thakur is a new Apache Hadoop Committer

[jira] [Resolved] (YARN-10352) Skip schedule on not heartbeated nodes in Multi Node Placement

[jira] [Resolved] (YARN-10585) Create a class which can convert from legacy mapping rule format to the new JSON format

[jira] [Resolved] (YARN-10601) The Yarn client should use the UGI who created the Yarn client for obtaining a delegation token for the remote log dir

[jira] [Created] (YARN-10613) Config to allow Intra-queue preemption to enable/disable conservativeDRF

Apache Hadoop qbt Report: trunk+JDK11 on Linux/x86_64

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86_64

[jira] [Created] (YARN-10612) Fix find bugs issue introduced in YARN-10585

[jira] [Created] (YARN-10611) Fix that shaded should be used for google guava imports in YARN-10352.

Re: [ANNOUNCE] Mukund Thakur is a new Apache Hadoop Committer

Re: [DISCUSS] Hadoop 3.3.1 release

Apache Hadoop qbt Report: branch-2.10+JDK7 on Linux/x86_64

Re: [ANNOUNCE] Mukund Thakur is a new Apache Hadoop Committer

13 matches

Site Navigation

Mail list logo

Footer information