[jira] [Commented] (YARN-8981) Virtual IP address support
[ https://issues.apache.org/jira/browse/YARN-8981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683355#comment-16683355 ] Xun Liu commented on YARN-8981: --- hi, [~Tagar] The virtual IP inside the YARN cluster is already supported by a parallel network created by calico. You can refer to the installation instructions for the Docker environment: [https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/src/site/markdown/InstallationScriptEN.md] And install using the runtime environment installation script; [https://github.com/apache/hadoop/tree/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine/installation] It is possible to have the Docker container in YARN run in an internal virtual parallel network. However, after YARN introduced Docker, there are still some features that need to be strengthened in the support of the network. For example, how the services in the internally virtualized Docker container access the network environment outside the cluster. In Kubernetes, the reverse proxy can be performed by the ingress latter Traefik. > Virtual IP address support > -- > > Key: YARN-8981 > URL: https://issues.apache.org/jira/browse/YARN-8981 > Project: Hadoop YARN > Issue Type: New Feature >Affects Versions: 3.0.2, 3.1.1 >Reporter: Ruslan Dautkhanov >Priority: Major > Labels: DNS, Docker, docker, service, service-engine, > service-orchestration, virtual_hosts > > I couldn't find support for virtual IP addresses in YARN framework. > This would be great if we have a docker-on-yarn service and if it for example > has to be failed over to another physical host, clients can still find it. > So the idea is for YARN to bring up that virtual IP address (an > additional/secondary IP address ) on a physical host where that particular > docker container is running, so the clients that use that container's > services don't have to change connection details every time that container > moves around in YARN cluster. > Similarly to virtual IP addresses in Kubernetes world: > [https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies] > One implementation could be through `ip address add` \ `ip address remove`. > Kubernetes uses a more complicated `kube-proxy`, similarly to `docker-proxy` > process in pure dockers / non-kubernetes docker deployments. > Another approach is running a separate DNS service for a DNS subdomain (main > DNS server would have to forward all requests for that DNS subdomain to a > YARN DNS service). In Oracle Clusterware similar process is called GNS: > https://docs.oracle.com/en/database/oracle/oracle-database/12.2/cwsol/about-the-grid-naming-service-vip-address.html#GUID-A4EE0CC6-A5F1-4507-82D6-D5C43E0F1584 > Would be great to have support for either virtual IP addresses managed by > YARN directly or something similar to Oracle's GNS dns service. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-2823) NullPointerException in RM HA enabled 3-node cluster
[
https://issues.apache.org/jira/browse/YARN-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683225#comment-16683225
]
Paul Lin edited comment on YARN-2823 at 11/12/18 6:57 AM:
--
[~imstefanlee] Hi, I'm facing the same issue with Flink applications. I tried
explicitly setting `KeepContainersAcrossApplicationAttempts` to false, but it
doesn't work. How do you solve the problem at last? And could you please point
me to the code where the default value of
`KeepContainersAcrossApplicationAttempts` is set to true? Thanks a lot!
was (Author: paul lin):
[~imstefanlee] Hi, I'm facing the same issue with Flink applications. I tried
explicitly setting `KeepContainersAcrossApplicationAttempts` to false, but it
doesn't work. How do you solve the problem at last? And could you please point
me to the code where the default value KeepContainersAcrossApplicationAttempts
is set to true? Thanks a lot!
> NullPointerException in RM HA enabled 3-node cluster
>
>
> Key: YARN-2823
> URL: https://issues.apache.org/jira/browse/YARN-2823
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Gour Saha
>Assignee: Jian He
>Priority: Critical
> Fix For: 2.6.0
>
> Attachments: YARN-2823.1.patch, logs_with_NPE_in_RM.zip
>
>
> Branch:
> 2.6.0
> Environment:
> A 3-node cluster with RM HA enabled. The HA setup went pretty smooth (used
> Ambari) and then installed HBase using Slider. After some time the RMs went
> down and would not come back up anymore. Following is the NPE we see in both
> the RM logs.
> {noformat}
> 2014-09-16 01:36:28,037 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(612)) - Error in handling event type
> APP_ATTEMPT_ADDED to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.transferStateFromPreviousAttempt(SchedulerApplicationAttempt.java:530)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:678)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1015)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:98)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:603)
> at java.lang.Thread.run(Thread.java:744)
> 2014-09-16 01:36:28,042 INFO resourcemanager.ResourceManager
> (ResourceManager.java:run(616)) - Exiting, bbye..
> {noformat}
> All the logs for this 3-node cluster has been uploaded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-8406) Do the improvement to the FSLeafQueue about calculating fair share for apps
[ https://issues.apache.org/jira/browse/YARN-8406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi reassigned YARN-8406: --- Assignee: zhuqi > Do the improvement to the FSLeafQueue about calculating fair share for apps > --- > > Key: YARN-8406 > URL: https://issues.apache.org/jira/browse/YARN-8406 > Project: Hadoop YARN > Issue Type: Sub-task > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: zhuqi >Assignee: zhuqi >Priority: Critical > Labels: patch > Attachments: YARN-7467-001.patch, YARN-7467-002.patch, test 02.png, > test.png > > Original Estimate: 24h > Remaining Estimate: 24h > > I want to help to do the improvement about that FSLeafQueue unnecessarily > calls ComputeFairShares.computeShare(). > h1. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9010) Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl
[
https://issues.apache.org/jira/browse/YARN-9010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-9010:
---
Attachment: YARN-9010-trunk-001.patch
> Fix the incorrect trailing slash deletion in constructor method of
> CGroupsHandlerImpl
> -
>
> Key: YARN-9010
> URL: https://issues.apache.org/jira/browse/YARN-9010
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-9010-trunk-001.patch
>
>
> In constructor method of CGroupsHandlerImpl:
> {code:java}
> this.cGroupPrefix = conf.get(YarnConfiguration.
> NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
> .replaceAll("^/", "").replaceAll("$/", "");{code}
> The "$/" regex expression is not working. And "^/" for leading slash handling
> is also not good enough.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8881) Add basic pluggable device plugin framework
[
https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683291#comment-16683291
]
Hadoop QA commented on YARN-8881:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 4 unchanged - 5 fixed = 4 total (was 9) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m
40s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8881 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947764/YARN-8881-trunk.008.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 85b9075040f6 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b5ec85d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22505/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yar
[jira] [Commented] (YARN-8881) Add basic pluggable device plugin framework
[ https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683297#comment-16683297 ] Zhankun Tang commented on YARN-8881: [~leftnoteasy], [~cheersyang] . Please help to review. > Add basic pluggable device plugin framework > --- > > Key: YARN-8881 > URL: https://issues.apache.org/jira/browse/YARN-8881 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-8881-trunk.001.patch, YARN-8881-trunk.002.patch, > YARN-8881-trunk.003.patch, YARN-8881-trunk.004.patch, > YARN-8881-trunk.005.patch, YARN-8881-trunk.006.patch, > YARN-8881-trunk.007.patch, YARN-8881-trunk.008.patch > > > It includes adding support in "ResourcePluginManager" to load plugin classes > based on configuration, an interface for the vendor to implement and the > adapter to decouple plugin and YARN internals. And the vendor device resource > discovery will be ready after this support -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9010) Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl
Zhankun Tang created YARN-9010:
--
Summary: Fix the incorrect trailing slash deletion in constructor
method of CGroupsHandlerImpl
Key: YARN-9010
URL: https://issues.apache.org/jira/browse/YARN-9010
Project: Hadoop YARN
Issue Type: Bug
Reporter: Zhankun Tang
Assignee: Zhankun Tang
In constructor method of CGroupsHandlerImpl:
{code:java}
this.cGroupPrefix = conf.get(YarnConfiguration.
NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
.replaceAll("^/", "").replaceAll("$/", "");{code}
The "$/" regex expression is not working.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9010) Fix the incorrect trailing slash deletion in constructor method of CGroupsHandlerImpl
[
https://issues.apache.org/jira/browse/YARN-9010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-9010:
---
Description:
In constructor method of CGroupsHandlerImpl:
{code:java}
this.cGroupPrefix = conf.get(YarnConfiguration.
NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
.replaceAll("^/", "").replaceAll("$/", "");{code}
The "$/" regex expression is not working. And "^/" for leading slash handling
is also not good enough.
was:
In constructor method of CGroupsHandlerImpl:
{code:java}
this.cGroupPrefix = conf.get(YarnConfiguration.
NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
.replaceAll("^/", "").replaceAll("$/", "");{code}
The "$/" regex expression is not working.
> Fix the incorrect trailing slash deletion in constructor method of
> CGroupsHandlerImpl
> -
>
> Key: YARN-9010
> URL: https://issues.apache.org/jira/browse/YARN-9010
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
>
> In constructor method of CGroupsHandlerImpl:
> {code:java}
> this.cGroupPrefix = conf.get(YarnConfiguration.
> NM_LINUX_CONTAINER_CGROUPS_HIERARCHY, "/hadoop-yarn")
> .replaceAll("^/", "").replaceAll("$/", "");{code}
> The "$/" regex expression is not working. And "^/" for leading slash handling
> is also not good enough.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8877) Extend service spec to allow setting resource attributes
[
https://issues.apache.org/jira/browse/YARN-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683284#comment-16683284
]
Sunil Govindan commented on YARN-8877:
--
In this context, {{attributes: null}} looks fine to me.
And thanks for pointing out to tests in TestResourceInformation. I ll commit
this patch later today if there are no objections.
> Extend service spec to allow setting resource attributes
>
>
> Key: YARN-8877
> URL: https://issues.apache.org/jira/browse/YARN-8877
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8877.001.patch, YARN-8877.002.patch
>
>
> Extend yarn native service spec to support setting resource attributes in the
> spec file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8902) [CSI] Add volume manager that manages CSI volume lifecycle
[ https://issues.apache.org/jira/browse/YARN-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil Govindan updated YARN-8902: - Summary: [CSI] Add volume manager that manages CSI volume lifecycle (was: Add volume manager that manages CSI volume lifecycle) > [CSI] Add volume manager that manages CSI volume lifecycle > -- > > Key: YARN-8902 > URL: https://issues.apache.org/jira/browse/YARN-8902 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: YARN-8902.001.patch, YARN-8902.002.patch, > YARN-8902.003.patch, YARN-8902.004.patch, YARN-8902.005.patch, > YARN-8902.006.patch, YARN-8902.007.patch, YARN-8902.008.patch, > YARN-8902.009.patch > > > The CSI volume manager is a service running in RM process, that manages all > CSI volumes' lifecycle. The details about volume's lifecycle states can be > found in [CSI > spec|https://github.com/container-storage-interface/spec/blob/master/spec.md]. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8902) [CSI] Add volume manager that manages CSI volume lifecycle
[ https://issues.apache.org/jira/browse/YARN-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683282#comment-16683282 ] Hudson commented on YARN-8902: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #15403 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15403/]) YARN-8902. [CSI] Add volume manager that manages CSI volume lifecycle. (sunilg: rev 4e7284443e6cf8dac3cd5d2581730c87ae6ffb55) * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/volume/csi/VolumeId.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/event/VolumeEvent.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/processor/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeStates.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/volume/csi/CsiConstants.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/volume/csi/exception/InvalidVolumeException.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/volume/csi/VolumeCapabilityRange.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/volume/csi/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/provisioner/VolumeProvisioningResults.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/volume/csi/exception/VolumeException.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/CsiAdaptorClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/TestVolumeCapabilityRange.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/volume/csi/exception/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMContext.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeManagerImpl.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/event/package-info.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/TestVolumeProcessor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeBuilder.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/event/ControllerPublishVolumeEvent.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/lifecycle/VolumeState.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMActiveServiceContext.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/
[jira] [Commented] (YARN-8877) Extend service spec to allow setting resource attributes
[
https://issues.apache.org/jira/browse/YARN-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683272#comment-16683272
]
Weiwei Yang commented on YARN-8877:
---
Hi [~sunilg]
Thanks for the comments.
{quote}toIndentedString may print null in best case or may cause pblm when
attributes is printed.
{quote}
With this patch when attribtues is not set in the service spec, toString prints
following message
{noformat}
class ResourceInformation {
value: 100
unit: Gi
attributes: null
}
{noformat}
I can add a null check in toString() to ignore printing attributes if you
insist.
{quote}I am thinking whether its better to have tests in TestResourceUtils or
TestResourceInformation or TestResource
{quote}
This patch modifies the {{ResourceInformation}} class in service package, the
UT is added in {{TestAppJsonResolve}}. I have already added test cases in
{{TestResourceInformation}}, {{TestResourcePBImpl}} when working on YARN-8836.
Pls let me know if this is sufficient.
> Extend service spec to allow setting resource attributes
>
>
> Key: YARN-8877
> URL: https://issues.apache.org/jira/browse/YARN-8877
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8877.001.patch, YARN-8877.002.patch
>
>
> Extend yarn native service spec to support setting resource attributes in the
> spec file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8877) Extend service spec to allow setting resource attributes
[
https://issues.apache.org/jira/browse/YARN-8877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683247#comment-16683247
]
Sunil Govindan commented on YARN-8877:
--
Hi [~cheersyang]
Some minor nits:
1. toIndentedString may print null in best case or may cause pblm when
attributes is printed. I think its better to set {{ImmutableMap.of}} in the
setter itself.
2. Since we add new element to ResoureInformation, I am thinking whether its
better to have tests in TestResourceUtils or TestResourceInformation or
TestResource. What is your thought on this?
> Extend service spec to allow setting resource attributes
>
>
> Key: YARN-8877
> URL: https://issues.apache.org/jira/browse/YARN-8877
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8877.001.patch, YARN-8877.002.patch
>
>
> Extend yarn native service spec to support setting resource attributes in the
> spec file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8902) Add volume manager that manages CSI volume lifecycle
[ https://issues.apache.org/jira/browse/YARN-8902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683236#comment-16683236 ] Sunil Govindan commented on YARN-8902: -- Thanks [~cheersyang] Looks good to me. +1. Committing shortly > Add volume manager that manages CSI volume lifecycle > > > Key: YARN-8902 > URL: https://issues.apache.org/jira/browse/YARN-8902 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > Attachments: YARN-8902.001.patch, YARN-8902.002.patch, > YARN-8902.003.patch, YARN-8902.004.patch, YARN-8902.005.patch, > YARN-8902.006.patch, YARN-8902.007.patch, YARN-8902.008.patch, > YARN-8902.009.patch > > > The CSI volume manager is a service running in RM process, that manages all > CSI volumes' lifecycle. The details about volume's lifecycle states can be > found in [CSI > spec|https://github.com/container-storage-interface/spec/blob/master/spec.md]. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8997) [Submarine] Small refactors of modifier, condition check and redundant local variables
[
https://issues.apache.org/jira/browse/YARN-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683232#comment-16683232
]
Zhankun Tang commented on YARN-8997:
[~giovanni.fumarola] , no unit test needed for this patch. could you please
help to review?
> [Submarine] Small refactors of modifier, condition check and redundant local
> variables
> ---
>
> Key: YARN-8997
> URL: https://issues.apache.org/jira/browse/YARN-8997
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8997-trunk-001.patch, YARN-8997-trunk-002.patch
>
>
> In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
> line.
> {code:java}
> if (content != null && content.contains("hdfs://")) {
> return true;
> }
> return false;{code}
> {code:java}
> return content != null && content.contains("hdfs://");{code}
> In CliUtils#argsForHelp
> {code:java}
> if (args[0].equals("-h") || args[0].equals("--help")) {
> return true;
> }
> {code}
> Can be simlified to:
> {code:java}
> return args[0].equals("-h") || args[0].equals("--help");{code}
> And several redundant variables can be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8997) [Submarine] Small refactors of modifier, condition check and redundant local variables
[
https://issues.apache.org/jira/browse/YARN-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683231#comment-16683231
]
Hadoop QA commented on YARN-8997:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
31s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8997 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947773/YARN-8997-trunk-002.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux d4be43fdace9 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b5ec85d |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/22504/testReport/ |
| Max. process+thread count | 444 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-submarine |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/22504/console |
| Powered by | Apache Yetus 0.8.0 http:
[jira] [Commented] (YARN-2823) NullPointerException in RM HA enabled 3-node cluster
[
https://issues.apache.org/jira/browse/YARN-2823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683225#comment-16683225
]
Paul Lin commented on YARN-2823:
[~imstefanlee] Hi, I'm facing the same issue with Flink applications. I tried
explicitly setting `KeepContainersAcrossApplicationAttempts` to false, but it
doesn't work. How do you solve the problem at last? And could you please point
me to the code where the default value KeepContainersAcrossApplicationAttempts
is set to true? Thanks a lot!
> NullPointerException in RM HA enabled 3-node cluster
>
>
> Key: YARN-2823
> URL: https://issues.apache.org/jira/browse/YARN-2823
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Gour Saha
>Assignee: Jian He
>Priority: Critical
> Fix For: 2.6.0
>
> Attachments: YARN-2823.1.patch, logs_with_NPE_in_RM.zip
>
>
> Branch:
> 2.6.0
> Environment:
> A 3-node cluster with RM HA enabled. The HA setup went pretty smooth (used
> Ambari) and then installed HBase using Slider. After some time the RMs went
> down and would not come back up anymore. Following is the NPE we see in both
> the RM logs.
> {noformat}
> 2014-09-16 01:36:28,037 FATAL resourcemanager.ResourceManager
> (ResourceManager.java:run(612)) - Error in handling event type
> APP_ATTEMPT_ADDED to the scheduler
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.transferStateFromPreviousAttempt(SchedulerApplicationAttempt.java:530)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplicationAttempt(CapacityScheduler.java:678)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1015)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:98)
> at
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$SchedulerEventDispatcher$EventProcessor.run(ResourceManager.java:603)
> at java.lang.Thread.run(Thread.java:744)
> 2014-09-16 01:36:28,042 INFO resourcemanager.ResourceManager
> (ResourceManager.java:run(616)) - Exiting, bbye..
> {noformat}
> All the logs for this 3-node cluster has been uploaded.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs
[
https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683219#comment-16683219
]
Hadoop QA commented on YARN-9001:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 45s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications: The patch generated 0
new + 44 unchanged - 1 fixed = 44 total (was 45) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 25s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 13s{color}
| {color:red} hadoop-yarn-services-core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
32s{color} | {color:green} hadoop-yarn-submarine in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 73m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.service.TestYarnNativeServices |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9001 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947768/YARN-9001.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2b2d8891e2a0 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / b5ec85d |
| maven | version: Apache Maven 3
[jira] [Updated] (YARN-8997) [Submarine] Small refactors of modifier, condition check and redundant local variables
[
https://issues.apache.org/jira/browse/YARN-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-8997:
---
Attachment: YARN-8997-trunk-002.patch
> [Submarine] Small refactors of modifier, condition check and redundant local
> variables
> ---
>
> Key: YARN-8997
> URL: https://issues.apache.org/jira/browse/YARN-8997
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8997-trunk-001.patch, YARN-8997-trunk-002.patch
>
>
> In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
> line.
> {code:java}
> if (content != null && content.contains("hdfs://")) {
> return true;
> }
> return false;{code}
> {code:java}
> return content != null && content.contains("hdfs://");{code}
> In CliUtils#argsForHelp
> {code:java}
> if (args[0].equals("-h") || args[0].equals("--help")) {
> return true;
> }
> {code}
> Can be simlified to:
> {code:java}
> return args[0].equals("-h") || args[0].equals("--help");{code}
> And several redundant variables can be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8997) [Submarine] Small refactors of modifier, condition check and redundant local variables
[
https://issues.apache.org/jira/browse/YARN-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-8997:
---
Description:
In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
line.
{code:java}
if (content != null && content.contains("hdfs://")) {
return true;
}
return false;{code}
{code:java}
return content != null && content.contains("hdfs://");{code}
In CliUtils#argsForHelp
{code:java}
if (args[0].equals("-h") || args[0].equals("--help")) {
return true;
}
{code}
Can be simlified to:
{code:java}
return args[0].equals("-h") || args[0].equals("--help");{code}
And several redundant variables can be removed.
was:
In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
line.
{code:java}
if (content != null && content.contains("hdfs://")) {
return true;
}
return false;{code}
{code:java}
return content != null && content.contains("hdfs://");{code}
> [Submarine] Small refactors of modifier, condition check and redundant local
> variables
> ---
>
> Key: YARN-8997
> URL: https://issues.apache.org/jira/browse/YARN-8997
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8997-trunk-001.patch, YARN-8997-trunk-002.patch
>
>
> In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
> line.
> {code:java}
> if (content != null && content.contains("hdfs://")) {
> return true;
> }
> return false;{code}
> {code:java}
> return content != null && content.contains("hdfs://");{code}
> In CliUtils#argsForHelp
> {code:java}
> if (args[0].equals("-h") || args[0].equals("--help")) {
> return true;
> }
> {code}
> Can be simlified to:
> {code:java}
> return args[0].equals("-h") || args[0].equals("--help");{code}
> And several redundant variables can be removed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8997) [Submarine] Small refactors of modifier, condition check and redundant local variables
[
https://issues.apache.org/jira/browse/YARN-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-8997:
---
Summary: [Submarine] Small refactors of modifier, condition check and
redundant local variables (was: [Submarine] Small refactors of modifier, )
> [Submarine] Small refactors of modifier, condition check and redundant local
> variables
> ---
>
> Key: YARN-8997
> URL: https://issues.apache.org/jira/browse/YARN-8997
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8997-trunk-001.patch
>
>
> In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
> line.
> {code:java}
> if (content != null && content.contains("hdfs://")) {
> return true;
> }
> return false;{code}
> {code:java}
> return content != null && content.contains("hdfs://");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8997) [Submarine] Small refactors of modifier,
[
https://issues.apache.org/jira/browse/YARN-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang updated YARN-8997:
---
Summary: [Submarine] Small refactors of modifier, (was: [Submarine]
Simplify the logic in YarnServiceJobSubmitter#needHdfs)
> [Submarine] Small refactors of modifier,
> -
>
> Key: YARN-8997
> URL: https://issues.apache.org/jira/browse/YARN-8997
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8997-trunk-001.patch
>
>
> In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
> line.
> {code:java}
> if (content != null && content.contains("hdfs://")) {
> return true;
> }
> return false;{code}
> {code:java}
> return content != null && content.contains("hdfs://");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8998) [Submarine] Simplify the condition check in CliUtils#argsForHelp
[
https://issues.apache.org/jira/browse/YARN-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683199#comment-16683199
]
Zhankun Tang edited comment on YARN-8998 at 11/12/18 4:19 AM:
--
Merge this with YARN-8997
was (Author: tangzhankun):
Merge this with -YARN-8997-
> [Submarine] Simplify the condition check in CliUtils#argsForHelp
>
>
> Key: YARN-8998
> URL: https://issues.apache.org/jira/browse/YARN-8998
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8998-trunk-001.patch
>
>
> {code:java}
> if (args[0].equals("-h") || args[0].equals("--help")) {
> return true;
> }
> {code}
> Can be simlified to:
> {code:java}
> return args[0].equals("-h") || args[0].equals("--help");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8999) [Submarine] Remove redundant local variables
[ https://issues.apache.org/jira/browse/YARN-8999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang resolved YARN-8999. Resolution: Duplicate Merge this with YARN-8997 > [Submarine] Remove redundant local variables > > > Key: YARN-8999 > URL: https://issues.apache.org/jira/browse/YARN-8999 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Minor > Attachments: YARN-8999-trunk-001.patch > > > Several methods have redundant local variables that can be removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8998) [Submarine] Simplify the condition check in CliUtils#argsForHelp
[
https://issues.apache.org/jira/browse/YARN-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683199#comment-16683199
]
Zhankun Tang edited comment on YARN-8998 at 11/12/18 4:16 AM:
--
Merge this with -YARN-8997-
was (Author: tangzhankun):
Merge this with YARN-8996
> [Submarine] Simplify the condition check in CliUtils#argsForHelp
>
>
> Key: YARN-8998
> URL: https://issues.apache.org/jira/browse/YARN-8998
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8998-trunk-001.patch
>
>
> {code:java}
> if (args[0].equals("-h") || args[0].equals("--help")) {
> return true;
> }
> {code}
> Can be simlified to:
> {code:java}
> return args[0].equals("-h") || args[0].equals("--help");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8998) [Submarine] Simplify the condition check in CliUtils#argsForHelp
[
https://issues.apache.org/jira/browse/YARN-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhankun Tang resolved YARN-8998.
Resolution: Duplicate
Merge this with YARN-8996
> [Submarine] Simplify the condition check in CliUtils#argsForHelp
>
>
> Key: YARN-8998
> URL: https://issues.apache.org/jira/browse/YARN-8998
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8998-trunk-001.patch
>
>
> {code:java}
> if (args[0].equals("-h") || args[0].equals("--help")) {
> return true;
> }
> {code}
> Can be simlified to:
> {code:java}
> return args[0].equals("-h") || args[0].equals("--help");
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8997) [Submarine] Simplify the logic in YarnServiceJobSubmitter#needHdfs
[
https://issues.apache.org/jira/browse/YARN-8997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683193#comment-16683193
]
Zhankun Tang commented on YARN-8997:
[~giovanni.fumarola] , Sure. Thanks for the review. Will merge to one patch and
submitted here.
> [Submarine] Simplify the logic in YarnServiceJobSubmitter#needHdfs
> --
>
> Key: YARN-8997
> URL: https://issues.apache.org/jira/browse/YARN-8997
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
>Priority: Minor
> Attachments: YARN-8997-trunk-001.patch
>
>
> In YarnServiceJobSubmitter#needHdfs. Below code can be simplified to just one
> line.
> {code:java}
> if (content != null && content.contains("hdfs://")) {
> return true;
> }
> return false;{code}
> {code:java}
> return content != null && content.contains("hdfs://");{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8925) Updating distributed node attributes only when necessary
[ https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683177#comment-16683177 ] Weiwei Yang commented on YARN-8925: --- Hi [~Tao Yang] Can you help to fix "Variable 'xxx' must be private and have accessor methods. [VisibilityModifier]" checkstyle issues? You can ignore the number of parameters warning. Overall the patch seems good. I will deploy this patch on a env and try out. Thanks > Updating distributed node attributes only when necessary > > > Key: YARN-8925 > URL: https://issues.apache.org/jira/browse/YARN-8925 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Labels: performance > Attachments: YARN-8925.001.patch, YARN-8925.002.patch, > YARN-8925.003.patch, YARN-8925.004.patch, YARN-8925.005.patch > > > Currently if distributed node attributes exist, even though there is no > change, updating for distributed node attributes will happen in every > heartbeat between NM and RM. Updating process will hold > NodeAttributesManagerImpl#writeLock and may have some influence in a large > cluster. We have found nodes UI of a large cluster is opened slowly and most > time it's waiting for the lock in NodeAttributesManagerImpl. I think this > updating should be called only when necessary to enhance the performance of > related process. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9001) [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs
[ https://issues.apache.org/jira/browse/YARN-9001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zac Zhou updated YARN-9001: --- Attachment: YARN-9001.003.patch > [Submarine] Use AppAdminClient instead of ServiceClient to sumbit jobs > -- > > Key: YARN-9001 > URL: https://issues.apache.org/jira/browse/YARN-9001 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zac Zhou >Assignee: Zac Zhou >Priority: Major > Attachments: YARN-9001.001.patch, YARN-9001.002.patch, > YARN-9001.003.patch > > > For now, submarine submit a service to yarn by using ServiceClient, We should > change it to AppAdminClient -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8881) Add basic pluggable device plugin framework
[ https://issues.apache.org/jira/browse/YARN-8881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-8881: --- Attachment: YARN-8881-trunk.008.patch > Add basic pluggable device plugin framework > --- > > Key: YARN-8881 > URL: https://issues.apache.org/jira/browse/YARN-8881 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Attachments: YARN-8881-trunk.001.patch, YARN-8881-trunk.002.patch, > YARN-8881-trunk.003.patch, YARN-8881-trunk.004.patch, > YARN-8881-trunk.005.patch, YARN-8881-trunk.006.patch, > YARN-8881-trunk.007.patch, YARN-8881-trunk.008.patch > > > It includes adding support in "ResourcePluginManager" to load plugin classes > based on configuration, an interface for the vendor to implement and the > adapter to decouple plugin and YARN internals. And the vendor device resource > discovery will be ready after this support -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9009) Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs
[ https://issues.apache.org/jira/browse/YARN-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683158#comment-16683158 ] Zhankun Tang commented on YARN-9009: [~OrDTesters] , The patch looks good to me. Could you generate a patch file like "YARN-9009-trunk-001.patch" and submitted here and then trigger a test? This is a little inconvenient, or I can help you to submit it if you are not willing to. > Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs > --- > > Key: YARN-9009 > URL: https://issues.apache.org/jira/browse/YARN-9009 > Project: Hadoop YARN > Issue Type: Bug > Environment: Ubuntu 18.04 > java version "1.8.0_181" > Java(TM) SE Runtime Environment (build 1.8.0_181-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) > > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T13:33:14-05:00) >Reporter: OrDTesters >Priority: Minor > > In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after > testMoveToDone. > testCleanLogs fails because testMoveToDone moves a file into the same > directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, > instead of 2 as testCleanLogs expects. > To fix the failure of testCleanLogs, we can delete the file after the file is > moved by testMoveToDone. > Pull request link: [https://github.com/apache/hadoop/pull/438] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9009) Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs
[ https://issues.apache.org/jira/browse/YARN-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] OrDTesters updated YARN-9009: - Description: In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after testMoveToDone. testCleanLogs fails because testMoveToDone moves a file into the same directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as testCleanLogs expects. To fix the failure of testCleanLogs, we can delete the file after the file is moved by testMoveToDone. Pull request link: [https://github.com/apache/hadoop/pull/438] was: In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after testMoveToDone. This is because testMoveToDone moves a file into the same directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as it expects. To fix this, we can delete the file after it is moved by testMoveToDone. Pull request link: https://github.com/apache/hadoop/pull/438 > Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs > --- > > Key: YARN-9009 > URL: https://issues.apache.org/jira/browse/YARN-9009 > Project: Hadoop YARN > Issue Type: Bug > Environment: Ubuntu 18.04 > java version "1.8.0_181" > Java(TM) SE Runtime Environment (build 1.8.0_181-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) > > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T13:33:14-05:00) >Reporter: OrDTesters >Priority: Minor > > In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after > testMoveToDone. > testCleanLogs fails because testMoveToDone moves a file into the same > directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, > instead of 2 as testCleanLogs expects. > To fix the failure of testCleanLogs, we can delete the file after the file is > moved by testMoveToDone. > Pull request link: [https://github.com/apache/hadoop/pull/438] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9009) Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs
[ https://issues.apache.org/jira/browse/YARN-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] OrDTesters updated YARN-9009: - Description: In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after testMoveToDone. This is because testMoveToDone moves a file into the same directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as it expects. To fix this, we can delete the file after it is moved by testMoveToDone. Pull request link: https://github.com/apache/hadoop/pull/438 was: In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after testMoveToDone. This is because testMoveToDone moves a file into the same directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as it expects. To fix this, we can delete the file after it is moved by testMoveToDone. Pull request link: to be added > Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs > --- > > Key: YARN-9009 > URL: https://issues.apache.org/jira/browse/YARN-9009 > Project: Hadoop YARN > Issue Type: Bug > Environment: Ubuntu 18.04 > java version "1.8.0_181" > Java(TM) SE Runtime Environment (build 1.8.0_181-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) > > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T13:33:14-05:00) >Reporter: OrDTesters >Priority: Minor > > In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after > testMoveToDone. > This is because testMoveToDone moves a file into the same directory that > testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as > it expects. > To fix this, we can delete the file after it is moved by testMoveToDone. > Pull request link: https://github.com/apache/hadoop/pull/438 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9009) Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs
[ https://issues.apache.org/jira/browse/YARN-9009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16683137#comment-16683137 ] ASF GitHub Bot commented on YARN-9009: -- GitHub user OrDTesters opened a pull request: https://github.com/apache/hadoop/pull/438 YARN-9009: Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after testMoveToDone. This is because testMoveToDone moves a file into the same directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as it expects. To fix this, we can delete the file after it is moved by testMoveToDone. Link to issue: [YARN-9009](https://issues.apache.org/jira/browse/YARN-9009) You can merge this pull request into a Git repository by running: $ git pull https://github.com/OrDTesters/hadoop fix-flaky-testCleanLogs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/438.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #438 commit db8e046352b96ead63523cfe4dc7e22c6acdfec8 Author: OrDTesters Date: 2018-11-12T02:25:21Z YARN-9009 Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs > Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs > --- > > Key: YARN-9009 > URL: https://issues.apache.org/jira/browse/YARN-9009 > Project: Hadoop YARN > Issue Type: Bug > Environment: Ubuntu 18.04 > java version "1.8.0_181" > Java(TM) SE Runtime Environment (build 1.8.0_181-b13) > Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) > > Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; > 2018-06-17T13:33:14-05:00) >Reporter: OrDTesters >Priority: Minor > > In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after > testMoveToDone. > This is because testMoveToDone moves a file into the same directory that > testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as > it expects. > To fix this, we can delete the file after it is moved by testMoveToDone. > Pull request link: to be added -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9009) Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs
OrDTesters created YARN-9009: Summary: Fix flaky test TestEntityGroupFSTimelineStore.testCleanLogs Key: YARN-9009 URL: https://issues.apache.org/jira/browse/YARN-9009 Project: Hadoop YARN Issue Type: Bug Environment: Ubuntu 18.04 java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode) Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 2018-06-17T13:33:14-05:00) Reporter: OrDTesters In TestEntityGroupFSTimelineStore, testCleanLogs fails when run after testMoveToDone. This is because testMoveToDone moves a file into the same directory that testCleanLogs cleans, causing testCleanLogs to clean 3 files, instead of 2 as it expects. To fix this, we can delete the file after it is moved by testMoveToDone. Pull request link: to be added -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8933) [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682986#comment-16682986 ] Hudson commented on YARN-8933: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15402 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15402/]) YARN-8933. [AMRMProxy] Fix potential empty fields in allocation (botong: rev b5ec85d96615e8214c14b57f8980a1dee6197ffa) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/FederationAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/TestBroadcastAMRMProxyFederationPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/TestLocalityMulticastAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/LocalityMulticastAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestFederationInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/BaseFederationPoliciesTest.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/TestRejectAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/BroadcastAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/HomeAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/RejectAMRMProxyPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/amrmproxy/TestHomeAMRMProxyPolicy.java > [AMRMProxy] Fix potential empty fields in allocation response, move > SubClusterTimeout to FederationInterceptor > -- > > Key: YARN-8933 > URL: https://issues.apache.org/jira/browse/YARN-8933 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy, federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 2.10.0, 3.3.0 > > Attachments: YARN-8933.v1.patch, YARN-8933.v2.patch, > YARN-8933.v3.patch > > > After YARN-8696, the allocate response by FederationInterceptor is merged > from the responses from a random subset of all sub-clusters, depending on the > async heartbeat timing. As a result, cluster-wide information fields in the > response, e.g. AvailableResources and NumClusterNodes, are not consistent at > all. It can even be null/zero because the specific response is merged from an > empty set of sub-cluster responses. > In this patch, we let FederationInterceptor remember the last allocate > response from all known sub-clusters, and always construct the cluster-wide > info fields from all of them. We also moved sub-cluster timeout from > LocalityMulticastAMRMProxyPolicy to FederationInterceptor, so that > sub-clusters that expired (haven't had a successful allocate response for a > while) won't be included in the computation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8933) [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682964#comment-16682964 ] Botong Huang commented on YARN-8933: Thanks [~bibinchundatt] for the comments and review, committing to trunk and branch-2 > [AMRMProxy] Fix potential empty fields in allocation response, move > SubClusterTimeout to FederationInterceptor > -- > > Key: YARN-8933 > URL: https://issues.apache.org/jira/browse/YARN-8933 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy, federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 2.10.0, 3.3.0 > > Attachments: YARN-8933.v1.patch, YARN-8933.v2.patch, > YARN-8933.v3.patch > > > After YARN-8696, the allocate response by FederationInterceptor is merged > from the responses from a random subset of all sub-clusters, depending on the > async heartbeat timing. As a result, cluster-wide information fields in the > response, e.g. AvailableResources and NumClusterNodes, are not consistent at > all. It can even be null/zero because the specific response is merged from an > empty set of sub-cluster responses. > In this patch, we let FederationInterceptor remember the last allocate > response from all known sub-clusters, and always construct the cluster-wide > info fields from all of them. We also moved sub-cluster timeout from > LocalityMulticastAMRMProxyPolicy to FederationInterceptor, so that > sub-clusters that expired (haven't had a successful allocate response for a > while) won't be included in the computation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8933) [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor
[ https://issues.apache.org/jira/browse/YARN-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Botong Huang updated YARN-8933: --- Fix Version/s: 3.3.0 2.10.0 > [AMRMProxy] Fix potential empty fields in allocation response, move > SubClusterTimeout to FederationInterceptor > -- > > Key: YARN-8933 > URL: https://issues.apache.org/jira/browse/YARN-8933 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy, federation >Reporter: Botong Huang >Assignee: Botong Huang >Priority: Major > Fix For: 2.10.0, 3.3.0 > > Attachments: YARN-8933.v1.patch, YARN-8933.v2.patch, > YARN-8933.v3.patch > > > After YARN-8696, the allocate response by FederationInterceptor is merged > from the responses from a random subset of all sub-clusters, depending on the > async heartbeat timing. As a result, cluster-wide information fields in the > response, e.g. AvailableResources and NumClusterNodes, are not consistent at > all. It can even be null/zero because the specific response is merged from an > empty set of sub-cluster responses. > In this patch, we let FederationInterceptor remember the last allocate > response from all known sub-clusters, and always construct the cluster-wide > info fields from all of them. We also moved sub-cluster timeout from > LocalityMulticastAMRMProxyPolicy to FederationInterceptor, so that > sub-clusters that expired (haven't had a successful allocate response for a > while) won't be included in the computation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8980) Mapreduce application container start fail after AM restart.
[ https://issues.apache.org/jira/browse/YARN-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682962#comment-16682962 ] Botong Huang commented on YARN-8980: I agree. I am also worried about container leaks, since the new attempt (old) AM is not even aware of the existing containers from the UAMs. Note that RM only supports one attempt for UAM and this UAM attempt is used throughout all AM attempts in home SC. I think on top of 1 you mentioned (clear token cache in RM), _FederationInterceptor_ needs to know the _keepContainer_ flag of the original AM. If it is false, after reattaching to the UAMs in _registerApplicationMaster_ it needs to release all running containers from UAM. > Mapreduce application container start fail after AM restart. > - > > Key: YARN-8980 > URL: https://issues.apache.org/jira/browse/YARN-8980 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin A Chundatt >Priority: Major > > UAM to subclusters are always launched with keepContainers. > On AM restart scenarios , UAM register again with RM . UAM receive running > containers with NMToken. NMToken received by UAM in > getPreviousAttemptContainersNMToken is never used by mapreduce application. > Federation Interceptor should take care of such scenarios too. Merge NMToken > received at registration to allocate response. > Container allocation response on same node will have NMToken empty. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates for queue and reserved container preemption
[
https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682851#comment-16682851
]
Tao Yang edited comment on YARN-9007 at 11/11/18 11:53 AM:
---
Add "for queue and reserved container preemption" to the end of this issue
name. This issue is proposed to skip selecting OPPORTUNISTIC containers for the
policies which want to balance GUARANTEED resource in preemption monitor.
{quote}
Few high level thoughts.
1. IMO, ExecutionType.OPPORTUNISTIC containers could be the first candidates
for any preemption. Rather than selecting GUARANTEED containers before
consuming OPPORTUNISTIC seems not best choice as per me.
2. On the flip side, i know OPPORTUNISTIC containers has a different lifecycle
when more demand comes on the node. But to me, both preemption and existing
model of upgrading/downgrading ExecutionType need to coexist together. So could
we discuss more on that line.
in either of above case, we need more discussion.
{quote}
[~sunilg], I agree with your thoughts. Sorry about not clarify this issue
before.
was (Author: tao yang):
Add "for queue and reserved container preemption" to the end of this issue
name. This issue is proposed to skip selecting OPPORTUNISTIC containers for the
policies which want to balance GUARANTEED resource in preemption monitor.
> CS preemption monitor should only select GUARANTEED containers as candidates
> for queue and reserved container preemption
>
>
> Key: YARN-9007
> URL: https://issues.apache.org/jira/browse/YARN-9007
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 3.2.1
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-9007.001.patch
>
>
> Currently CS preemption monitor doesn't consider execution type of
> containers, so OPPORTUNISTIC containers maybe selected and killed without
> effect.
> In some scenario with OPPORTUNISTIC containers, not even preemption can't
> work properly to balance resources, but also some apps with OPPORTUNISTIC
> containers maybe effected and unable to work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates for queue and reserved container preemption
[ https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682851#comment-16682851 ] Tao Yang commented on YARN-9007: Add "for queue and reserved container preemption" to the end of this issue name. This issue is proposed to skip selecting OPPORTUNISTIC containers for the policies which want to balance GUARANTEED resource in preemption monitor. > CS preemption monitor should only select GUARANTEED containers as candidates > for queue and reserved container preemption > > > Key: YARN-9007 > URL: https://issues.apache.org/jira/browse/YARN-9007 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9007.001.patch > > > Currently CS preemption monitor doesn't consider execution type of > containers, so OPPORTUNISTIC containers maybe selected and killed without > effect. > In some scenario with OPPORTUNISTIC containers, not even preemption can't > work properly to balance resources, but also some apps with OPPORTUNISTIC > containers maybe effected and unable to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates for queue and reserved container preemption
[ https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-9007: --- Summary: CS preemption monitor should only select GUARANTEED containers as candidates for queue and reserved container preemption (was: CS preemption monitor should only select GUARANTEED containers as candidates for queue/reserved) > CS preemption monitor should only select GUARANTEED containers as candidates > for queue and reserved container preemption > > > Key: YARN-9007 > URL: https://issues.apache.org/jira/browse/YARN-9007 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9007.001.patch > > > Currently CS preemption monitor doesn't consider execution type of > containers, so OPPORTUNISTIC containers maybe selected and killed without > effect. > In some scenario with OPPORTUNISTIC containers, not even preemption can't > work properly to balance resources, but also some apps with OPPORTUNISTIC > containers maybe effected and unable to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates for queue/reserved
[ https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Yang updated YARN-9007: --- Summary: CS preemption monitor should only select GUARANTEED containers as candidates for queue/reserved (was: CS preemption monitor should only select GUARANTEED containers as candidates) > CS preemption monitor should only select GUARANTEED containers as candidates > for queue/reserved > --- > > Key: YARN-9007 > URL: https://issues.apache.org/jira/browse/YARN-9007 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9007.001.patch > > > Currently CS preemption monitor doesn't consider execution type of > containers, so OPPORTUNISTIC containers maybe selected and killed without > effect. > In some scenario with OPPORTUNISTIC containers, not even preemption can't > work properly to balance resources, but also some apps with OPPORTUNISTIC > containers maybe effected and unable to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates
[ https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682846#comment-16682846 ] Tao Yang commented on YARN-9007: Thanks [~sunilg] for your attention. IIUC, intra-queue/inter-queue/reserved-containers/queue-priority preemptions are all used to balance the GUARANTEED resource, inside the preemption monitor, the calculator and related conditions are based on queue capacity/max-capacity and reserved containers, I think all of them are not related to OPPORTUNISTIC containers. That's why I created this issue. Perhaps I missed somewhere, please help me to figure out it. Thanks. > CS preemption monitor should only select GUARANTEED containers as candidates > > > Key: YARN-9007 > URL: https://issues.apache.org/jira/browse/YARN-9007 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9007.001.patch > > > Currently CS preemption monitor doesn't consider execution type of > containers, so OPPORTUNISTIC containers maybe selected and killed without > effect. > In some scenario with OPPORTUNISTIC containers, not even preemption can't > work properly to balance resources, but also some apps with OPPORTUNISTIC > containers maybe effected and unable to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8933) [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor
[
https://issues.apache.org/jira/browse/YARN-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682820#comment-16682820
]
Hadoop QA commented on YARN-8933:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
44s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 39s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server:
The patch generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
57s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
5s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}100m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-8933 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12947298/YARN-8933.v3.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux bbcbe273e264 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 2664248 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_181 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https
[jira] [Commented] (YARN-9007) CS preemption monitor should only select GUARANTEED containers as candidates
[ https://issues.apache.org/jira/browse/YARN-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16682811#comment-16682811 ] Sunil Govindan commented on YARN-9007: -- Thanks [~Tao Yang]. Few high level thoughts. # IMO, ExecutionType.OPPORTUNISTIC containers could be the first candidates for any preemption. Rather than selecting GUARANTEED containers before consuming OPPORTUNISTIC seems not best choice as per me. # On the flip side, i know OPPORTUNISTIC containers has a different lifecycle when more demand comes on the node. But to me, both preemption and existing model of upgrading/downgrading ExecutionType need to coexist together. So could we discuss more on that line. in either of above case, we need more discussion. Mean time could you pls explain it more how this existing patch helps in below cases. # when we have OPPORTUNISTIC and GUARANTEED containers in list of possible containers to preempt, what will happen if ## GUARANTEED containers are not enough to meet the demand of higher priority apps or similar kind ## after consuming all selected GUARANTEED for preemption, what ll happen to OPPORTUNISTIC containers which are left? > CS preemption monitor should only select GUARANTEED containers as candidates > > > Key: YARN-9007 > URL: https://issues.apache.org/jira/browse/YARN-9007 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 3.2.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Major > Attachments: YARN-9007.001.patch > > > Currently CS preemption monitor doesn't consider execution type of > containers, so OPPORTUNISTIC containers maybe selected and killed without > effect. > In some scenario with OPPORTUNISTIC containers, not even preemption can't > work properly to balance resources, but also some apps with OPPORTUNISTIC > containers maybe effected and unable to work. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
