[jira] [Commented] (YARN-4088) RM should be able to process heartbeats from NM concurrently

[stefanlee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383305#comment-16383305
 ] 

stefanlee commented on YARN-4088:
-

[~jlowe]  if  the OOB heartbeat could reduce the ability of ResourceManager 
schedule?

> RM should be able to process heartbeats from NM concurrently
> 
>
> Key: YARN-4088
> URL: https://issues.apache.org/jira/browse/YARN-4088
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager, scheduler
>Reporter: Srikanth Kandula
>Priority: Major
>
> Today, the RM sequentially processes one heartbeat after another. 
> Imagine a 3000 server cluster with each server heart-beating every 3s. This 
> gives the RM 1ms on average to process each NM heartbeat. That is tough.
> It is true that there are several underlying datastructures that will be 
> touched during heartbeat processing. So, it is non-trivial to parallelize the 
> NM heartbeat. Yet, it is quite doable...
> Parallelizing the NM heartbeat would substantially improve the scalability of 
> the RM, allowing it to either 
> a) run larger clusters or 
> b) support faster heartbeats or dynamic scaling of heartbeats
> c) take more asks from each application or 
> c) use cleverer/ more expensive algorithms such as node labels or better 
> packing or ...
> Indeed the RM's scalability limit has been cited as the motivating reason for 
> a variety of efforts which will become less needed if this can be solved. 
> Ditto for slow heartbeats.  See Sparrow and Mercury papers for example.
> Can we take a shot at this?
> If not, could we discuss why.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7626) Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount

[Zian Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383294#comment-16383294
 ] 

Zian Chen commented on YARN-7626:
-

[~leftnoteasy] , could you help me check the latest patch? Thanks!

> Allow regular expression matching in container-executor.cfg for devices and 
> named docker volumes mount
> --
>
> Key: YARN-7626
> URL: https://issues.apache.org/jira/browse/YARN-7626
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-7626.001.patch, YARN-7626.002.patch, 
> YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch, 
> YARN-7626.006.patch, YARN-7626.007.patch, YARN-7626.008.patch, 
> YARN-7626.009.patch, YARN-7626.010.patch
>
>
> Currently when we config some of the GPU devices related fields (like ) in 
> container-executor.cfg, these fields are generated based on different driver 
> versions or GPU device names. We want to enable regular expression matching 
> so that user don't need to manually set up these fields when config 
> container-executor.cfg,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7871) Node attributes reporting from NM to RM

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383256#comment-16383256
 ] 

Weiwei Yang commented on YARN-7871:
---

Hi [~Naganarasimha]

Please take a look at v3 patch and let me know your thoughts. Thanks.

> Node attributes reporting from NM to RM 
> 
>
> Key: YARN-7871
> URL: https://issues.apache.org/jira/browse/YARN-7871
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7871-YARN-3409.001.patch, 
> YARN-7871-YARN-3409.002.patch, YARN-7871-YARN-3409.003.patch
>
>
> Support to initialize proper attribute provider based on user's configuration.
> NM collects node attributes from a configured attribute provider and send 
> them to RM via HB. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7871) Node attributes reporting from NM to RM

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383251#comment-16383251
 ] 

genericqa commented on YARN-7871:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-3409 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m  
4s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 
36s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
56s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 7s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
55s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
14m 40s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
19s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in 
YARN-3409 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
21s{color} | {color:green} YARN-3409 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
36s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  4s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 2 new + 343 unchanged - 3 fixed = 345 total (was 346) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 10s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
41s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
5s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 18m 
36s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 59m 39s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
36s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 15s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce 

[jira] [Commented] (YARN-7987) Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks.

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383223#comment-16383223
 ] 

Suma Shivaprasad commented on YARN-7987:


Thanks [~shaneku...@gmail.com] for your inputs. yes you are right - The FQDN is 
being set to the . as you mentioned in case of 
both qualified and non-qualified container names and not from the domain-name 
specified in --name or --hostName. Sorry for not being clear in the previous 
comment, I meant YARN sets --name to non-qualified name which fails DNS 
resolution.

Tested --network-alias with all the available docker network modes and user 
defined network types i.e bridge, overlay and macvlan. Here are the 
observations.
 # network-alias is not a valid option in the "default bridge"and "host" 
networking modes and adding this flag fails. Will need to check and add this to 
docker container runs only in the case of user defined networks.  Not sure what 
is the behaviour in case of third party network plugins. This will need to be 
validated.
 # --network-alias is not being checked for unique-ness by docker embedded DNS 
while starting a container. However --name values are validated to be unique 
across the network. I could start two containers with the --network-alias 
having the same value and the DNS resolution points to the container's IP which 
started first. However, since the YARN generated hostnames (generated from 
container-id) are unique, this should not be a major issue IMO. 

 

> Docker container name(--name) needs to be DNS friendly for DNS resolution to 
> work in user defined networks. 
> 
>
> Key: YARN-7987
> URL: https://issues.apache.org/jira/browse/YARN-7987
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
>
> User defined networks like overlays support DNS resolution through Docker 
> Embedded DNS which needs the container name (–name parameter value in docker 
> run) to be a FQDN for container names to be resolved - Please refer 
> documentation 
> [https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/]
> However Yarn sets the container name to the container's id which is not DNS 
> friendly(eg: container_e26_1519402686002_0035_01_03) and is not a FQDN. 
> The proposal is to set a FQDN(eg: 
> ctr-e26-1519402686002-0035-01-03.domain-name) as the docker container's 
> name for containers to be able to communicate to each other via hostnames in 
> user defined networks like overlays, bridges etc. The domain name will be 
> picked up from the YARN DNS registry configuration 
> (hadoop.registry.dns.domain-name)
>  
>  
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[Suma Shivaprasad (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383204#comment-16383204
 ] 

Suma Shivaprasad commented on YARN-7935:


Thanks [~leftnoteasy] for the review. Answers below

1)  This patch introduces a new env CONTAINER_HOSTNAME which is se by NM in 
prepareContainer. Even if an AM sets this, it will be overridden by NM in the 
environment.

2) ContainerLaunch writes out the launch env immediately after calling 
prepareContainer.  User specified hostname in

YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_HOSTNAME

is given precedence as per earlier code flow and there is no change in this 
behaviour in the current patch .

However there is change in --hostName behaviour in "host" mode. Earlier the 
--hostName was being set to YARN defined hostname(generated from container_id) 
which is incorrect in host network mode. In this patch, it is set to the NM's 
host name which is the correct host name. Although docker seems to be ignoring 
the --hostName set in "host" Mode in linux and promotes the /etc/hosts file 
from the host, this is not the case in docker on Mac where it is honoring the 
value passed in --hostName

> Expose container's hostname to applications running within the docker 
> container
> ---
>
> Key: YARN-7935
> URL: https://issues.apache.org/jira/browse/YARN-7935
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
> Attachments: YARN-7935.1.patch, YARN-7935.2.patch, YARN-7935.3.patch
>
>
> Some applications have a need to bind to the container's hostname (like 
> Spark) which is different from the NodeManager's hostname(NM_HOST which is 
> available as an env during container launch) when launched through Docker 
> runtime. The container's hostname can be exposed to applications via an env 
> CONTAINER_HOSTNAME. Another potential candidate is the container's IP but 
> this can be addressed in a separate jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7511) NPE in ContainerLocalizer when localization failed for running container

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383160#comment-16383160
 ] 

genericqa commented on YARN-7511:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 49s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 17s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
 The patch generated 1 new + 57 unchanged - 0 fixed = 58 total (was 57) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 12s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m  
8s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
22s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 65m 32s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-7511 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12897976/YARN-7511.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux c8da91342d5a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 923e177 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/19868/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/19868/testReport/ |
| Max. process+thread count | 303 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 

[jira] [Commented] (YARN-7511) NPE in ContainerLocalizer when localization failed for running container

[Tao Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383155#comment-16383155
 ] 

Tao Yang commented on YARN-7511:


Hi [~Sammi], It's ok to modify the target version since nobody helps to review 
so far, I almost forgot this issue. [~jlowe], can you help to review this 
patch? Thank you.

> NPE in ContainerLocalizer when localization failed for running container
> 
>
> Key: YARN-7511
> URL: https://issues.apache.org/jira/browse/YARN-7511
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha4, 2.9.1
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-7511.001.patch
>
>
> Error log:
> {noformat}
> 2017-09-30 20:14:32,839 FATAL [AsyncDispatcher event handler] 
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
>         at 
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
>         at 
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceSet.resourceLocalizationFailed(ResourceSet.java:151)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:821)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:813)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1335)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:95)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1372)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1365)
>         at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
>         at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
>         at java.lang.Thread.run(Thread.java:834)
> 2017-09-30 20:14:32,842 INFO [AsyncDispatcher ShutDown handler] 
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
> {noformat}
> Reproduce this problem:
> 1. Container was running and ContainerManagerImpl#localize was called for 
> this container
> 2. Localization failed in ResourceLocalizationService$LocalizerRunner#run and 
> sent out ContainerResourceFailedEvent with null LocalResourceRequest.
> 3. NPE when ResourceLocalizationFailedWhileRunningTransition#transition --> 
> container.resourceSet.resourceLocalizationFailed(null)
> I think we can fix this problem through ensuring that request is not null 
> before remove it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7871) Node attributes reporting from NM to RM

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383118#comment-16383118
 ] 

Weiwei Yang commented on YARN-7871:
---

Hi [~Naganarasimha]

bq. having a single api same as earlier except for taking additional param 
prefix

Make sense. I've done this in v3 patch.

bq.NMDistributedNodeLabelsHandler earlier used to ensure that the labels were 
not sent across repeatatively in every heartbeat

As discussed, we can track this improvement later in a separate JIRA. This JIRA 
will be focusing on building essentials for distributed node attributes report.

Thanks

> Node attributes reporting from NM to RM 
> 
>
> Key: YARN-7871
> URL: https://issues.apache.org/jira/browse/YARN-7871
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7871-YARN-3409.001.patch, 
> YARN-7871-YARN-3409.002.patch, YARN-7871-YARN-3409.003.patch
>
>
> Support to initialize proper attribute provider based on user's configuration.
> NM collects node attributes from a configured attribute provider and send 
> them to RM via HB. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7871) Node attributes reporting from NM to RM

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-7871:
--
Attachment: YARN-7871-YARN-3409.003.patch

> Node attributes reporting from NM to RM 
> 
>
> Key: YARN-7871
> URL: https://issues.apache.org/jira/browse/YARN-7871
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7871-YARN-3409.001.patch, 
> YARN-7871-YARN-3409.002.patch, YARN-7871-YARN-3409.003.patch
>
>
> Support to initialize proper attribute provider based on user's configuration.
> NM collects node attributes from a configured attribute provider and send 
> them to RM via HB. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383113#comment-16383113
 ] 

Rohith Sharma K S commented on YARN-7346:
-

Thanks [~busbey] for suggestions. Should we raise any corresponding bug in 
Yetus to make this change? If it is going to be delayed much more then 
expected, could we do this change post committing this patch? This is first 
step for supporting hbase-2 profile, and there are many stability issues with 
hbase-2 which need to be fixed. And many other stories such as ACLs are depends 
on this JIRA. Basically we have blocked by this patch.

However ATSv2 folks takes *responsibility of ensuring that there are NO* 
findbugs and javadoc errors are introduced for hbase-2 profile modules. We have 
discussed this plan in weekly call as I commented earlier  also to have 
separate jenkins in branch with hbase-2 profile as default. As a long term, we 
definitely want to go ahead with your suggestion which we can start discussion 
how to achieve it.

> Add a profile to allow optional compilation for ATSv2 with HBase-2.0
> 
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> [https://pastebin.com/Ms4jYEVB]
> This issue is to fix the compilation errors.
> The scope of the Jira is to add a profile to allow optional compilation for 
> ATSv2 with HBase2.0. The default compilation for trunk will still be for 
> hbase 1.2.6. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7511) NPE in ContainerLocalizer when localization failed for running container

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7511?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383107#comment-16383107
 ] 

SammiChen commented on YARN-7511:
-

Hi [~Tao Yang], is it still on target for 2.9.1? if not, can we push this out 
from 2.9.1 to next release? 

> NPE in ContainerLocalizer when localization failed for running container
> 
>
> Key: YARN-7511
> URL: https://issues.apache.org/jira/browse/YARN-7511
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 3.0.0-alpha4, 2.9.1
>Reporter: Tao Yang
>Assignee: Tao Yang
>Priority: Major
> Attachments: YARN-7511.001.patch
>
>
> Error log:
> {noformat}
> 2017-09-30 20:14:32,839 FATAL [AsyncDispatcher event handler] 
> org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
> java.lang.NullPointerException
>         at 
> java.util.concurrent.ConcurrentHashMap.replaceNode(ConcurrentHashMap.java:1106)
>         at 
> java.util.concurrent.ConcurrentHashMap.remove(ConcurrentHashMap.java:1097)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceSet.resourceLocalizationFailed(ResourceSet.java:151)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:821)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl$ResourceLocalizationFailedWhileRunningTransition.transition(ContainerImpl.java:813)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory$SingleInternalArc.doTransition(StateMachineFactory.java:362)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
>         at 
> org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:1335)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl.handle(ContainerImpl.java:95)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1372)
>         at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl$ContainerEventDispatcher.handle(ContainerManagerImpl.java:1365)
>         at 
> org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184)
>         at 
> org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110)
>         at java.lang.Thread.run(Thread.java:834)
> 2017-09-30 20:14:32,842 INFO [AsyncDispatcher ShutDown handler] 
> org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..
> {noformat}
> Reproduce this problem:
> 1. Container was running and ContainerManagerImpl#localize was called for 
> this container
> 2. Localization failed in ResourceLocalizationService$LocalizerRunner#run and 
> sent out ContainerResourceFailedEvent with null LocalResourceRequest.
> 3. NPE when ResourceLocalizationFailedWhileRunningTransition#transition --> 
> container.resourceSet.resourceLocalizationFailed(null)
> I think we can fix this problem through ensuring that request is not null 
> before remove it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7736) Fix itemization in YARN federation document

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383103#comment-16383103
 ] 

SammiChen commented on YARN-7736:
-

Hi [~ajisakaa], is it still on target for 2.9.1? If not, can we push this out 
from 2.9.1 to next release? 

> Fix itemization in YARN federation document
> ---
>
> Key: YARN-7736
> URL: https://issues.apache.org/jira/browse/YARN-7736
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: documentation
>Reporter: Akira Ajisaka
>Priority: Minor
>  Labels: newbie
>
> https://hadoop.apache.org/docs/r3.0.0/hadoop-yarn/hadoop-yarn-site/Federation.html
> {noformat}
> Assumptions:
> * We assume reasonably good connectivity across sub-clusters (e.g., we are 
> not looking to federate across DC yet, though future investigations of this 
> are not excluded).
> * We rely on HDFS federation (or equivalently scalable DFS solutions) to take 
> care of scalability of the store side.
> {noformat}
> Blank line should be inserted before itemization to render correctly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383096#comment-16383096
 ] 

Sean Busbey commented on YARN-7346:
---

I believe the easiest way to make the change I'm suggesting would be to alter 
the project specific personality used in our precommit runs. I believe that 
personality currently lives in the yetus repository. since you'd be activating 
a profile, it should only cause a WARN to have the change in place prior to 
this patch.

> Add a profile to allow optional compilation for ATSv2 with HBase-2.0
> 
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> [https://pastebin.com/Ms4jYEVB]
> This issue is to fix the compilation errors.
> The scope of the Jira is to add a profile to allow optional compilation for 
> ATSv2 with HBase2.0. The default compilation for trunk will still be for 
> hbase 1.2.6. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383079#comment-16383079
 ] 

Rohith Sharma K S commented on YARN-7346:
-

bq. We should fix this by updating the Hadoop personality so that we only run 
in each module with an appropriate profile active
Sorry didn't get it. Could you elaborate more? Does this change is in Yetus or 
current patch? 

> Add a profile to allow optional compilation for ATSv2 with HBase-2.0
> 
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> [https://pastebin.com/Ms4jYEVB]
> This issue is to fix the compilation errors.
> The scope of the Jira is to add a profile to allow optional compilation for 
> ATSv2 with HBase2.0. The default compilation for trunk will still be for 
> hbase 1.2.6. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7992) launch_container.sh has the wrong file permission

[Eric Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7992:

Description: 
Docker workload fail to launch because launch_container.sh has wrong permission 
644 instead of 700.  The pid file has permission 700 instead of 644.

Expected Permission of files in localized directory:
{code}
2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
./launch_container.sh
2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
./container_tokens
{code}

Currently Permission of files in localized directory:
{code}
total 24
drwxr-xr-x. 2 yarn hadoop 4096 Mar  2 01:20 .
drwxr-xr-x. 3 yarn hadoop   51 Mar  2 01:20 ..
-rwx--. 1 yarn hadoop4 Mar  2 01:20 
container_1519953398658_0001_01_03.pid
-rw-r--r--. 1 yarn hadoop  222 Mar  2 01:20 
container_1519953398658_0001_01_03.tokens
-rw-r--r--. 1 yarn hadoop   12 Mar  2 01:20 
.container_1519953398658_0001_01_03.tokens.crc
-rw-r--r--. 1 yarn hadoop 3799 Mar  2 01:20 launch_container.sh
-rw-r--r--. 1 yarn hadoop   40 Mar  2 01:20 .launch_container.sh.crc
{code}

  was:
Docker workload fail to launch because launch_container.sh has wrong permission 
644 instead of 700.  The pid file has permission 700 instead of 644.  This 
regression is caused by changes in YARN-7815.

Permission of files in localized directory before YARN-7815:
{code}
2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
./launch_container.sh
2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
./container_tokens
{code}

Permission of files in localized directory after YARN-7815:
{code}
total 24
drwxr-xr-x. 2 yarn hadoop 4096 Mar  2 01:20 .
drwxr-xr-x. 3 yarn hadoop   51 Mar  2 01:20 ..
-rwx--. 1 yarn hadoop4 Mar  2 01:20 
container_1519953398658_0001_01_03.pid
-rw-r--r--. 1 yarn hadoop  222 Mar  2 01:20 
container_1519953398658_0001_01_03.tokens
-rw-r--r--. 1 yarn hadoop   12 Mar  2 01:20 
.container_1519953398658_0001_01_03.tokens.crc
-rw-r--r--. 1 yarn hadoop 3799 Mar  2 01:20 launch_container.sh
-rw-r--r--. 1 yarn hadoop   40 Mar  2 01:20 .launch_container.sh.crc
{code}


> launch_container.sh has the wrong file permission
> -
>
> Key: YARN-7992
> URL: https://issues.apache.org/jira/browse/YARN-7992
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Eric Yang
>Priority: Major
>
> Docker workload fail to launch because launch_container.sh has wrong 
> permission 644 instead of 700.  The pid file has permission 700 instead of 
> 644.
> Expected Permission of files in localized directory:
> {code}
> 2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
> 2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
> 2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
> ./launch_container.sh
> 2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
> ./container_tokens
> {code}
> Currently Permission of files in localized directory:
> {code}
> total 24
> drwxr-xr-x. 2 yarn hadoop 4096 Mar  2 01:20 .
> drwxr-xr-x. 3 yarn hadoop   51 Mar  2 01:20 ..
> -rwx--. 1 yarn hadoop4 Mar  2 01:20 
> container_1519953398658_0001_01_03.pid
> -rw-r--r--. 1 yarn hadoop  222 Mar  2 01:20 
> container_1519953398658_0001_01_03.tokens
> -rw-r--r--. 1 yarn hadoop   12 Mar  2 01:20 
> .container_1519953398658_0001_01_03.tokens.crc
> -rw-r--r--. 1 yarn hadoop 3799 Mar  2 01:20 launch_container.sh
> -rw-r--r--. 1 yarn hadoop   40 Mar  2 01:20 .launch_container.sh.crc
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7944) Remove master node link from headers of application pages

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383054#comment-16383054
 ] 

Sunil G commented on YARN-7944:
---

+1. Committing shortly.

> Remove master node link from headers of application pages
> -
>
> Key: YARN-7944
> URL: https://issues.apache.org/jira/browse/YARN-7944
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7944.001.patch, YARN-7944.002.patch, 
> YARN-7944.003.patch
>
>
> Rm UI2 has links for Master container log and master node. 
> This link published on application and service page. This links are not 
> required on all pages because AM container node link and container log link 
> are already present in Application view. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4858) start-yarn and stop-yarn scripts to support timeline and sharedcachemanager

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383052#comment-16383052
 ] 

genericqa commented on YARN-4858:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} docker {color} | {color:red}  0m  
5s{color} | {color:red} Docker failed to build yetus/hadoop:17213a0. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-4858 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12795001/YARN-4858-branch-2.001.patch
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/19867/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> start-yarn and stop-yarn scripts to support timeline and sharedcachemanager
> ---
>
> Key: YARN-4858
> URL: https://issues.apache.org/jira/browse/YARN-4858
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scripts
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>  Labels: oct16-easy
> Attachments: YARN-4858-001.patch, YARN-4858-branch-2.001.patch
>
>
> The start-yarn and stop-yarn scripts don't have any (even commented out) 
> support for the  timeline and sharedcachemanager
> Proposed:
> * bash and cmd start-yarn scripts have commented out start actions
> * stop-yarn scripts stop the servers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7957) Yarn service delete option disappears after stopping application

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383050#comment-16383050
 ] 

Sunil G commented on YARN-7957:
---

Thanks [~gsaha] This makes sense. Filed a Jira under this for UI changes.

> Yarn service delete option disappears after stopping application
> 
>
> Key: YARN-7957
> URL: https://issues.apache.org/jira/browse/YARN-7957
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: YARN-7957.01.patch
>
>
> Steps:
> 1) Launch yarn service
> 2) Go to service page and click on Setting button->"Stop Service". The 
> application will be stopped.
> 3) Refresh page
> Here, setting button disappears. Thus, user can not delete service from UI 
> after stopping application
> Expected behavior:
> Setting button should be present on UI page after application is stopped. If 
> application is stopped, setting button should only have "Delete Service" 
> action available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7993) [UI2] yarn-service page need to consider ServiceState to show stop/delete buttons

[Sunil G (JIRA)]

Sunil G created YARN-7993:
-

 Summary: [UI2] yarn-service page need to consider ServiceState to 
show stop/delete buttons
 Key: YARN-7993
 URL: https://issues.apache.org/jira/browse/YARN-7993
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn-ui-v2
Reporter: Sunil G


yarn service page has stop/delete buttons. These buttons has to be shown/hidden 
based on ServiceState of each app from ATS.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4858) start-yarn and stop-yarn scripts to support timeline and sharedcachemanager

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383040#comment-16383040
 ] 

SammiChen commented on YARN-4858:
-

Hi [~ste...@apache.org], does this still target for 2.9.1?  If not, can we push 
this out to next 2.9.2 release? 

> start-yarn and stop-yarn scripts to support timeline and sharedcachemanager
> ---
>
> Key: YARN-4858
> URL: https://issues.apache.org/jira/browse/YARN-4858
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: scripts
>Affects Versions: 2.8.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Minor
>  Labels: oct16-easy
> Attachments: YARN-4858-001.patch, YARN-4858-branch-2.001.patch
>
>
> The start-yarn and stop-yarn scripts don't have any (even commented out) 
> support for the  timeline and sharedcachemanager
> Proposed:
> * bash and cmd start-yarn scripts have commented out start actions
> * stop-yarn scripts stop the servers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7450) ATS Client should retry on intermittent Kerberos issues.

[SammiChen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383031#comment-16383031
 ] 

SammiChen commented on YARN-7450:
-

Hi [~raviprak],  does this still target for 2.9.1?  If not, can we push this 
out to next 2.9.2 release? 

> ATS Client should retry on intermittent Kerberos issues.
> 
>
> Key: YARN-7450
> URL: https://issues.apache.org/jira/browse/YARN-7450
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: ATSv2
>Affects Versions: 2.7.3
> Environment: Hadoop-2.7.3
>Reporter: Ravi Prakash
>Priority: Major
>
> We saw a stack trace (posted in the first comment) in the ResourceManager 
> logs for the TimelineClientImpl not being able to relogin from keytab.
> I'm guessing there was an intermittent issue that failed the kerberos relogin 
> from keytab. However, I'm assuming this was *not* retried because I only saw 
> one instance of this stack trace.  I propose that this operation should have 
> been retried.
> It seems, this caused events at the ResourceManager to queue up and 
> eventually stop responding to even basic {{yarn application -list}} commands.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7992) launch_container.sh has the wrong file permission

[Eric Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7992:

Description: 
Docker workload fail to launch because launch_container.sh has wrong permission 
644 instead of 700.  The pid file has permission 700 instead of 644.  This 
regression is caused by changes in YARN-7815.

Permission of files in localized directory before YARN-7815:
{code}
2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
./launch_container.sh
2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
./container_tokens
{code}

Permission of files in localized directory after YARN-7815:
{code}
total 24
drwxr-xr-x. 2 yarn hadoop 4096 Mar  2 01:20 .
drwxr-xr-x. 3 yarn hadoop   51 Mar  2 01:20 ..
-rwx--. 1 yarn hadoop4 Mar  2 01:20 
container_1519953398658_0001_01_03.pid
-rw-r--r--. 1 yarn hadoop  222 Mar  2 01:20 
container_1519953398658_0001_01_03.tokens
-rw-r--r--. 1 yarn hadoop   12 Mar  2 01:20 
.container_1519953398658_0001_01_03.tokens.crc
-rw-r--r--. 1 yarn hadoop 3799 Mar  2 01:20 launch_container.sh
-rw-r--r--. 1 yarn hadoop   40 Mar  2 01:20 .launch_container.sh.crc
{code}

  was:
Docker workload fail to launch because launch_container.sh has wrong permission 
644 instead of 700.  The pid file has permission 700 instead of 644.  This 
regression is caused by changes in YARN-7815.

Permission of files in localized directory before YARN-7815:
{code}
2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
./launch_container.sh
2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
./container_tokens
{code}

Permission of files in localized directory after YARN-7815:
{code}

{code}


> launch_container.sh has the wrong file permission
> -
>
> Key: YARN-7992
> URL: https://issues.apache.org/jira/browse/YARN-7992
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Eric Yang
>Priority: Major
>
> Docker workload fail to launch because launch_container.sh has wrong 
> permission 644 instead of 700.  The pid file has permission 700 instead of 
> 644.  This regression is caused by changes in YARN-7815.
> Permission of files in localized directory before YARN-7815:
> {code}
> 2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
> 2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
> 2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
> ./launch_container.sh
> 2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
> ./container_tokens
> {code}
> Permission of files in localized directory after YARN-7815:
> {code}
> total 24
> drwxr-xr-x. 2 yarn hadoop 4096 Mar  2 01:20 .
> drwxr-xr-x. 3 yarn hadoop   51 Mar  2 01:20 ..
> -rwx--. 1 yarn hadoop4 Mar  2 01:20 
> container_1519953398658_0001_01_03.pid
> -rw-r--r--. 1 yarn hadoop  222 Mar  2 01:20 
> container_1519953398658_0001_01_03.tokens
> -rw-r--r--. 1 yarn hadoop   12 Mar  2 01:20 
> .container_1519953398658_0001_01_03.tokens.crc
> -rw-r--r--. 1 yarn hadoop 3799 Mar  2 01:20 launch_container.sh
> -rw-r--r--. 1 yarn hadoop   40 Mar  2 01:20 .launch_container.sh.crc
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7992) launch_container.sh has the wrong file permission

[Eric Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Yang updated YARN-7992:

Description: 
Docker workload fail to launch because launch_container.sh has wrong permission 
644 instead of 700.  The pid file has permission 700 instead of 644.  This 
regression is caused by changes in YARN-7815.

Permission of files in localized directory before YARN-7815:
{code}
2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
./launch_container.sh
2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
./container_tokens
{code}

Permission of files in localized directory after YARN-7815:
{code}

{code}

  was:Docker workload fail to launch because launch_container.sh has wrong 
permission 644 instead of 700.  The pid file has permission 700 instead of 644. 
 This regression is caused by changes in YARN-7815.


> launch_container.sh has the wrong file permission
> -
>
> Key: YARN-7992
> URL: https://issues.apache.org/jira/browse/YARN-7992
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Eric Yang
>Priority: Major
>
> Docker workload fail to launch because launch_container.sh has wrong 
> permission 644 instead of 700.  The pid file has permission 700 instead of 
> 644.  This regression is caused by changes in YARN-7815.
> Permission of files in localized directory before YARN-7815:
> {code}
> 2771363410 drwxr-s---   3 hbasehadoop 65 Mar  2 00:01 .
> 2943315000 drwxr-s---   2 hbasehadoop  6 Mar  2 00:01 ./tmp
> 2771363424 -rwx--   1 hbasehadoop   3799 Mar  2 00:01 
> ./launch_container.sh
> 2771363434 -rw---   1 hbasehadoop222 Mar  2 00:01 
> ./container_tokens
> {code}
> Permission of files in localized directory after YARN-7815:
> {code}
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7707) [GPG] Policy generator framework

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382985#comment-16382985
 ] 

genericqa commented on YARN-7707:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
26s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
20s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
55s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m  
5s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 34s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in 
YARN-7402 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
9s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 3 new + 226 unchanged - 0 fixed = 229 total (was 226) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 23s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
41s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
12s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
16s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
37s{color} | {color:green} hadoop-yarn-server-globalpolicygenerator in the 
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
33s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 79m 10s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7707 |
| JIRA Patch URL | 

[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container

[Eric Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382976#comment-16382976
 ] 

Eric Yang commented on YARN-7654:
-

[~ebadger]
{quote}
Don't we need the workdir to be defined in the image? If not, the container 
will write as a user that the NM won't be able to clean up after.
{quote}

It appears that we need to support two modes of operation.  The current mode 
where workdir is bind-mounted into the same location as host to allow Hadoop 
native apps to work.  The second mode will launch container and bind-mount HDFS 
via NFS gateway for non-Hadoop workload.  Image can write to HDFS and follows 
hdfs permission rules.  There is no workdir used to write output.  Hence, the 
clean up will not be a problem.

{quote}
This approach could overwrite variables that are defined by the image. If 
variables are in the whitelist, then we want them to be set, but only if they 
aren't in the docker image. However, this would always overwrite whatever the 
image had defined
{quote}

I plan to retain the current behavior. If USE_ENTRY_POINT is enabled, then it 
follows the second method for environment construction, and user defined 
environment variable may override image supplied environment.  This depends on 
how the image is arranged.

> Support ENTRY_POINT for docker container
> 
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in 
> the current implementation.  It would be nice if we can detect existence of 
> {{launch_command}} and base on this variable launch docker container in 
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7871) Node attributes reporting from NM to RM

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382973#comment-16382973
 ] 

Naganarasimha G R commented on YARN-7871:
-

Thanks for the patch [~cheersyang], 

Just skimmed through the patch, few high level comments :
 * do we require 2 methods of replace ? i think even from CLI replace should 
not replace everything hence was thinking of having a single api same as 
earlier except for taking additional param prefix
 * NMDistributedNodeLabelsHandler earlier used to ensure that the labels were 
not sent across repeatatively in every heartbeat. I think we require that more 
here with attributes rather than labels ?

> Node attributes reporting from NM to RM 
> 
>
> Key: YARN-7871
> URL: https://issues.apache.org/jira/browse/YARN-7871
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7871-YARN-3409.001.patch, 
> YARN-7871-YARN-3409.002.patch
>
>
> Support to initialize proper attribute provider based on user's configuration.
> NM collects node attributes from a configured attribute provider and send 
> them to RM via HB. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-7988) Refactor FSNodeLabelStore code for attributes store support

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382946#comment-16382946
 ] 

Naganarasimha G R edited comment on YARN-7988 at 3/2/18 1:02 AM:
-

Thanks for working on the patch [~bibinchundatt]
 May be little write up on the class organization would have been a help here 
as it involves lot of generics. And also what gets reused in terms of Attributes

Few comments :

FSNodeStoreLogOp.java
 * ln no 46, Documentation required for understanding a class with generics. 
Else we will be lost assuming what they represent
 *  ln no 46, Also instead of W,R & M can we have OUT, IN, MGR ? or even more 
specific that OUT extends FSDataOutputStream & IN extends FSDataInputStream. 
Also are we expecting anything else over here ?
 *  ln no 46, If this class is specific to labels then we should better have it 
as FSNodeLabelStoreLogOp? abstract is only used for factoring here.
 *  ln no 53,54,57,58,59,70,100,139,179 Generics havent been used to the 
fullest and hence eclipse shows warnings @ these lines. Some can be easily 
avoided.
 * .ln no 70, getStoreOp is not used anywhere, is it required?

FileSystemNodeLabelsStore.java
 * ln no 44, AbstractFSNodeStore cant be parameterised here ? I think we can 
have the specifix manager we want here

FSNodeLogOpCode.java
 * Do we require this static operation ? cant we make use of enum's values() 
method directly ? and also given the string name of enum we can get the 
specified Enum.

FSNodeStoreMirrorOp.java
 *  do we require it to be abstract ? we are not doing any common functionality 
other than having factory kind of interface here and also IMO NodeLabelMirrorOp 
should be class alone and not embedded here

StoreOp.java
 * i think we require one more interface lile _FileStoreOP extends 
StoreOp_ to make other extending 
classes simplified

Also we have all the new classes under 
"org.apache.hadoop.yarn.nodelabels.store" package may be better to add it under 
some other package name ? And as well PackageInfo.Java is missing for the new 
package.

Also lot of checkstyle issues are there which can be addressed, please have a 
look once the approach is finalized.


was (Author: naganarasimha):
Thanks for working on the patch [~bibinchundatt]
May be little write up on the class organization would have been a help here as 
it involves lot of generics. And also what gets reused in terms of Attributes

Few comments :

FSNodeStoreLogOp.java
 * ln no 46, Documentation required for understanding a class with generics. 
Else we will be lost assuming what they represent
 *  ln no 46, Also instead of W,R & M can we have OUT, IN, MGR ? or even more 
specific that OUT extends FSDataOutputStream & IN extends FSDataInputStream. 
Also are we expecting anything else over here ?
 *  ln no 46, If this class is specific to labels then we should better have it 
as FSNodeLabelStoreLogOp? abstract is only used for factoring here.
 *  ln no 53,54,57,58,59,70,100,139,179 Generics havent been used to the 
fullest and hence eclipse shows warnings @ these lines. Some can be easily 
avoided.
 * .ln no 70, getStoreOp is not used anywhere, is it required?


FileSystemNodeLabelsStore.java
 * ln no 44, AbstractFSNodeStore cant be parameterised here ? I think we can 
have the specifix manager we want here

FSNodeLogOpCode.java
 * Do we require this static operation ? cant we make use of enum's values() 
method directly ? and also given the string name of enum we can get the 
specified Enum.

FSNodeStoreMirrorOp.java
 *  do we require it to be abstract ? we are not doing any common functionality 
other than having factory kind of interface here and also IMO NodeLabelMirrorOp 
should be class alone and not embedded here

StoreOp.java
 * i think we require one more interface lile _FileStoreOP extends 
StoreOp_ to make other extending 
classes simplified


Also we have all the new classes under 
"org.apache.hadoop.yarn.nodelabels.store" package may be better to add it under 
some other package name ? And as well PackageInfo.Java is missing for the new 
package

> Refactor FSNodeLabelStore code for attributes store support
> ---
>
> Key: YARN-7988
> URL: https://issues.apache.org/jira/browse/YARN-7988
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7988-YARN-3409.002.patch, YARN-7988.001.patch
>
>
> # Abstract out file FileSystemStore operation
> # Define EditLog Operartions  and Mirror operation
> # Support compatibility with old nodelabel store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (YARN-7988) Refactor FSNodeLabelStore code for attributes store support

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382946#comment-16382946
 ] 

Naganarasimha G R commented on YARN-7988:
-

Thanks for working on the patch [~bibinchundatt]
May be little write up on the class organization would have been a help here as 
it involves lot of generics. And also what gets reused in terms of Attributes

Few comments :

FSNodeStoreLogOp.java
 * ln no 46, Documentation required for understanding a class with generics. 
Else we will be lost assuming what they represent
 *  ln no 46, Also instead of W,R & M can we have OUT, IN, MGR ? or even more 
specific that OUT extends FSDataOutputStream & IN extends FSDataInputStream. 
Also are we expecting anything else over here ?
 *  ln no 46, If this class is specific to labels then we should better have it 
as FSNodeLabelStoreLogOp? abstract is only used for factoring here.
 *  ln no 53,54,57,58,59,70,100,139,179 Generics havent been used to the 
fullest and hence eclipse shows warnings @ these lines. Some can be easily 
avoided.
 * .ln no 70, getStoreOp is not used anywhere, is it required?


FileSystemNodeLabelsStore.java
 * ln no 44, AbstractFSNodeStore cant be parameterised here ? I think we can 
have the specifix manager we want here

FSNodeLogOpCode.java
 * Do we require this static operation ? cant we make use of enum's values() 
method directly ? and also given the string name of enum we can get the 
specified Enum.

FSNodeStoreMirrorOp.java
 *  do we require it to be abstract ? we are not doing any common functionality 
other than having factory kind of interface here and also IMO NodeLabelMirrorOp 
should be class alone and not embedded here

StoreOp.java
 * i think we require one more interface lile _FileStoreOP extends 
StoreOp_ to make other extending 
classes simplified


Also we have all the new classes under 
"org.apache.hadoop.yarn.nodelabels.store" package may be better to add it under 
some other package name ? And as well PackageInfo.Java is missing for the new 
package

> Refactor FSNodeLabelStore code for attributes store support
> ---
>
> Key: YARN-7988
> URL: https://issues.apache.org/jira/browse/YARN-7988
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7988-YARN-3409.002.patch, YARN-7988.001.patch
>
>
> # Abstract out file FileSystemStore operation
> # Define EditLog Operartions  and Mirror operation
> # Support compatibility with old nodelabel store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3610) FairScheduler: Add steady-fair-shares to the REST API documentation

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382942#comment-16382942
 ] 

genericqa commented on YARN-3610:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
24m 37s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 36s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 42s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-3610 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912697/YARN-3610.003.patch |
| Optional Tests |  asflicense  mvnsite  |
| uname | Linux 4d61ec8366d0 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 923e177 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 434 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/19866/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> FairScheduler: Add steady-fair-shares to the REST API documentation
> ---
>
> Key: YARN-3610
> URL: https://issues.apache.org/jira/browse/YARN-3610
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: documentation, fairscheduler
>Affects Versions: 2.7.0
>Reporter: Karthik Kambatla
>Assignee: Ray Chiang
>Priority: Major
> Attachments: YARN-3610.001.patch, YARN-3610.002.patch, 
> YARN-3610.003.patch
>
>
> YARN-1050 adds documentation for FairScheduler REST API, but is missing the 
> steady-fair-share.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7952) Find a way to persist the log aggregation status

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382934#comment-16382934
 ] 

Wangda Tan commented on YARN-7952:
--

Thanks Xuan Gong for working on the patch,

A quick question, do we need to introduce a new timeout config to YarnConfig? 
(IIRC we have a setting in RM to tell if application log aggregation is 
timeout, if we have that, can we use the same config?) I think it is: 
org.apache.hadoop.yarn.conf.YarnConfiguration#LOG_AGGREGATION_STATUS_TIME_OUT_MS

Will include a thorough review in the next comment.

> Find a way to persist the log aggregation status
> 
>
> Key: YARN-7952
> URL: https://issues.apache.org/jira/browse/YARN-7952
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Major
> Attachments: YARN-7952-poc.patch, YARN-7952.1.patch
>
>
> In MAPREDUCE-6415, we have created a CLI to har the aggregated logs, and In 
> YARN-4946: RM should write out Aggregated Log Completion file flag next to 
> logs, we have a discussion on how we can get the log aggregation status: make 
> a client call to RM or get it directly from the Distributed file system(HDFS).
> No matter which approach we would like to choose, we need to figure out a way 
> to persist the log aggregation status first. This ticket is used to track the 
> working progress for this purpose.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7992) launch_container.sh has the wrong file permission

[Eric Yang (JIRA)]

Eric Yang created YARN-7992:
---

 Summary: launch_container.sh has the wrong file permission
 Key: YARN-7992
 URL: https://issues.apache.org/jira/browse/YARN-7992
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Eric Yang


Docker workload fail to launch because launch_container.sh has wrong permission 
644 instead of 700.  The pid file has permission 700 instead of 644.  This 
regression is caused by changes in YARN-7815.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7707) [GPG] Policy generator framework

[Young Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Young Chen updated YARN-7707:
-
Attachment: YARN-7707-YARN-7402.08.patch

> [GPG] Policy generator framework
> 
>
> Key: YARN-7707
> URL: https://issues.apache.org/jira/browse/YARN-7707
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Young Chen
>Priority: Major
>  Labels: federation, gpg
> Attachments: YARN-7707-YARN-7402.01.patch, 
> YARN-7707-YARN-7402.02.patch, YARN-7707-YARN-7402.03.patch, 
> YARN-7707-YARN-7402.04.patch, YARN-7707-YARN-7402.05.patch, 
> YARN-7707-YARN-7402.06.patch, YARN-7707-YARN-7402.07.patch, 
> YARN-7707-YARN-7402.08.patch
>
>
> This JIRA tracks the development of a generic framework for querying 
> sub-clusters for metrics, running policies, and updating them in the 
> FederationStateStore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7707) [GPG] Policy generator framework

[Young Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382920#comment-16382920
 ] 

Young Chen commented on YARN-7707:
--

Accidentally submitted a patch that includes YARN-7708. Adding a corrected 
patch.

> [GPG] Policy generator framework
> 
>
> Key: YARN-7707
> URL: https://issues.apache.org/jira/browse/YARN-7707
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Young Chen
>Priority: Major
>  Labels: federation, gpg
> Attachments: YARN-7707-YARN-7402.01.patch, 
> YARN-7707-YARN-7402.02.patch, YARN-7707-YARN-7402.03.patch, 
> YARN-7707-YARN-7402.04.patch, YARN-7707-YARN-7402.05.patch, 
> YARN-7707-YARN-7402.06.patch, YARN-7707-YARN-7402.07.patch
>
>
> This JIRA tracks the development of a generic framework for querying 
> sub-clusters for metrics, running policies, and updating them in the 
> FederationStateStore.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-3610) FairScheduler: Add steady-fair-shares to the REST API documentation

[Ray Chiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-3610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated YARN-3610:
-
Attachment: YARN-3610.003.patch

> FairScheduler: Add steady-fair-shares to the REST API documentation
> ---
>
> Key: YARN-3610
> URL: https://issues.apache.org/jira/browse/YARN-3610
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: documentation, fairscheduler
>Affects Versions: 2.7.0
>Reporter: Karthik Kambatla
>Assignee: Ray Chiang
>Priority: Major
> Attachments: YARN-3610.001.patch, YARN-3610.002.patch, 
> YARN-3610.003.patch
>
>
> YARN-1050 adds documentation for FairScheduler REST API, but is missing the 
> steady-fair-share.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7626) Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382908#comment-16382908
 ] 

genericqa commented on YARN-7626:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
 7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
24m 58s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 41s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 
10s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 38s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7626 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912683/YARN-7626.010.patch |
| Optional Tests |  asflicense  compile  cc  mvnsite  javac  unit  |
| uname | Linux a55fa8e8d64c 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 29233c3 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/19864/testReport/ |
| Max. process+thread count | 473 (vs. ulimit of 1) |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/19864/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Allow regular expression matching in container-executor.cfg for devices and 
> named docker volumes mount
> --
>
> Key: YARN-7626
> URL: https://issues.apache.org/jira/browse/YARN-7626
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-7626.001.patch, YARN-7626.002.patch, 
> YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch, 
> YARN-7626.006.patch, YARN-7626.007.patch, YARN-7626.008.patch, 
> YARN-7626.009.patch, YARN-7626.010.patch
>
>
> Currently when we config some of the GPU devices related fields (like ) in 

[jira] [Commented] (YARN-7626) Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount

[Zian Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382887#comment-16382887
 ] 

Zian Chen commented on YARN-7626:
-

Address failed test cases and rebuild the patch

> Allow regular expression matching in container-executor.cfg for devices and 
> named docker volumes mount
> --
>
> Key: YARN-7626
> URL: https://issues.apache.org/jira/browse/YARN-7626
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-7626.001.patch, YARN-7626.002.patch, 
> YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch, 
> YARN-7626.006.patch, YARN-7626.007.patch, YARN-7626.008.patch, 
> YARN-7626.009.patch, YARN-7626.010.patch
>
>
> Currently when we config some of the GPU devices related fields (like ) in 
> container-executor.cfg, these fields are generated based on different driver 
> versions or GPU device names. We want to enable regular expression matching 
> so that user don't need to manually set up these fields when config 
> container-executor.cfg,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382864#comment-16382864
 ] 

Wangda Tan commented on YARN-7494:
--

Thanks [~sunilg] for updating the patch, reviewed overall workflow of the 
patch, comments: 
1) PartitionBasedCandidateNodeSet: It doesn't look like added anything other 
than SimpleCandidateNodeSet, maybe reuse the same class?

2) CS#getCandidateNodeSet, the get_node_nodes_from_give_partition operation is 
very expensive, maybe cache partition -> nodes inside ClusterNodeTracker?

3) Ideally, MultiNodeLookupPolicy initialization should not determined by 
Scheduler, and implementation of MultiNodeSortingManager should not bind to 
specific scheduler.
{code} 
CapacityScheduler cs = ((CapacityScheduler) rmContext.getResourceManager()
.getResourceScheduler());
nodeLookupPolicy = (MultiNodeLookupPolicy) cs.getMultiNodeSortingManager();
{code}

3.1 type cast from MultiNodeSortingManager to MultiNodeLookupPolicy looks 
incorrect.
3.2 Inside MultiNodeSortingManager, the type cast to {{CapacityScheduler}} 
looks unnecessary since we already made class to be a generic type: {{}}

4) Suggestions to MultiNodeLookupPolicy (and other new added interfaces) design:

In order to support application specify its own sorting policy, we may need to 
take application's input to initialize policy.
- Existing implementation of MultiNodeLookupPolicy initialization code assumed 
it is shared by apps. (All apps with the same policy can get the same sorting 
order). However, we need both: for example, load balance sorter can be shared 
by apps, but locality sorter should be different for different apps. So to make 
it work, I suggest to add a MultiNodeLookupPolicyFactory class, depends on user 
requested policy class. It can either return whatever cached in 
{{MultiNodeSortingManager}} (which assumes it will be shared to apps), or 
initialize new policies. Class of MultiNodeLookupPolicy should be determined by 
{{applicationSchedulingEnvs}}.
- Add MultiNodeLookupPolicy to {{AppPlacementAllocator#initialize}} parameter 
list.
- {{MultiNodeLookupPolicy#initPolicy}} should take 
{{AppSchedulingInfo#applicationSchedulingEnvs}} as input to initialize its own 
fields. The multiNodeSortingAlgorithm should not be a separate item in the 
initPolicy parameter list.
- {{MultiNodeLookupPolicy#addAndRefreshNodesSet}} Should take CandidateNodeSet 
as input, we can include version to let MultiNodeLookupPolicy if it need to 
invalidated cached sorted result or not.

5) Typo:
bq. LOG.error("Exception raised while executing preemption"


> Add muti node lookup support for better placement
> -
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch, 
> YARN-7494.v0.patch, YARN-7494.v1.patch, multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup 
> based on partition to start with.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-4781) Support intra-queue preemption for fairness ordering policy.

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382847#comment-16382847
 ] 

Eric Payne edited comment on YARN-4781 at 3/1/18 11:24 PM:
---

A lot has happened since this JIRA was opened, but I think there is still value 
in pursuing the original intent. That is, intra-queue preemption should 
consider FairOrderingPolicy.
{quote}Currently, if a job in queue A is using 100% of the cluster resources, 
and a new job arrives in queue A, it sometimes cannot even get an application 
master!
{quote}
{quote}one big query is taking all resources of a queue lets say Q1. And when i 
am launching another query in Q1, almost always it is hanging in ACCEPTED
{quote}
[~milesc] and [~anuptiwari], I think this use case is covered by YARN-2009 and 
related JIRAs. I think this JIRA covers a slightly different use case.

FairOrderingPolicy tries to evenly assign containers across users and across 
apps within a user (as long as the user is below the user limit). Currently, 
the FairOrderingPolicy does not honor application priority AFAICT.

We have seen the following use case in a large and extremely busy queue where 
we have FairOrderingPolicy set, one user takes up a lot of the queue, and then 
other, later users, fight for the remaining resources, with the youngest users 
/ apps getting constantly preempted while the larger, older user is not 
preempted.

For example,
 QueueA: minimum-user-limit-percent = 25
 QueueA: resources = 1000
| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|300|0|0|
|User3 / App3|300|0|0|
|User4 / App4|0|100|0|
 - Intra-queue preemption preempts 50 from App2 and 50 from App3.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|250|0|50|
|User3 / App3|250|0|50|
|User4 / App4|100|0|0|
 - App3 finishes and resources are given back to App2 and App3.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|300|0|50|
|User3 / App3|300|0|50|
 - Then, User4 submits App5, and the process repeates.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|250|0|100|
|User3 / App3|250|0|100|
|User4 / App5|100|0|0|

Then, while all 4 users have running apps, User5 comes along and can't get any 
resources, they see that User1 is using 62% more resources than everyone else, 
and wonders why they can't get any resources. (yes, I recognize the reason in 
this case is because MULP = 25%, but I'm trying to make the user case simple).

This is somewhat simplified because in our case, we have up to 50 active users, 
and since the queue is large, the difference between the largest user and the 
others is even more apparent.

 

[~sunilg] and [~leftnoteasy], Thoughts?


was (Author: eepayne):
A lot has happened since this JIRA was opened, but I think there is still value 
in pursuing the original intent. That is, intra-queue preemption should 
consider FairOrderingPolicy.
{quote}Currently, if a job in queue A is using 100% of the cluster resources, 
and a new job arrives in queue A, it sometimes cannot even get an application 
master!
{quote}
{quote}one big query is taking all resources of a queue lets say Q1. And when i 
am launching another query in Q1, almost always it is hanging in ACCEPTED
{quote}
[~milesc] and [~anuptiwari], I think this use case is covered by YARN-2009 and 
related JIRAs. I think this JIRA covers a slightly different use case.

FairOrderingPolicy tries to evenly assign containers across users and across 
apps within a user (as long as the user is below the user limit). Currently, 
the FairOrderingPolicy does not honor application priority AFAICT.

We have seen the following use case in a large and extremely busy queue where 
we have FairOrderingPolicy set, one user takes up a lot of the queue, and then 
other, later users, fight for the remaining resources, with the youngest users 
/ apps getting constantly preempted while the larger, older user is not 
preempted.

For example,
 QueueA: minimum-user-limit-percent = 25
 QueueA: resources = 1000
| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|300|0|0|
|User3 / App3|300|0|0|
|User4 / App4|0|100|0|
 - Intra-queue preemption preempts 50 from App2 and 50 from App3.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|250|0|50|
|User3 / App3|250|0|50|
|User4 / App4|100|0|0|
 - App3 finishes and resources are given back to App2 and App3.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|300|0|50|
|User3 / App3|300|0|50|
 - Then, User4 submits App5, and the process repeates.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|250|0|100|
|User3 / App3|250|0|100|
|User4 / App5|100|0|0|

Then, while all 4 users have running apps, User5 comes along and can't get any 
resources, they see that User1 is using 62% more resources than everyone else, 
and wonders why they can't get any resources. (yes, I recognize the reason in 
this 

[jira] [Commented] (YARN-4781) Support intra-queue preemption for fairness ordering policy.

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382847#comment-16382847
 ] 

Eric Payne commented on YARN-4781:
--

A lot has happened since this JIRA was opened, but I think there is still value 
in pursuing the original intent. That is, intra-queue preemption should 
consider FairOrderingPolicy.
{quote}Currently, if a job in queue A is using 100% of the cluster resources, 
and a new job arrives in queue A, it sometimes cannot even get an application 
master!
{quote}
{quote}one big query is taking all resources of a queue lets say Q1. And when i 
am launching another query in Q1, almost always it is hanging in ACCEPTED
{quote}
[~milesc] and [~anuptiwari], I think this use case is covered by YARN-2009 and 
related JIRAs. I think this JIRA covers a slightly different use case.

FairOrderingPolicy tries to evenly assign containers across users and across 
apps within a user (as long as the user is below the user limit). Currently, 
the FairOrderingPolicy does not honor application priority AFAICT.

We have seen the following use case in a large and extremely busy queue where 
we have FairOrderingPolicy set, one user takes up a lot of the queue, and then 
other, later users, fight for the remaining resources, with the youngest users 
/ apps getting constantly preempted while the larger, older user is not 
preempted.

For example,
 QueueA: minimum-user-limit-percent = 25
 QueueA: resources = 1000
| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|300|0|0|
|User3 / App3|300|0|0|
|User4 / App4|0|100|0|
 - Intra-queue preemption preempts 50 from App2 and 50 from App3.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|250|0|50|
|User3 / App3|250|0|50|
|User4 / App4|100|0|0|
 - App3 finishes and resources are given back to App2 and App3.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|300|0|50|
|User3 / App3|300|0|50|
 - Then, User4 submits App5, and the process repeates.

| |Used|Pending|Preempted|
|User1 / App1|400|0|0|
|User2 / App2|250|0|100|
|User3 / App3|250|0|100|
|User4 / App5|100|0|0|

Then, while all 4 users have running apps, User5 comes along and can't get any 
resources, they see that User1 is using 62% more resources than everyone else, 
and wonders why they can't get any resources. (yes, I recognize the reason in 
this case is because MULP = 25%, but I'm trying to make the user case simple).

This is somewhat simplified because in our case, we have up to 50 active users, 
and since the queue is large, the difference between the largest user and the 
others is even more apparent.

> Support intra-queue preemption for fairness ordering policy.
> 
>
> Key: YARN-4781
> URL: https://issues.apache.org/jira/browse/YARN-4781
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Major
>
> We introduced fairness queue policy since YARN-3319, which will let large 
> applications make progresses and not starve small applications. However, if a 
> large application takes the queue’s resources, and containers of the large 
> app has long lifespan, small applications could still wait for resources for 
> long time and SLAs cannot be guaranteed.
> Instead of wait for application release resources on their own, we need to 
> preempt resources of queue with fairness policy enabled.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7221) Add security check for privileged docker container

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382846#comment-16382846
 ] 

genericqa commented on YARN-7221:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 24s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 48s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 19m  9s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
18s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 53s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7221 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12911425/YARN-7221.005.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  cc  |
| uname | Linux b93a8973a95a 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 96e8f26 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| whitespace | 
https://builds.apache.org/job/PreCommit-YARN-Build/19863/artifact/out/whitespace-eol.txt
 |
| unit | 

[jira] [Updated] (YARN-7626) Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount

[Zian Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Zian Chen updated YARN-7626:

Attachment: YARN-7626.010.patch

> Allow regular expression matching in container-executor.cfg for devices and 
> named docker volumes mount
> --
>
> Key: YARN-7626
> URL: https://issues.apache.org/jira/browse/YARN-7626
> Project: Hadoop YARN
>  Issue Type: New Feature
>Reporter: Zian Chen
>Assignee: Zian Chen
>Priority: Major
> Attachments: YARN-7626.001.patch, YARN-7626.002.patch, 
> YARN-7626.003.patch, YARN-7626.004.patch, YARN-7626.005.patch, 
> YARN-7626.006.patch, YARN-7626.007.patch, YARN-7626.008.patch, 
> YARN-7626.009.patch, YARN-7626.010.patch
>
>
> Currently when we config some of the GPU devices related fields (like ) in 
> container-executor.cfg, these fields are generated based on different driver 
> versions or GPU device names. We want to enable regular expression matching 
> so that user don't need to manually set up these fields when config 
> container-executor.cfg,



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4488) CapacityScheduler: Compute per-container allocation latency and roll up to get per-application and per-queue

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382744#comment-16382744
 ] 

Wangda Tan commented on YARN-4488:
--

[~maniraj...@gmail.com], thanks for the explanation, I can understand the 
approach better now. 

Regarding to the metrics, here's what I expected behavior: 

Delay of container should be T1 (container_allocated_time) - T2 (requested 
time). In your proposal, T2 is {{time while creating ResourceRequest object}}, 
which may not be correct to me. We have to consider a complex case. 

What I expected:
{code:java}
(time=1) An app has a resource request asks 5 * 2G containers
(time=3) 3 containers allocated, delay of the 3 containers = 2. Pending ask = 2
(time=5) App requested 10 containers (instead of 2) on the same priority.
(time=7) 5 containers allocated, 2 containers have delay (which is from the 
original ask) has delay = 7-1 = 6
 And 3 containers have delay (which is from the additional ask) = 7-5 = 
2{code}
This is a common scenario for apps have additional asks for failed containers 
(for example MR), if a container failed, it will ask additional containers use 
the same priority (FAILED_MAPPER_PRIORITY), so we should consider it.

The downside of this approach is it needs additional memory to record accurate 
requested time for each resource request. An alternative approach is remember 
an average requested time for each priority. (Assume we have X container 
requested at T1, Y additional container requested at T2, the average time will 
be {{(X * T1 + Y * T2) / (X + Y)}}). 

*Regarding to implementation:* 

I'm not sure if a massive changes required, let's figure out semantics of the 
delay first, and look at implementation later.

+ [~sunilg] 

+ [~ywskycn] to the thread: You pinged me offline about metrics related stuffs 
before, I think you might be interested about this Jira.

 

 

> CapacityScheduler: Compute per-container allocation latency and roll up to 
> get per-application and per-queue
> 
>
> Key: YARN-4488
> URL: https://issues.apache.org/jira/browse/YARN-4488
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Karthik Kambatla
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-4485.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382721#comment-16382721
 ] 

Wangda Tan commented on YARN-7935:
--

Thanks [~suma.shivaprasad] for working on the change, several questions while 
reviewing the patch: 

1) This patch introduced a new env to let apps set hostname of containers by 
theirselves, is this required? Or letting YARN set container's hostname is 
enough. And I'm not sure if allow app to choose container hostname could 
introduce any risks to the cluster or not. 

2) Regarding to implementation: {{a.setHostnameinEnv}} is called by 
{{prepareContainer}}, and {{b.setHostname}} is called by {{launchContainer}}. 
Since a is called prior to b, does this mean all user specified hostname will 
be overwrite by YARN defined hostname (which is generated from container-id).

> Expose container's hostname to applications running within the docker 
> container
> ---
>
> Key: YARN-7935
> URL: https://issues.apache.org/jira/browse/YARN-7935
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
> Attachments: YARN-7935.1.patch, YARN-7935.2.patch, YARN-7935.3.patch
>
>
> Some applications have a need to bind to the container's hostname (like 
> Spark) which is different from the NodeManager's hostname(NM_HOST which is 
> available as an env during container launch) when launched through Docker 
> runtime. The container's hostname can be exposed to applications via an env 
> CONTAINER_HOSTNAME. Another potential candidate is the container's IP but 
> this can be addressed in a separate jira.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382714#comment-16382714
 ] 

genericqa commented on YARN-7599:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
31s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
38s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  7m 
48s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 3s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
33s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m 26s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
11s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in 
YARN-7402 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
7s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green}  6m 
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
42s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 230 unchanged - 0 fixed = 231 total (was 230) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red}  0m  
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply 
{color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 18s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  4m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
17s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  2m 
27s{color} | {color:green} hadoop-yarn-server-common in the patch passed. 
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
32s{color} | {color:green} hadoop-yarn-server-globalpolicygenerator in the 
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 40s{color} | 
{color:black} {color} |
\\
\\
|| 

[jira] [Commented] (YARN-7952) Find a way to persist the log aggregation status

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382678#comment-16382678
 ] 

genericqa commented on YARN-7952:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m  
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  6s{color} 
| {color:red} YARN-7952 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7952 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912615/YARN-7952.1.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/19862/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Find a way to persist the log aggregation status
> 
>
> Key: YARN-7952
> URL: https://issues.apache.org/jira/browse/YARN-7952
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Major
> Attachments: YARN-7952-poc.patch, YARN-7952.1.patch
>
>
> In MAPREDUCE-6415, we have created a CLI to har the aggregated logs, and In 
> YARN-4946: RM should write out Aggregated Log Completion file flag next to 
> logs, we have a discussion on how we can get the log aggregation status: make 
> a client call to RM or get it directly from the Distributed file system(HDFS).
> No matter which approach we would like to choose, we need to figure out a way 
> to persist the log aggregation status first. This ticket is used to track the 
> working progress for this purpose.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0

[Sean Busbey (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382652#comment-16382652
 ] 

Sean Busbey commented on YARN-7346:
---

Sorry, still catching up on things and haven't gotten to dig in here yet. but 
one quick bit:

{quote}
The findbugs & javadocs from yetus will fail for one hbase module when the 
compilation profile is set for the other hbase version. But we can check the 
jenkins builds for patches for each hbase version and confirm that the patches 
are good for their respective hbase versions.
{quote}

We should fix this by updating the Hadoop personality so that we only run in 
each module with an appropriate profile active

> Add a profile to allow optional compilation for ATSv2 with HBase-2.0
> 
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> [https://pastebin.com/Ms4jYEVB]
> This issue is to fix the compilation errors.
> The scope of the Jira is to add a profile to allow optional compilation for 
> ATSv2 with HBase2.0. The default compilation for trunk will still be for 
> hbase 1.2.6. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7891) LogAggregationIndexedFileController should support HAR file

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382612#comment-16382612
 ] 

Wangda Tan commented on YARN-7891:
--

Thanks [~xgong]

*Overall questions:*

What's the JIRA about? I'm trying to review the patch, but it looks like 
existing code (see below) already handled har file, so what's new added by the 
patch? Could you elaborate this in description?
{code:java}
for (FileStatus file : files) {
  String nodeName = file.getPath().getName();
  if ((nodeId == null || nodeId.isEmpty()
  || nodeName.contains(LogAggregationUtils
  .getNodeString(nodeId))) && !nodeName.endsWith(
  LogAggregationUtils.TMP_FILE_SUFFIX) &&
  !nodeName.endsWith(CHECK_SUM_FILE_SUFFIX)) {
if (nodeName.equals(appId + ".har")) {
  Path p = new Path("har:///" + file.getPath().toUri().getRawPath());
  files = Arrays.asList(HarFs.get(p.toUri(), conf).listStatus(p));
  continue;
}
listOfFiles.add(file);
  }
}
{code}
And this looks a bit confusing to me:
{code:java}
 
files = Arrays.asList(HarFs.get(p.toUri(), conf).listStatus(p));
{code}
Instead of replacing the iterating var, should we call 
listOfFiles.add(HarFs.get(p.toUri(), conf).listStatus(p)) instead? What if 
there're more files apart from the .har file? If it is not possible, should we 
check this and throw exception if other files exist in the same folder of .har?

*Minor:*

1) In {{readAggregatedLogs}}/{{readAggregatedLogsMeta}}, following logics are 
mostly identical, could you merge common logics if possible?
{code:java}
RemoteIterator nodeFiles = LogAggregationUtils
.getRemoteNodeFileDir(conf, appId, appOwner, this.remoteRootLogDir,
this.remoteRootLogDirSuffix);
if (!nodeFiles.hasNext()) {
  throw new IOException("There is no available log fils for "
  + "application:" + appId);
}
List allFiles = getAllNodeFiles(nodeFiles, appId);
if (allFiles.isEmpty()) {
  throw new IOException("There is no available log fils for "
  + "application:" + appId);
}
Map checkSumFiles = parseCheckSumFiles(allFiles);
List fileToRead = getNodeLogFileToRead(
allFiles, nodeIdStr, appId);
{code}

> LogAggregationIndexedFileController should support HAR file
> ---
>
> Key: YARN-7891
> URL: https://issues.apache.org/jira/browse/YARN-7891
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Major
> Attachments: YARN-7891.1.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7599) [GPG] ApplicationCleaner in Global Policy Generator

[Botong Huang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Botong Huang updated YARN-7599:
---
Attachment: YARN-7599-YARN-7402.v2.patch

> [GPG] ApplicationCleaner in Global Policy Generator
> ---
>
> Key: YARN-7599
> URL: https://issues.apache.org/jira/browse/YARN-7599
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Minor
>  Labels: federation, gpg
> Attachments: YARN-7599-YARN-7402.v1.patch, 
> YARN-7599-YARN-7402.v2.patch
>
>
> In Federation, we need a cleanup service for StateStore as well as Yarn 
> Registry. For the former, we need to remove old application records. For the 
> latter, failed and killed applications might leave records in the Yarn 
> Registry (see YARN-6128). We plan to do both cleanup work in 
> ApplicationCleaner in GPG



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7654) Support ENTRY_POINT for docker container

[Eric Badger (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382590#comment-16382590
 ] 

Eric Badger commented on YARN-7654:
---

bq. Will we eliminate setting the workdir in this model and leave that up to 
the image? This has been a complaint I've heard with the bring your own image 
model.
Don't we need the workdir to be defined in the image? If not, the container 
will write as a user that the NM won't be able to clean up after. 

{quote}
The new proposal is to change construct_docker_command as:

exporting all environment and command line arguments to a template that looks 
like:

{noformat}
#!/bin/bash
{ { docker run ... -e K=V -e K-V [image-name] } > >(tee stdout.txt ); } \
  2> >(tee stderr.txt >&2 )
{noformat}
{quote}
This approach could overwrite variables that are defined by the image. If 
variables are in the whitelist, then we want them to be set, but only if they 
aren't in the docker image. However, this would always overwrite whatever the 
image had defined

> Support ENTRY_POINT for docker container
> 
>
> Key: YARN-7654
> URL: https://issues.apache.org/jira/browse/YARN-7654
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Affects Versions: 3.1.0
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
>
> Docker image may have ENTRY_POINT predefined, but this is not supported in 
> the current implementation.  It would be nice if we can detect existence of 
> {{launch_command}} and base on this variable launch docker container in 
> different ways:
> h3. Launch command exists
> {code}
> docker run [image]:[version]
> docker exec [container_id] [launch_command]
> {code}
> h3. Use ENTRY_POINT
> {code}
> docker run [image]:[version]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7637) GPU volume creation command fails when work preserving is disabled at NM

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-7637:
-
Fix Version/s: (was: 3.2.0)

> GPU volume creation command fails when work preserving is disabled at NM
> 
>
> Key: YARN-7637
> URL: https://issues.apache.org/jira/browse/YARN-7637
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Sunil G
>Assignee: Zian Chen
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: YARN-7637.001.patch
>
>
> When work preserving is disabled, NM uses {{NMNullStateStoreService}}. Hence 
> resource mappings related to GPU wont be saved at Container.
> This has to  be rechecked and store accordingly.
> cc/ [~leftnoteasy] and [~Zian Chen]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7944) Remove master node link from headers of application pages

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382556#comment-16382556
 ] 

genericqa commented on YARN-7944:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
 8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
27m 14s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 17s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7944 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912635/YARN-7944.003.patch |
| Optional Tests |  asflicense  shadedclient  |
| uname | Linux 78b36fe158b4 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 22928c0 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/19860/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Remove master node link from headers of application pages
> -
>
> Key: YARN-7944
> URL: https://issues.apache.org/jira/browse/YARN-7944
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7944.001.patch, YARN-7944.002.patch, 
> YARN-7944.003.patch
>
>
> Rm UI2 has links for Master container log and master node. 
> This link published on application and service page. This links are not 
> required on all pages because AM container node link and container log link 
> are already present in Application view. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382541#comment-16382541
 ] 

Arun Suresh commented on YARN-7953:
---

[~curino], Thanks for working on this.

I am thinking though, there might be a far easier way to encapsulate send 
Scheduler State.

* A Scheduler maintains QueueMetrics, which itself is hierarchical.
* The queuemetrics for all queues are query-able via the JMX console as simple 
KV pairs.
* The GPG just needs to periodically pull the queue-metrics of each 
participation RM in the federation and for each key, diff with the previous 
value it received - which will result in the delta - which is what we are 
interested in I believe.

> [GQ] Data structures for federation global queues calculations
> --
>
> Key: YARN-7953
> URL: https://issues.apache.org/jira/browse/YARN-7953
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Carlo Curino
>Assignee: Carlo Curino
>Priority: Major
> Attachments: YARN-7953.v1.patch
>
>
> This Jira tracks data structures and helper classes used by the core 
> algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7991) Use ServiceState values to publish to ATS

[Gour Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382496#comment-16382496
 ] 

Gour Saha commented on YARN-7991:
-

I will work on a patch for this.

> Use ServiceState values to publish to ATS
> -
>
> Key: YARN-7991
> URL: https://issues.apache.org/jira/browse/YARN-7991
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Gour Saha
>Priority: Major
>
> Add the state DELETED to ServiceState and then use ServiceState values to 
> publish to ATS (instead of FinalApplicationStatus). 
> Refer to parent issue for more details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-7991) Use ServiceState values to publish to ATS

[Gour Saha (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gour Saha reassigned YARN-7991:
---

Assignee: Gour Saha

> Use ServiceState values to publish to ATS
> -
>
> Key: YARN-7991
> URL: https://issues.apache.org/jira/browse/YARN-7991
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-native-services
>Reporter: Gour Saha
>Assignee: Gour Saha
>Priority: Major
>
> Add the state DELETED to ServiceState and then use ServiceState values to 
> publish to ATS (instead of FinalApplicationStatus). 
> Refer to parent issue for more details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-7991) Use ServiceState values to publish to ATS

[Gour Saha (JIRA)]

Gour Saha created YARN-7991:
---

 Summary: Use ServiceState values to publish to ATS
 Key: YARN-7991
 URL: https://issues.apache.org/jira/browse/YARN-7991
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: yarn-native-services
Reporter: Gour Saha


Add the state DELETED to ServiceState and then use ServiceState values to 
publish to ATS (instead of FinalApplicationStatus). 

Refer to parent issue for more details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7944) Remove master node link from headers of application pages

[Yesha Vora (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yesha Vora updated YARN-7944:
-
Attachment: YARN-7944.003.patch

> Remove master node link from headers of application pages
> -
>
> Key: YARN-7944
> URL: https://issues.apache.org/jira/browse/YARN-7944
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-7944.001.patch, YARN-7944.002.patch, 
> YARN-7944.003.patch
>
>
> Rm UI2 has links for Master container log and master node. 
> This link published on application and service page. This links are not 
> required on all pages because AM container node link and container log link 
> are already present in Application view. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7988) Refactor FSNodeLabelStore code for attributes store support

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382467#comment-16382467
 ] 

genericqa commented on YARN-7988:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-3409 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  3m  
7s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
42s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m  
8s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  1m 
 2s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
32s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
12m 44s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
31s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
17s{color} | {color:green} YARN-3409 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  6m 
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  6m 
56s{color} | {color:green} hadoop-yarn-project_hadoop-yarn generated 0 new + 86 
unchanged - 1 fixed = 86 total (was 87) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 38 new + 52 unchanged - 19 fixed = 90 total (was 71) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 54s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m  
3s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 31s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
39s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}142m 32s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels |
|   | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7988 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912622/YARN-7988-YARN-3409.002.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux fd8b3f87d1ea 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 

[jira] [Commented] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0

[Vrushali C (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382429#comment-16382429
 ] 

Vrushali C commented on YARN-7346:
--

[~rohithsharma] and I had an offline discussion. We had earlier proposes to 
have a default branch that will have compilation for hbase 2.x. Let's get that 
going so that we can track the findbugs & javadoc compilation success/failures 
for both hbase compilation profiles in this jira.

 

The findbugs & javadocs from yetus will fail for one hbase module when the 
compilation profile is set for the other hbase version. But we can check the 
jenkins builds for patches for each hbase version and confirm that the patches 
are good for their respective hbase versions.

 

> Add a profile to allow optional compilation for ATSv2 with HBase-2.0
> 
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> [https://pastebin.com/Ms4jYEVB]
> This issue is to fix the compilation errors.
> The scope of the Jira is to add a profile to allow optional compilation for 
> ATSv2 with HBase2.0. The default compilation for trunk will still be for 
> hbase 1.2.6. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7986) ATSv2 REST API queries do not return results for uppercase application tags

[Haibo Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382420#comment-16382420
 ] 

Haibo Chen commented on YARN-7986:
--

Agree this is a tag issue, not necessarily some problem with ATSv2. The case 
insensitivity is a conscious design choice from the git log. I think it 
suffices to document in TimelineServiceV2.md.

> ATSv2 REST API queries do not return results for uppercase application tags
> ---
>
> Key: YARN-7986
> URL: https://issues.apache.org/jira/browse/YARN-7986
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Charan Hebri
>Priority: Critical
>
> When applications are submitted to YARN with application tags, the tags are 
> converted to lowercase. This can be seen on the old/new UI. But using the 
> original tags for ATSv2 REST API queries do not return results as they expect 
> the query url to have the tags in lowercase. 
> This is additional work for the client because each tag needs to be 
> lowercased before running a query.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7957) Yarn service delete option disappears after stopping application

[Gour Saha (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382405#comment-16382405
 ] 

Gour Saha commented on YARN-7957:
-

I clearly see that we need to add the state DELETED to ServiceState. We should 
also switch to publishing ServiceState instead of FinalApplicationStatus (as 
[~rohithsharma]  mentioned) and make sure that the DELETED state is published 
to ATS on the destroy event.

Note, the DELETED state will not be used to respond to cli or REST API GET 
calls on a previously-existing-but-deleted service. It will stick to the 
current behavior of responding with a 404 with no payload (or at least this 
Jira will not address this).

The DELETED state (along with the other already existing ones in ServiceState) 
can be used to cover all the logic in the UI as to when to show the Stop/Delete 
and even a Start option in the Settings drop-down.

I will file a sub-task for the Service side of changes. [~sunilg] , you can 
file sub-task(s) to address the UI changes required after the service change is 
available.

> Yarn service delete option disappears after stopping application
> 
>
> Key: YARN-7957
> URL: https://issues.apache.org/jira/browse/YARN-7957
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn-ui-v2
>Affects Versions: 3.1.0
>Reporter: Yesha Vora
>Assignee: Sunil G
>Priority: Critical
> Attachments: YARN-7957.01.patch
>
>
> Steps:
> 1) Launch yarn service
> 2) Go to service page and click on Setting button->"Stop Service". The 
> application will be stopped.
> 3) Refresh page
> Here, setting button disappears. Thus, user can not delete service from UI 
> after stopping application
> Expected behavior:
> Setting button should be present on UI page after application is stopped. If 
> application is stopped, setting button should only have "Delete Service" 
> action available.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7494) Add muti node lookup support for better placement

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-7494:
--
Attachment: multi-node-designProposal.png

> Add muti node lookup support for better placement
> -
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch, 
> YARN-7494.v0.patch, YARN-7494.v1.patch, multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup 
> based on partition to start with.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7494) Add muti node lookup support for better placement

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382377#comment-16382377
 ] 

Sunil G commented on YARN-7494:
---

Updated v2 patch as per design proposal updated as 
"multi-node-designProposal.png".

cc [~leftnoteasy] [~cheersyang] [~Tao Yang]

> Add muti node lookup support for better placement
> -
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch, 
> YARN-7494.v0.patch, YARN-7494.v1.patch, multi-node-designProposal.png
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup 
> based on partition to start with.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7494) Add muti node lookup support for better placement

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-7494:
--
Attachment: YARN-7494.002.patch

> Add muti node lookup support for better placement
> -
>
> Key: YARN-7494
> URL: https://issues.apache.org/jira/browse/YARN-7494
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Sunil G
>Assignee: Sunil G
>Priority: Major
> Attachments: YARN-7494.001.patch, YARN-7494.002.patch, 
> YARN-7494.v0.patch, YARN-7494.v1.patch
>
>
> Instead of single node, for effectiveness we can consider a multi node lookup 
> based on partition to start with.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0

[Vrushali C (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7346:
-
Description: 
When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, I 
got the following errors:

[https://pastebin.com/Ms4jYEVB]

This issue is to fix the compilation errors.

The scope of the Jira is to add a profile to allow optional compilation for 
ATSv2 with HBase2.0. The default compilation for trunk will still be for hbase 
1.2.6. 

  was:
When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, I 
got the following errors:

https://pastebin.com/Ms4jYEVB

This issue is to fix the compilation errors.


> Add a profile to allow optional compilation for ATSv2 with HBase-2.0
> 
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> [https://pastebin.com/Ms4jYEVB]
> This issue is to fix the compilation errors.
> The scope of the Jira is to add a profile to allow optional compilation for 
> ATSv2 with HBase2.0. The default compilation for trunk will still be for 
> hbase 1.2.6. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7346) Add a profile to allow optional compilation for ATSv2 with HBase-2.0

[Vrushali C (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vrushali C updated YARN-7346:
-
Summary: Add a profile to allow optional compilation for ATSv2 with 
HBase-2.0  (was: Add a profile to compile ATSv2 with HBase-2.0)

> Add a profile to allow optional compilation for ATSv2 with HBase-2.0
> 
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> https://pastebin.com/Ms4jYEVB
> This issue is to fix the compilation errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7637) GPU volume creation command fails when work preserving is disabled at NM

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382351#comment-16382351
 ] 

Hudson commented on YARN-7637:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13751 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13751/])
YARN-7637. GPU volume creation command fails when work preserving is (sunilg: 
rev 22928c0c60de3ccbd6b27ac3a410e7566ce4136b)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMNullStateStoreService.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/resources/gpu/TestGpuResourceHandler.java


> GPU volume creation command fails when work preserving is disabled at NM
> 
>
> Key: YARN-7637
> URL: https://issues.apache.org/jira/browse/YARN-7637
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Sunil G
>Assignee: Zian Chen
>Priority: Critical
> Fix For: 3.1.0, 3.2.0
>
> Attachments: YARN-7637.001.patch
>
>
> When work preserving is disabled, NM uses {{NMNullStateStoreService}}. Hence 
> resource mappings related to GPU wont be saved at Container.
> This has to  be rechecked and store accordingly.
> cc/ [~leftnoteasy] and [~Zian Chen]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5592) Add support for dynamic resource updates with multiple resource types

[Manikandan R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382324#comment-16382324
 ] 

Manikandan R commented on YARN-5592:


[~sunilg] Thanks for your explanation.

IIUC, Do we need something similar to {{AdminService#refreshNodeResources}}? It 
uses DR configuration file as described in 
https://issues.apache.org/jira/browse/YARN-313 to refresh node resources at run 
time. 

As far as this JIRA is concerned, first we will need to add a option to 
configure new resource types at RM level, then only it is possible to configure 
those resource types at NM too.

> Add support for dynamic resource updates with multiple resource types
> -
>
> Key: YARN-5592
> URL: https://issues.apache.org/jira/browse/YARN-5592
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Manikandan R
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7637) GPU volume creation command fails when work preserving is disabled at NM

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382317#comment-16382317
 ] 

Sunil G commented on YARN-7637:
---

Thanks, i ll commit today.

> GPU volume creation command fails when work preserving is disabled at NM
> 
>
> Key: YARN-7637
> URL: https://issues.apache.org/jira/browse/YARN-7637
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager
>Affects Versions: 3.1.0
>Reporter: Sunil G
>Assignee: Zian Chen
>Priority: Critical
> Attachments: YARN-7637.001.patch
>
>
> When work preserving is disabled, NM uses {{NMNullStateStoreService}}. Hence 
> resource mappings related to GPU wont be saved at Container.
> This has to  be rechecked and store accordingly.
> cc/ [~leftnoteasy] and [~Zian Chen]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382312#comment-16382312
 ] 

Hudson commented on YARN-5028:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13750 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13750/])
YARN-5028. RMStateStore should trim down app state for completed (yufei: rev 
6ca4e3017b994a7444b4b8bf936c1d5d19603ccd)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStore.java


> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7346) Add a profile to compile ATSv2 with HBase-2.0

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382293#comment-16382293
 ] 

Rohith Sharma K S commented on YARN-7346:
-

[~haibochen] Its better to attach findbugs report xml file which is compiles 
against default profile and compiles against hbase-2 profile. And also we 
should attach javadoc compilation for HBase-2 profile.

This gives clarity for viewers that we haven't committed this patch without 
looking into findbugs or java doc errors. This is necessary because for any 
viewer who come and look at JIRA, they think that we have committed that patch 
even though findbugs and javadoc errors reported by jenkins. So, lets upload a 
final patch findbugs and javadoc compilation report for both the profiles. You 
can use below commands to run from 
_hadoop-yarn-server-timelineservice-hbase-server_ module.
 - Default compilation : *mvn clean install findbugs:findbugs javadoc:javadoc 
-DskipTests=true*
 - HBase-2 profile compilation : *mvn clean install findbugs:findbugs 
javadoc:javadoc -DskipTests=true -Dhbase.profile=2.0*

With the above reports, we will be in safer side to convince folks if someone 
question regarding jenkins report.

> Add a profile to compile ATSv2 with HBase-2.0
> -
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> https://pastebin.com/Ms4jYEVB
> This issue is to fix the compilation errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7988) Refactor FSNodeLabelStore code for attributes store support

[Bibin A Chundatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382280#comment-16382280
 ] 

Bibin A Chundatt commented on YARN-7988:


Thank you  [~cheersyang] for review
{quote}
line 38, 44, 50, 63, 65: remove abstract
{quote}
done
{quote}
It seems you map a sub class of FSNodeStoreLogOp to an enum FSNodeLogOpCode, 
that seems to be just for class initializing, is this required? Can it be 
replaced with simple factory method without reflection code?
{quote}
In my opinion addition of new LogOp definition become easy within this case. 
Any negative you find with approach?
{quote}
There seems to have chance to leak file descriptors, need to make sure close is 
wrapped in finally clause, e.g AbstractFSNodeStore: line 126, line 160, line 177
{quote}
try-with-resource is used now.
{quote}
line 205, replace the deprecated cleanup method
{quote}
done
{quote}
TestFileSystemNodeLabelsStore
{quote}
During testcase debugging i did update timeout missed to revert the 
same.Handled in latest patch.


> Refactor FSNodeLabelStore code for attributes store support
> ---
>
> Key: YARN-7988
> URL: https://issues.apache.org/jira/browse/YARN-7988
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7988-YARN-3409.002.patch, YARN-7988.001.patch
>
>
> # Abstract out file FileSystemStore operation
> # Define EditLog Operartions  and Mirror operation
> # Support compatibility with old nodelabel store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382279#comment-16382279
 ] 

Yufei Gu commented on YARN-5028:


Committed the addendum to trunk.

> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7988) Refactor FSNodeLabelStore code for attributes store support

[Bibin A Chundatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bibin A Chundatt updated YARN-7988:
---
Attachment: YARN-7988-YARN-3409.002.patch

> Refactor FSNodeLabelStore code for attributes store support
> ---
>
> Key: YARN-7988
> URL: https://issues.apache.org/jira/browse/YARN-7988
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7988-YARN-3409.002.patch, YARN-7988.001.patch
>
>
> # Abstract out file FileSystemStore operation
> # Define EditLog Operartions  and Mirror operation
> # Support compatibility with old nodelabel store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382246#comment-16382246
 ] 

Yufei Gu commented on YARN-5028:


The addendum patch looks good to me. +1. This original one only committed to 
trunk. 

> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7958) ServiceMaster should only wait for recovery of containers with id that match the current application id

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382244#comment-16382244
 ] 

Hudson commented on YARN-7958:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13749 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/13749/])
YARN-7958. ServiceMaster should only wait for recovery of containers (billie: 
rev 5ed689e33ac0becfc36d5fa424466bb1732aa59b)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/ServiceScheduler.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestServiceAM.java


> ServiceMaster should only wait for recovery of containers with id that match 
> the current application id
> ---
>
> Key: YARN-7958
> URL: https://issues.apache.org/jira/browse/YARN-7958
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Critical
> Fix For: 3.1.0
>
> Attachments: YARN-7958.001.patch, YARN-7958.002.patch, 
> YARN-7958.003.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7952) Find a way to persist the log aggregation status

[Xuan Gong (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382151#comment-16382151
 ] 

Xuan Gong commented on YARN-7952:
-

Did the manual test, and verify after RM restart/failover, we can recover the 
log aggregation status for some applications if there are in the defined 
log-aggregation-status-rolling-over-period.

> Find a way to persist the log aggregation status
> 
>
> Key: YARN-7952
> URL: https://issues.apache.org/jira/browse/YARN-7952
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Major
> Attachments: YARN-7952-poc.patch, YARN-7952.1.patch
>
>
> In MAPREDUCE-6415, we have created a CLI to har the aggregated logs, and In 
> YARN-4946: RM should write out Aggregated Log Completion file flag next to 
> logs, we have a discussion on how we can get the log aggregation status: make 
> a client call to RM or get it directly from the Distributed file system(HDFS).
> No matter which approach we would like to choose, we need to figure out a way 
> to persist the log aggregation status first. This ticket is used to track the 
> working progress for this purpose.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7952) Find a way to persist the log aggregation status

[Xuan Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong updated YARN-7952:

Attachment: YARN-7952.1.patch

> Find a way to persist the log aggregation status
> 
>
> Key: YARN-7952
> URL: https://issues.apache.org/jira/browse/YARN-7952
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Major
> Attachments: YARN-7952-poc.patch, YARN-7952.1.patch
>
>
> In MAPREDUCE-6415, we have created a CLI to har the aggregated logs, and In 
> YARN-4946: RM should write out Aggregated Log Completion file flag next to 
> logs, we have a discussion on how we can get the log aggregation status: make 
> a client call to RM or get it directly from the Distributed file system(HDFS).
> No matter which approach we would like to choose, we need to figure out a way 
> to persist the log aggregation status first. This ticket is used to track the 
> working progress for this purpose.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382127#comment-16382127
 ] 

Rohith Sharma K S commented on YARN-5028:
-

I have verified basic RM restart scenarios and looks fine if we include 
addendum patch. [~sunilg]/[~yufeigu] Could you guys please help to commit this 
addendum patch? I am not sure which are all the branches this is committed 
earlier!

> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[Gergo Repas (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382113#comment-16382113
 ] 

Gergo Repas commented on YARN-5028:
---

[~rohithsharma] Apologies for causing the break, and thanks for the addendum 
patch! The addendum patch looks good to me.

(I'm also fine with having my change reverted, and adding it back later after I 
tested it with ATSv2, even if it's for a later release).

> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7987) Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks.

[Shane Kumpf (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16382029#comment-16382029
 ] 

Shane Kumpf commented on YARN-7987:
---

[~suma.shivaprasad] Thanks for reporting this. It's really a shame that Docker 
doesn't honor the hostname here. 

IIRC, with an overlay network, the FQDN is {{.}}, 
not the domain defined in YARN DNS. Is that not the case when {{\-\-name}} is 
fully qualified? 

Have you looked into the {{\-\-network-alias}} option? I think this will 
associate an additional "alias" hostname with the container's default hostname 
(like a CNAME). If you set that to the same value as {{\-\-hostname}}, that 
might deal with the resolution issues. Can we first explore if that might work?



> Docker container name(--name) needs to be DNS friendly for DNS resolution to 
> work in user defined networks. 
> 
>
> Key: YARN-7987
> URL: https://issues.apache.org/jira/browse/YARN-7987
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Suma Shivaprasad
>Assignee: Suma Shivaprasad
>Priority: Major
>
> User defined networks like overlays support DNS resolution through Docker 
> Embedded DNS which needs the container name (–name parameter value in docker 
> run) to be a FQDN for container names to be resolved - Please refer 
> documentation 
> [https://docs.docker.com/v17.09/engine/userguide/networking/configure-dns/]
> However Yarn sets the container name to the container's id which is not DNS 
> friendly(eg: container_e26_1519402686002_0035_01_03) and is not a FQDN. 
> The proposal is to set a FQDN(eg: 
> ctr-e26-1519402686002-0035-01-03.domain-name) as the docker container's 
> name for containers to be able to communicate to each other via hostnames in 
> user defined networks like overlays, bridges etc. The domain name will be 
> picked up from the YARN DNS registry configuration 
> (hadoop.registry.dns.domain-name)
>  
>  
>  
>  
>  
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4488) CapacityScheduler: Compute per-container allocation latency and roll up to get per-application and per-queue

[Manikandan R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381997#comment-16381997
 ] 

Manikandan R commented on YARN-4488:


[~leftnoteasy] Had a chance to review this approach? Can you please check?

> CapacityScheduler: Compute per-container allocation latency and roll up to 
> get per-application and per-queue
> 
>
> Key: YARN-4488
> URL: https://issues.apache.org/jira/browse/YARN-4488
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Karthik Kambatla
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-4485.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5590) Add support for increase and decrease of container resources with resource profiles

[Manikandan R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381964#comment-16381964
 ] 

Manikandan R commented on YARN-5590:


[~sunilg] Thanks.

While trying to understand https://issues.apache.org/jira/browse/YARN-5592, got 
an impression that there is an support for resource types already available at 
the core level. So made changes in test cases only just to know whether 
increase/decrease containers size even for resource types works or not. It 
worked and felt that it is already meeting this JIRA requirements. Am I missing 
anything here?

> Add support for increase and decrease of container resources with resource 
> profiles
> ---
>
> Key: YARN-5590
> URL: https://issues.apache.org/jira/browse/YARN-5590
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Varun Vasudev
>Assignee: Manikandan R
>Priority: Major
> Attachments: YARN-5590.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381950#comment-16381950
 ] 

genericqa commented on YARN-5028:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 19s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 25s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 39s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}109m 22s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-5028 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912576/YARN-5028.007-addendum.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 4cac7f56f129 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 55d04a6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/19858/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/19858/testReport/ |
| Max. process+thread count | 885 (vs. ulimit of 1) |
| 

[jira] [Commented] (YARN-7346) Add a profile to compile ATSv2 with HBase-2.0

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381951#comment-16381951
 ] 

Rohith Sharma K S commented on YARN-7346:
-

{quote}I'd like to see everything under control before it goes into trunk 
though; that's get javadocs and findbugs to shut up
{quote}
By default yetus runs for hbase-1 profile which is default. This is causing 
findbug and java doc errors for hbase-2 modules. We can produce a findbug and 
java doc report for HBase-2 profile separately and can be attached to JIRA as a 
proof. But if someone expect that jenkins has to run for both hbase-1 and 
hbase-2 profile all the time then really I don't have solution. IIUC from other 
projects like Tez, jenkins runs for single profile always but Yetus is running 
for modular compilation which is generating findbug and java doc errors.
{quote}setting the hbase-2 profile will allow the rest of Hadoop to build with 
guava.version=21.0?
{quote}
This is something we have to discuss/test irrespective of HBase-2 profile that 
compiles against HBase-2.0-beta1 which is doing in this JIRA. I am not sure why 
HADOOP-15272 depends on this JIRA! ATSv2 is already uses Hbase-1.2.6 as back 
end and uses guava 11. 

> Add a profile to compile ATSv2 with HBase-2.0
> -
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> https://pastebin.com/Ms4jYEVB
> This issue is to fix the compilation errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7972) Support inter-app placement constraints for allocation tags by application ID

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381913#comment-16381913
 ] 

genericqa commented on YARN-7972:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
44s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 13m 
51s{color} | {color:red} hadoop-yarn in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m  
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m  4s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
4s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in 
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
46s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  1m 
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m  
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m  
8s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 1 new + 24 unchanged - 1 fixed = 25 total (was 25) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 47s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
43s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 21s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}153m  3s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector 
|
|   | hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7972 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912572/YARN-7972.004.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux 91aeaa870e28 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | 

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381912#comment-16381912
 ] 

genericqa commented on YARN-5028:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 24s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
11m 16s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 72m 47s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
26s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}122m 22s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.resourcemanager.TestOpportunisticContainerAllocatorAMService 
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-5028 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912576/YARN-5028.007-addendum.patch
 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  shadedclient  findbugs  checkstyle  |
| uname | Linux b438bea5a14b 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 55d04a6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/19857/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/19857/testReport/ |
| Max. process+thread count | 799 (vs. ulimit of 1) |
| 

[jira] [Commented] (YARN-7346) Add a profile to compile ATSv2 with HBase-2.0

[Steve Loughran (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381885#comment-16381885
 ] 

Steve Loughran commented on YARN-7346:
--

I am really excited about this if it lets us move off Guava 11 —I'd created a 
new "try to upgrade guava again" JIRA only yesterday: HADOOP-15272. 

* I'd like to see everything under control before it goes into trunk though; 
that's get javadocs and findbugs to shut up
* and something in /BUILDING

Presumably once this is in, setting the hbase-2 profile will allow the rest of 
Hadoop to build with guava.version=21.0? Has anyone tested this?


> Add a profile to compile ATSv2 with HBase-2.0
> -
>
> Key: YARN-7346
> URL: https://issues.apache.org/jira/browse/YARN-7346
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Haibo Chen
>Priority: Major
> Attachments: YARN-7346.00.patch, YARN-7346.01.patch, 
> YARN-7346.02.patch, YARN-7346.03-incremental.patch, YARN-7346.03.patch, 
> YARN-7346.04-incremental.patch, YARN-7346.04.patch, YARN-7346.05.patch, 
> YARN-7346.06.patch, YARN-7346.07.patch, YARN-7346.08.patch, 
> YARN-7346.prelim1.patch, YARN-7346.prelim2.patch, YARN-7581.prelim.patch
>
>
> When compiling hadoop-yarn-server-timelineservice-hbase against 2.0.0-alpha3, 
> I got the following errors:
> https://pastebin.com/Ms4jYEVB
> This issue is to fix the compilation errors.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7871) Node attributes reporting from NM to RM

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381860#comment-16381860
 ] 

genericqa commented on YARN-7871:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
29s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 5 new or modified test 
files. {color} |
|| || || || {color:brown} YARN-3409 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
24s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 
38s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 
37s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
59s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
33s{color} | {color:green} YARN-3409 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
13m  4s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
9s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in 
YARN-3409 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m  
9s{color} | {color:green} YARN-3409 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  2m 
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  8m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  8m 
14s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
1m  0s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch 
generated 6 new + 344 unchanged - 3 fixed = 350 total (was 347) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}  
9m 58s{color} | {color:green} patch has no errors when building and testing our 
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  5m  
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
35s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  3m 
27s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 21m 30s{color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 75m 52s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
28s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 14s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.TestContainerManager |
|   | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels |
\\
\\
|| Subsystem || Report/Notes ||
| 

[jira] [Commented] (YARN-7988) Refactor FSNodeLabelStore code for attributes store support

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381853#comment-16381853
 ] 

Weiwei Yang commented on YARN-7988:
---

Hi [~bibinchundatt]

Thanks for the work on the refactoring, it helps a lot for code reuse. I can't 
say I am fully qualified to review these code, I'll try to give some general 
comments,

*NodeLabelsStore*
line 38, 44, 50, 63, 65: remove {{abstract}}

*FSNodeStoreLogOp*
It seems you map a sub class of {{FSNodeStoreLogOp}} to an enum 
{{FSNodeLogOpCode}}, that seems to be just for class initializing, is this 
required? Can it be replaced with simple factory method without reflection code?

*AbstractFSNodeStore*
# There seems to have chance to leak file descriptors, need to make sure close 
is wrapped in finally clause, e.g AbstractFSNodeStore: line 126, line 160, line 
177
# line 205, replace the deprecated cleanup method

*TestFileSystemNodeLabelsStore*
line 170: why the timeout is removed?

Hope it helps.
Thanks.

> Refactor FSNodeLabelStore code for attributes store support
> ---
>
> Key: YARN-7988
> URL: https://issues.apache.org/jira/browse/YARN-7988
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Major
> Attachments: YARN-7988.001.patch
>
>
> # Abstract out file FileSystemStore operation
> # Define EditLog Operartions  and Mirror operation
> # Support compatibility with old nodelabel store



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381801#comment-16381801
 ] 

Rohith Sharma K S commented on YARN-5028:
-

Updated patch adding app name and app tags included.

> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5028) RMStateStore should trim down app state for completed applications

[Rohith Sharma K S (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rohith Sharma K S updated YARN-5028:

Attachment: YARN-5028.007-addendum.patch

> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007-addendum.patch, YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5028) RMStateStore should trim down app state for completed applications

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381784#comment-16381784
 ] 

Rohith Sharma K S commented on YARN-5028:
-

I found that couple of more issues where app report also throw NPE for 
following URL i.e *http://localhost:8088/ws/v1/cluster/apps*. It seems we 
should add application name and application tags as well. 

> RMStateStore should trim down app state for completed applications
> --
>
> Key: YARN-5028
> URL: https://issues.apache.org/jira/browse/YARN-5028
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Gergo Repas
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: YARN-5028.000.patch, YARN-5028.001.patch, 
> YARN-5028.002.patch, YARN-5028.003.patch, YARN-5028.004.patch, 
> YARN-5028.005.patch, YARN-5028.006.patch, YARN-5028.007-addendum.patch, 
> YARN-5028.007.patch
>
>
> RMStateStore stores enough information to recover applications in case of a 
> restart. The store also retains this information for completed applications 
> to serve their status to REST, WebUI, Java and CLI clients. We don't need all 
> the information we store today to serve application status; for instance, we 
> don't need the {{ApplicationSubmissionContext}}. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7972) Support inter-app placement constraints for allocation tags by application ID

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-7972:
--
Attachment: YARN-7972.004.patch

> Support inter-app placement constraints for allocation tags by application ID
> -
>
> Key: YARN-7972
> URL: https://issues.apache.org/jira/browse/YARN-7972
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-7972.001.patch, YARN-7972.002.patch, 
> YARN-7972.003.patch, YARN-7972.004.patch
>
>
> Per discussion in [this 
> comment|https://issues.apache.org/jira/browse/YARN-6599focusedCommentId=16319662=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16319662]
>  in  YARN-6599, we need to support inter-app PC for allocation tags.
> This will help to do better placement when dealing with potential competing 
> resource applications, e.g don't place two tensorflow workers from two 
> different applications on one same node.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7989) Yarn ui2 configuration tables have the same search and pagination instances

[genericqa (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381736#comment-16381736
 ] 

genericqa commented on YARN-7989:
-

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
26m 29s{color} | {color:green} branch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 
10m 40s{color} | {color:green} patch has no errors when building and testing 
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 34s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7989 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12912558/YARN-7989.001.patch |
| Optional Tests |  asflicense  shadedclient  |
| uname | Linux 53f6da7a8b61 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31 
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 10663b7 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 410 (vs. ulimit of 1) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/19854/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> Yarn ui2 configuration tables have the same search and pagination instances
> ---
>
> Key: YARN-7989
> URL: https://issues.apache.org/jira/browse/YARN-7989
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn, yarn-ui-v2
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: Screen Shot 2018-03-01 at 2.47.30 PM.png, 
> YARN-7989.001.patch
>
>
> Entering the search term in any of the three configuration tables (core, 
> yarn, mapred) will result in filtering across all tables. Pagination also 
> behaves the same way as clicking on one table changes the page in all the 
> tables. This results in weird behavior since each table can have different 
> number of pages present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7989) Yarn ui2 configuration tables have the same search and pagination instances

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381729#comment-16381729
 ] 

Sunil G commented on YARN-7989:
---

Thanks. Looks good to me.

+1 pending jenkins.

> Yarn ui2 configuration tables have the same search and pagination instances
> ---
>
> Key: YARN-7989
> URL: https://issues.apache.org/jira/browse/YARN-7989
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn, yarn-ui-v2
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: Screen Shot 2018-03-01 at 2.47.30 PM.png, 
> YARN-7989.001.patch
>
>
> Entering the search term in any of the three configuration tables (core, 
> yarn, mapred) will result in filtering across all tables. Pagination also 
> behaves the same way as clicking on one table changes the page in all the 
> tables. This results in weird behavior since each table can have different 
> number of pages present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-7989) Yarn ui2 configuration tables have the same search and pagination instances

[Tarun Parimi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16381722#comment-16381722
 ] 

Tarun Parimi commented on YARN-7989:


[~sunilg] Checked that the search is local to each table now. Attached a 
screen. Thanks for adding as contributor.

> Yarn ui2 configuration tables have the same search and pagination instances
> ---
>
> Key: YARN-7989
> URL: https://issues.apache.org/jira/browse/YARN-7989
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn, yarn-ui-v2
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: Screen Shot 2018-03-01 at 2.47.30 PM.png, 
> YARN-7989.001.patch
>
>
> Entering the search term in any of the three configuration tables (core, 
> yarn, mapred) will result in filtering across all tables. Pagination also 
> behaves the same way as clicking on one table changes the page in all the 
> tables. This results in weird behavior since each table can have different 
> number of pages present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7989) Yarn ui2 configuration tables have the same search and pagination instances

[Tarun Parimi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tarun Parimi updated YARN-7989:
---
Attachment: Screen Shot 2018-03-01 at 2.47.30 PM.png

> Yarn ui2 configuration tables have the same search and pagination instances
> ---
>
> Key: YARN-7989
> URL: https://issues.apache.org/jira/browse/YARN-7989
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn, yarn-ui-v2
>Reporter: Tarun Parimi
>Assignee: Tarun Parimi
>Priority: Major
> Attachments: Screen Shot 2018-03-01 at 2.47.30 PM.png, 
> YARN-7989.001.patch
>
>
> Entering the search term in any of the three configuration tables (core, 
> yarn, mapred) will result in filtering across all tables. Pagination also 
> behaves the same way as clicking on one table changes the page in all the 
> tables. This results in weird behavior since each table can have different 
> number of pages present.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7990) Node attribute prefix definition and validation

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-7990:
--
Description: 
Summary:
# Centralized: rm.yarn.io
# Distributed: nm.yarn.io
# System: *.yarn.io (yarn.io suffix is preserved for other yarn system set node 
attributes)
# User-Defined: other form of prefixes except reserved ones

See detail in design doc  [^Node Attribute Prefix Definition v1.pdf] 

  was:
Summary:
# Centralized: rm.yarn.io
# Distributed: nm.yarn.io
# System: *.yarn.io (yarn.io suffix is preserved for other yarn system set node 
attributes)
# User-Defined: other form of prefixes except reserved ones

See detail in design doc


> Node attribute prefix definition and validation 
> 
>
> Key: YARN-7990
> URL: https://issues.apache.org/jira/browse/YARN-7990
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Node Attribute Prefix Definition v1.pdf
>
>
> Summary:
> # Centralized: rm.yarn.io
> # Distributed: nm.yarn.io
> # System: *.yarn.io (yarn.io suffix is preserved for other yarn system set 
> node attributes)
> # User-Defined: other form of prefixes except reserved ones
> See detail in design doc  [^Node Attribute Prefix Definition v1.pdf] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-7990) Node attribute prefix definition and validation

[Weiwei Yang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Weiwei Yang updated YARN-7990:
--
Attachment: Node Attribute Prefix Definition v1.pdf

> Node attribute prefix definition and validation 
> 
>
> Key: YARN-7990
> URL: https://issues.apache.org/jira/browse/YARN-7990
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: Node Attribute Prefix Definition v1.pdf
>
>
> Summary:
> # Centralized: rm.yarn.io
> # Distributed: nm.yarn.io
> # System: *.yarn.io (yarn.io suffix is preserved for other yarn system set 
> node attributes)
> # User-Defined: other form of prefixes except reserved ones
> See detail in design doc



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org