[jira] [Commented] (YARN-4957) Add getNewReservation in ApplicationClientProtocol

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290509#comment-15290509
 ] 

Hadoop QA commented on YARN-4957:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 9 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 29s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 10s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
22s {color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped branch modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 8s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 18s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 38s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 38s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 23s 
{color} | {color:red} root: patch generated 2 new + 468 unchanged - 6 fixed = 
470 total (was 474) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 47s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m 
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s 
{color} | {color:blue} Skipped patch modules with no Java source: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
50s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 7s 
{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 3 new + 1255 unchanged - 0 fixed = 1258 total (was 1255) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 3m 7s 
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client 
generated 2 new + 183 unchanged - 0 fixed = 185 total (was 183) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 38s 
{color} | {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s 
{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 13s 
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 17s {color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 9s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 20s {color} 
| {color:red} 

[jira] [Commented] (YARN-5114) TestRMWebServicesApps#testInvalidAppAttempts failure

[Bibin A Chundatt (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290476#comment-15290476
 ] 

Bibin A Chundatt commented on YARN-5114:


Looks like YARN-4417 is not available in 2.8 

> TestRMWebServicesApps#testInvalidAppAttempts failure
> 
>
> Key: YARN-5114
> URL: https://issues.apache.org/jira/browse/YARN-5114
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Priority: Minor
>
> Testcase failure in 2.8
> {noformat}
> java.lang.AssertionError: expected: but was:
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:144)
>   at 
> org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps.testInvalidAppAttempts(TestRMWebServicesApps.java:1519)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
>   at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
>   at java.lang.reflect.Method.invoke(Unknown Source)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
>   at 
> org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
>   at 
> org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
>   at 
> org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5114) TestRMWebServicesApps#testInvalidAppAttempts failure

[Bibin A Chundatt (JIRA)]

Bibin A Chundatt created YARN-5114:
--

 Summary: TestRMWebServicesApps#testInvalidAppAttempts failure
 Key: YARN-5114
 URL: https://issues.apache.org/jira/browse/YARN-5114
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Bibin A Chundatt
Priority: Minor


Testcase failure in 2.8

{noformat}
java.lang.AssertionError: expected: but was:
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps.testInvalidAppAttempts(TestRMWebServicesApps.java:1519)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at 
org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:86)
at 
org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at 
org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)


{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3840) Resource Manager web ui issue when sorting application by id (with application having id > 9999)

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290444#comment-15290444
 ] 

Rohith Sharma K S commented on YARN-3840:
-

At first glance, test failures looks not related!! [~varun_saxena] / 
[~bibinchundatt] would you also confirm QA result?

> Resource Manager web ui issue when sorting application by id (with 
> application having id > )
> 
>
> Key: YARN-3840
> URL: https://issues.apache.org/jira/browse/YARN-3840
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 2.7.0
>Reporter: LINTE
>Assignee: Varun Saxena
> Fix For: 2.8.0, 2.7.3
>
> Attachments: RMApps.png, RMApps_Sorted.png, YARN-3840-1.patch, 
> YARN-3840-2.patch, YARN-3840-3.patch, YARN-3840-4.patch, YARN-3840-5.patch, 
> YARN-3840-6.patch, YARN-3840-branch-2.8.01.patch, 
> YARN-3840.reopened.001.patch, yarn-3840-7.patch
>
>
> On the WEBUI, the global main view page : 
> http://resourcemanager:8088/cluster/apps doesn't display applications over 
> .
> With command line it works (# yarn application -list).
> Regards,
> Alexandre



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5016) Add support for a minimum retry interval

[Jun Gong (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290433#comment-15290433
 ] 

Jun Gong commented on YARN-5016:


Thanks [~vvasudev] for the review. Attach a new patch to address above problems.

> Add support for a minimum retry interval
> 
>
> Key: YARN-5016
> URL: https://issues.apache.org/jira/browse/YARN-5016
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Vasudev
>Assignee: Jun Gong
> Attachments: YARN-5016.01.patch, YARN-5016.02.patch
>
>
> The NM container re-launch feature should add support to specify a minimum 
> restart interval so that the minimum time between restarts can be set by 
> admins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5016) Add support for a minimum retry interval

[Jun Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Gong updated YARN-5016:
---
Attachment: YARN-5016.02.patch

> Add support for a minimum retry interval
> 
>
> Key: YARN-5016
> URL: https://issues.apache.org/jira/browse/YARN-5016
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Varun Vasudev
>Assignee: Jun Gong
> Attachments: YARN-5016.01.patch, YARN-5016.02.patch
>
>
> The NM container re-launch feature should add support to specify a minimum 
> restart interval so that the minimum time between restarts can be set by 
> admins.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5047) Refactor nodeUpdate across schedulers

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290416#comment-15290416
 ] 

Karthik Kambatla commented on YARN-5047:


Excited to see progress here. Thanks for working on this, [~rchiang]. 

Comments: 
# I wonder what the @Lock annotation on getNode does? Looking at the 
{{@interface Lock}}
# {{AbstractYarnScheduler#nodeUpdate}}:
## While we are formatting, we should probably move the definition of 
{{releaseResources}} closer to its use at {{realeasedContainers}}, and may be 
also renamed to "released" resources? 
## Java 7 and beyond, we don't can emit the type when calling constructor. 
Definitions of newlyLaunchedContainers and completedContainers
## Is the TODO still required? If yes, mind filing a follow-up JIRA and 
annotating that TODO
## Shouldn't nodeUpdateInternal be called right at the end? The log message 
says we are looking at node for allocations. And, the scheduling code might 
want to use the latest info on aggregated containers' and node utilization. 
YARN-1011 does this. 
## Should we mark this final so it can't be overriden? I see that FairScheduler 
overrides this to capture duration of node update. I don't remember the details 
clearly; are we primarily interested in the duration to schedule or processing 
the entire node heartbeat? 
# There is interest to track {{SchedulerHealth}} for FairScheduler as well. Can 
we file a follow-up JIRA here to move the corresponding update code also to 
AbstractYarnScheduler so we can reuse it in FairScheduler? 
# Why do we need the change to FiCaSchedulerApp? Looks like this comes from 
{{FifoScheduler#nodeUpdateInternal}}. Shouldn't the type of getNode there be 
FiCaSchedulerNode? 

> Refactor nodeUpdate across schedulers
> -
>
> Key: YARN-5047
> URL: https://issues.apache.org/jira/browse/YARN-5047
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacityscheduler, fairscheduler, scheduler
>Affects Versions: 3.0.0-alpha1
>Reporter: Ray Chiang
>Assignee: Ray Chiang
> Attachments: YARN-5047.001.patch, YARN-5047.002.patch
>
>
> FairScheduler#nodeUpdate() and CapacityScheduler#nodeUpdate() have a lot of 
> commonality in their code.  See about refactoring the common parts into 
> AbstractYARNScheduler.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290352#comment-15290352
 ] 

Daniel Templeton commented on YARN-4464:


Whoops.  The patch got merged with another.  I'll tease them apart and repost.

I'm not sure it's a good idea to change defaults in branch-2.  We don't want to 
create incompatibilities during upgrades.

> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch, YARN-4664.002.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5110) Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290312#comment-15290312
 ] 

Hadoop QA commented on YARN-5110:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 4 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 26s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
6s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 56s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 59s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
23s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m 
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 19s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 4m 31s {color} | 
{color:red} hadoop-yarn-project_hadoop-yarn generated 1 new + 2 unchanged - 1 
fixed = 3 total (was 3) {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 19s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 19s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 42s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 17 new + 
440 unchanged - 34 fixed = 457 total (was 474) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s 
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git 
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 
25s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 2m 41s 
{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager
 generated 3 new + 589 unchanged - 3 fixed = 592 total (was 592) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 24s {color} 
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s 
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 40s {color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 52s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 31s {color} 
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
21s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 84m 28s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 

[jira] [Assigned] (YARN-5111) YARN container system metrics are not aggregated to application

[Naganarasimha G R (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Naganarasimha G R reassigned YARN-5111:
---

Assignee: Naganarasimha G R

> YARN container system metrics are not aggregated to application
> ---
>
> Key: YARN-5111
> URL: https://issues.apache.org/jira/browse/YARN-5111
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Naganarasimha G R
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
>
> It appears that the container system metrics (CPU and memory) are not being 
> aggregated onto the application.
> I definitely see container system metrics when I query for YARN_CONTAINER. 
> However, there is no corresponding metrics on the parent application.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290289#comment-15290289
 ] 

Naganarasimha G R commented on YARN-4464:
-

Hi [~templedf],
Seems like latest patch uploaded by you is not for this jira,
Given that we have timelineserver to store the finished app information its not 
required to store 1 apps, IMO it would be ideal to have this in for 
branch-2 also but may be with a better lower value like 500 or so( any val < 
1000), so that its does completely break the compatability. Thoughts ?

> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch, YARN-4664.002.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4987) Read cache concurrency issue between read and evict in EntityGroupFS timeline store

[Li Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Lu updated YARN-4987:

Attachment: YARN-4987-trunk.003.patch

003 patch addresses checkstyle issues. 

> Read cache concurrency issue between read and evict in EntityGroupFS timeline 
> store 
> 
>
> Key: YARN-4987
> URL: https://issues.apache.org/jira/browse/YARN-4987
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Li Lu
>Assignee: Li Lu
>Priority: Critical
> Attachments: YARN-4987-trunk.001.patch, YARN-4987-trunk.002.patch, 
> YARN-4987-trunk.003.patch
>
>
> To handle concurrency issues, key value based timeline storage may return 
> null on reads that are concurrent to service stop. This is actually caused by 
> a concurrency issue between cache reads and evicts. Specifically, if the 
> storage is being read when it gets evicted, the storage may turn into null. 
> EntityGroupFS timeline store needs to handle this case gracefully. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5110) Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290275#comment-15290275
 ] 

Hudson commented on YARN-5110:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #9819 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9819/])
YARN-5110. Fix OpportunisticContainerAllocator to insert complete (arun suresh: 
rev 1597630681c784a3d59f5605b87e96197b8139d7)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/queuing/QueuingContainerManagerImpl.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/distributed/NodeQueueLoadMonitor.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/scheduler/TestLocalScheduler.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/scheduler/OpportunisticContainerAllocator.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/scheduler/LocalScheduler.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/records/QueuedContainersStatus.java


> Fix OpportunisticContainerAllocator to insert complete HostAddress in issued 
> ContainerTokenIds
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Couple of Logs can be moved to debug rather than info
> * Minor fix in MiniYarnCluster to start a QueuingContainerManager if queueing 
> is enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4866) FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290261#comment-15290261
 ] 

Hadoop QA commented on YARN-4866:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 12s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 45s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 23s 
{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 patch generated 1 new + 213 unchanged - 1 fixed = 214 total (was 214) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 23s 
{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 26s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
19s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 38s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | 
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 |
|  |  Null pointer dereference of queue in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource,
 FSQueue)  Dereferenced at FSLeafQueue.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource,
 FSQueue)  Dereferenced at FSLeafQueue.java:[line 523] |
|  |  Load of known null value in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource,
 FSQueue)  At FSLeafQueue.java:in 
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSLeafQueue.isVCoreOverMaxResource(Resource,
 FSQueue)  At FSLeafQueue.java:[line 522] |
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
|   | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart |
|   | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804785/YARN-4866.005.patch |
| JIRA Issue | YARN-4866 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  

[jira] [Updated] (YARN-5110) Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Parent Issue: YARN-2877  (was: YARN-4742)

> Fix OpportunisticContainerAllocator to insert complete HostAddress in issued 
> ContainerTokenIds
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Fix For: 3.0.0-alpha1
>
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Couple of Logs can be moved to debug rather than info
> * Minor fix in MiniYarnCluster to start a QueuingContainerManager if queueing 
> is enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4878) Expose scheduling policy and max running apps over JMX for Yarn queues

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290257#comment-15290257
 ] 

Hadoop QA commented on YARN-4878:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
13s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
51s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 18s 
{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 patch generated 4 new + 53 unchanged - 0 fixed = 57 total (was 53) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 31m 2s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 44m 21s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
|   | hadoop.yarn.server.resourcemanager.TestContainerResourceUsage |
|   | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
|   | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804817/YARN-4878.003.patch |
| JIRA Issue | YARN-4878 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a97e72ccbd0d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 010e6ac |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/11537/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| unit test logs |  

[jira] [Updated] (YARN-5110) Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Description: 
* Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
address, not just the hostname.
* Couple of Logs can be moved to debug rather than info
* Minor fix in MiniYarnCluster to start a QueuingContainerManager if queueing 
is enabled.

  was:
* Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
address, not just the hostname.
* Couple of Logs can be moved to debug rather than info
* Minor refactoring w.r.t method attribute names


> Fix OpportunisticContainerAllocator to insert complete HostAddress in issued 
> ContainerTokenIds
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Couple of Logs can be moved to debug rather than info
> * Minor fix in MiniYarnCluster to start a QueuingContainerManager if queueing 
> is enabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5110) Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Summary: Fix OpportunisticContainerAllocator to insert complete HostAddress 
in issued ContainerTokenIds  (was: Minor fixes for Distributed Scheduling)

> Fix OpportunisticContainerAllocator to insert complete HostAddress in issued 
> ContainerTokenIds
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Couple of Logs can be moved to debug rather than info
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4308) ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290249#comment-15290249
 ] 

Hadoop QA commented on YARN-4308:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 35s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 
46s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 46s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 6s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 6s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 33s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn: patch generated 1 new + 
82 unchanged - 1 fixed = 83 total (was 83) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 11s 
{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 24s {color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 24s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804727/0003-YARN-4308.patch |
| JIRA Issue | YARN-4308 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux a2a27e06128d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 010e6ac |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/11534/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/11534/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
| unit test logs |  

[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290250#comment-15290250
 ] 

Hadoop QA commented on YARN-4464:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s {color} 
| {color:red} YARN-4464 does not apply to trunk. Rebase required? Wrong Branch? 
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804833/YARN-4664.002.patch |
| JIRA Issue | YARN-4464 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/11543/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch, YARN-4664.002.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5110) Minor fixes for Distributed Scheduling

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Description: 
* Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
address, not just the hostname.
* Couple of Logs can be moved to debug rather than info
* Minor refactoring w.r.t method attribute names

  was:
* Couple of Logs can be moved to debug rather than info
* Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
address, not just the hostname.
* Minor refactoring w.r.t method attribute names


> Minor fixes for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Couple of Logs can be moved to debug rather than info
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4987) Read cache concurrency issue between read and evict in EntityGroupFS timeline store

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290247#comment-15290247
 ] 

Hadoop QA commented on YARN-4987:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 10s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
24s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 9s 
{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage:
 patch generated 2 new + 7 unchanged - 3 fixed = 9 total (was 10) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 8s 
{color} | {color:green} hadoop-yarn-server-timeline-pluginstorage in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
15s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 35s {color} 
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804825/YARN-4987-trunk.002.patch
 |
| JIRA Issue | YARN-4987 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 14ee975c0712 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 010e6ac |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/11542/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timeline-pluginstorage.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/11542/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage
 |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/11542/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Read cache concurrency issue between read and evict in EntityGroupFS timeline 
> store 
> 

[jira] [Updated] (YARN-5113) Refactoring and other clean-up for YARN-2877

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5113:
-
Description: This JIRA focuses on the refactoring of classes related to 
YARN-2877.

> Refactoring and other clean-up for YARN-2877
> 
>
> Key: YARN-5113
> URL: https://issues.apache.org/jira/browse/YARN-5113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5113.001.patch
>
>
> This JIRA focuses on the refactoring of classes related to YARN-2877.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5113) Refactoring and other clean-up for YARN-2877

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5113:
-
Attachment: YARN-5113.001.patch

Adding first version of the patch.

> Refactoring and other clean-up for YARN-2877
> 
>
> Key: YARN-5113
> URL: https://issues.apache.org/jira/browse/YARN-5113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5113.001.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5110) Minor fixes for Distributed Scheduling

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290244#comment-15290244
 ] 

Arun Suresh commented on YARN-5110:
---

Committing this shortly... Thanks again [~kkaranasos]

> Minor fixes for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5110) Minor fixes for Distributed Scheduling

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290242#comment-15290242
 ] 

Arun Suresh commented on YARN-5110:
---

[~kkaranasos], lets stick with the 002 version of the patch for this JIRA. I 
have raised YARN-5113 to address larger clean-up and refactoring for YARN-2877 


> Minor fixes for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5113) Refactoring and other clean-up for YARN-2877

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5113:
--
Assignee: Konstantinos Karanasos  (was: Arun Suresh)

> Refactoring and other clean-up for YARN-2877
> 
>
> Key: YARN-5113
> URL: https://issues.apache.org/jira/browse/YARN-5113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5113) Refactoring and other clean-up for YARN-2877

[Arun Suresh (JIRA)]

Arun Suresh created YARN-5113:
-

 Summary: Refactoring and other clean-up for YARN-2877
 Key: YARN-5113
 URL: https://issues.apache.org/jira/browse/YARN-5113
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Arun Suresh






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-5113) Refactoring and other clean-up for YARN-2877

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh reassigned YARN-5113:
-

Assignee: Arun Suresh

> Refactoring and other clean-up for YARN-2877
> 
>
> Key: YARN-5113
> URL: https://issues.apache.org/jira/browse/YARN-5113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Attachment: (was: YARN-5110.003.patch)

> Minor fixes and refactoring for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5110) Minor fixes for Distributed Scheduling

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Summary: Minor fixes for Distributed Scheduling  (was: Minor fixes and 
refactoring for Distributed Scheduling)

> Minor fixes for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5112) Excessive log warnings on NM recovery

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290232#comment-15290232
 ] 

Hadoop QA commented on YARN-5112:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
39s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s 
{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
 patch generated 6 new + 194 unchanged - 5 fixed = 200 total (was 199) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 42s {color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
17s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 48s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804834/YARN-5112.patch |
| JIRA Issue | YARN-5112 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux d9c2eba187fc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 010e6ac |
| Default Java | 1.8.0_91 |
| findbugs | v3.0.0 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
| unit test logs |  
https://builds.apache.org/job/PreCommit-YARN-Build/11536/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/11536/testReport/ |
| modules | C: 

[jira] [Updated] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5110:
-
Attachment: YARN-5110.003.patch

Attaching new version with more refactoring and minor fixes.

> Minor fixes and refactoring for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch, 
> YARN-5110.003.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290210#comment-15290210
 ] 

Hadoop QA commented on YARN-5110:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
59s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
49s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
31s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 15s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 39s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
45s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 26s 
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 50s {color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 37m 12s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 4s {color} | 
{color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 83m 32s {color} 
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics |
|   | hadoop.yarn.server.resourcemanager.TestRMAdminService |
|   | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
|   | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
|   | hadoop.yarn.server.TestContainerManagerSecurity |
|   | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804823/YARN-5110.002.patch |
| JIRA Issue | YARN-5110 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 2e88c8afb7c2 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed 
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |

[jira] [Commented] (YARN-4878) Expose scheduling policy and max running apps over JMX for Yarn queues

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290208#comment-15290208
 ] 

Yufei Gu commented on YARN-4878:


My visual diff has the same problem. I tried some ways to get ride of it, but 
failed. Thanks [~rchiang].

> Expose scheduling policy and max running apps over JMX for Yarn queues
> --
>
> Key: YARN-4878
> URL: https://issues.apache.org/jira/browse/YARN-4878
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-4878.001.patch, YARN-4878.002.patch, 
> YARN-4878.003.patch
>
>
> There are two things that are not currently visible over JMX: the current 
> scheduling policy for a queue, and the number of max running apps. It would 
> be great if these could be exposed over JMX as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4878) Expose scheduling policy and max running apps over JMX for Yarn queues

[Ray Chiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290199#comment-15290199
 ] 

Ray Chiang commented on YARN-4878:
--

Oh weird.  My visual diff made it look like "public synchronized" was added by 
you.  Looks fine to leave alone.

> Expose scheduling policy and max running apps over JMX for Yarn queues
> --
>
> Key: YARN-4878
> URL: https://issues.apache.org/jira/browse/YARN-4878
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-4878.001.patch, YARN-4878.002.patch, 
> YARN-4878.003.patch
>
>
> There are two things that are not currently visible over JMX: the current 
> scheduling policy for a queue, and the number of max running apps. It would 
> be great if these could be exposed over JMX as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4957) Add getNewReservation in ApplicationClientProtocol

[Carlo Curino (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290194#comment-15290194
 ] 

Carlo Curino commented on YARN-4957:


[~seanpo03] thanks for addressing my comments right away. I agree on keeping 
things aligned with application API, and I am happy with the rest of your 
answers (including the justification of test failures).  
Please open a JIRA for the refactoring of test cases (agreed is beyond the 
scope of this patch, but we should keep track of this). 

I will wait for jenkins to do its thing to double check checkstyle and 
javadocs, and then proceed to commit.

> Add getNewReservation in ApplicationClientProtocol
> --
>
> Key: YARN-4957
> URL: https://issues.apache.org/jira/browse/YARN-4957
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: applications, client, resourcemanager
>Affects Versions: 2.8.0
>Reporter: Subru Krishnan
>Assignee: Sean Po
>  Labels: api-breaking
> Attachments: YARN-4957.v0.patch, YARN-4957.v1.patch, 
> YARN-4957.v2.patch, YARN-4957.v3.patch, YARN-4957.v4.patch, 
> YARN-4957.v5.1.patch, YARN-4957.v5.2.patch, YARN-4957.v5.patch, 
> YARN-4957.v7.patch
>
>
> Currently submitReservation returns a ReservationId if sucessful. This JIRA 
> propose adding a getNewReservation in ApplicationClientProtocol for the 
> following reasons:
>   * Prevent zombie reservations in the face of client and/or network failures 
> post submitReservation
>   * Align reservation submission with application submission



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4957) Add getNewReservation in ApplicationClientProtocol

[Sean Po (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290175#comment-15290175
 ] 

Sean Po commented on YARN-4957:
---

The following are all the test failures with this patch. Below a list of test 
failures, are links to other JIRAs documenting these known test failures.

--
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics
https://issues.apache.org/jira/browse/YARN-5107
--
hadoop.yarn.server.resourcemanager.TestAMAuthorization
hadoop.yarn.server.resourcemanager.TestClientRMTokens
https://issues.apache.org/jira/browse/YARN-5091
--
hadoop.yarn.client.TestGetGroups
https://issues.apache.org/jira/browse/YARN-4351
--
hadoop.mapred.TestMiniMRChildTask
https://issues.apache.org/jira/browse/MAPREDUCE-6702?jql=text%20~%20%22TestMiniMRChildTask%22
--
hadoop.yarn.client.api.impl.TestAMRMProxy - Test passes locally
hadoop.yarn.client.api.impl.TestYarnClient - Test passes locally
org.apache.hadoop.yarn.client.cli.TestYarnCLI
org.apache.hadoop.yarn.client.api.impl.TestAMRMClient
org.apache.hadoop.yarn.client.api.impl.TestNMClient
https://issues.apache.org/jira/browse/YARN-4982?jql=text%20~%20%22TestYarnCLI%22

> Add getNewReservation in ApplicationClientProtocol
> --
>
> Key: YARN-4957
> URL: https://issues.apache.org/jira/browse/YARN-4957
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: applications, client, resourcemanager
>Affects Versions: 2.8.0
>Reporter: Subru Krishnan
>Assignee: Sean Po
>  Labels: api-breaking
> Attachments: YARN-4957.v0.patch, YARN-4957.v1.patch, 
> YARN-4957.v2.patch, YARN-4957.v3.patch, YARN-4957.v4.patch, 
> YARN-4957.v5.1.patch, YARN-4957.v5.2.patch, YARN-4957.v5.patch, 
> YARN-4957.v7.patch
>
>
> Currently submitReservation returns a ReservationId if sucessful. This JIRA 
> propose adding a getNewReservation in ApplicationClientProtocol for the 
> following reasons:
>   * Prevent zombie reservations in the face of client and/or network failures 
> post submitReservation
>   * Align reservation submission with application submission



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290166#comment-15290166
 ] 

Sangjin Lee commented on YARN-5109:
---

Here is a proposal. Instead of blindly splitting along the separator 
boundaries, we need to be able to tell the Separator to tokenize using the size 
also. The following is a quick prototype I put together to show the concept. 
This is an overloaded {{split()}} method:

{code:title=TimelineStorageUtils.java|borderStyle=solid}
  public static final int NO_LIMIT = 0;

  public static List splitRanges(byte[] source, byte[] separator,
  int[] sizes) {
List segments = new ArrayList();
if (source == null || separator == null || sizes == null) {
  return segments;
}
int start = 0;
int i = 0;
int k = 0;
itersource: while (i < source.length && segments.size() < sizes.length) {
  int currentTokenSize = sizes[k];
  if (currentTokenSize > NO_LIMIT) {
// we explicitly grab a fixed number of bytes
if (start + currentTokenSize > source.length) {
  // it's seeking beyond the source boundary
  throw new IllegalArgumentException("source is " + source.length +
  " bytes long and we're asking for " + (start + currentTokenSize));
}
segments.add(new Range(start, start + currentTokenSize));
start += currentTokenSize;
i += currentTokenSize;
k++;
// if there is more to parse, there must be a separator; strip it
if (k <= sizes.length - 1) {
  for (int j = 0; j < separator.length; j++) {
if (source[i + j] != separator[j]) {
  throw new IllegalArgumentException("separator is expected");
}
continue;
  }
  // matched the separator
  start = i + separator.length;
  i += separator.length;
}
  } else if (currentTokenSize == NO_LIMIT) { // use the separator
// continue until we match the separator
for (int j = 0; j < separator.length; j++) {
  if (source[i + j] != separator[j]) {
i++;
continue itersource;
  }
  // we just matched all separator elements
  segments.add(new Range(start, i));
  start = i + separator.length;
  i += separator.length;
  k++;
}
  } else {
throw new IllegalArgumentException("negative size provided");
  }
}
// add the final segment
if (start < source.length && segments.size() < sizes.length) {
  // by deduction this can happen only if the token size = NO_LIMIT
  segments.add(new Range(start, source.length));
}
return segments;
  }
{code}

You can basically instruct the utility if you expect certain size tokens when 
it parses the bytes. Value = 0 indicates the existing parsing behavior (parse 
until you hit the separator). Positive values indicate the number of bytes to 
grab whether or not there is a separator in the middle. For example, if we 
expect the structure to be a string, a long (as bytes), and a string you can 
invoke it with \{0, Long.BYTES, 0\}.

That way, we can dictate how the row keys and column name qualifiers should be 
parsed.

> timestamps are stored unencoded causing parse errors
> 
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
>  Labels: yarn-2928-1st-milestone
>
> When we store timestamps (for example as part of the row key or part of the 
> column name for an event), the bytes are used as is without any encoding. If 
> the byte value happens to contain a separator character we use (e.g. "!" or 
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
>  incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event) 
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event 
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, 

[jira] [Commented] (YARN-4457) Cleanup unchecked types for EventHandler

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290164#comment-15290164
 ] 

Daniel Templeton commented on YARN-4457:


Oh, [~kasha], if only it were that simple.  A {{Dispatcher}} is intended to 
handle a variety of event types.  The {{register()}} method lets classes 
register a specific handler for a specific event type.  It is really untyped 
with respect to events.  The confusion comes from the {{getEventHandler()}} 
convenience method, which returns a generic event handler, not one of the ones 
that was registered.

In theory we could have the method return an {{EventHandler}}, 
which would be more specific.  It doesn't actually change anything, however, as 
the {{EventHandler.handle()}} method (it's only method) would still accept a 
parameter of type {{Event}}, and it's bad practice to have a wildcarded type as 
the return type.  Wildcards are icky, and forcing callers to deal with them by 
putting them in the return type is bad form.

> Cleanup unchecked types for EventHandler
> 
>
> Key: YARN-4457
> URL: https://issues.apache.org/jira/browse/YARN-4457
> Project: Hadoop YARN
>  Issue Type: Improvement
>Affects Versions: 2.7.1
>Reporter: Daniel Templeton
>Assignee: Daniel Templeton
> Attachments: YARN-4457.001.patch, YARN-4457.002.patch, 
> YARN-4457.003.patch, YARN-4457.004.patch, YARN-4457.005.patch
>
>
> The EventHandler class is often used in an untyped context resulting in a 
> bunch of warnings about unchecked usage.  The culprit is the 
> {{Dispatcher.getHandler()}} method.  Fixing the typing on the method to 
> return {{EventHandler}} instead of {{EventHandler}} clears up the 
> errors and doesn't not introduce any incompatible changes.  In the case that 
> some code does:
> {code}
> EventHandler h = dispatcher.getHandler();
> {code}
> it will still work and will issue a compiler warning about raw types.  There 
> are, however, no instances of this issue in the current source base.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5112) Excessive log warnings on NM recovery

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290125#comment-15290125
 ] 

Jian He commented on YARN-5112:
---

move the check and init part to the common LogAggregationService init.

> Excessive log warnings on NM recovery
> -
>
> Key: YARN-5112
> URL: https://issues.apache.org/jira/browse/YARN-5112
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5112.patch
>
>
> When there are a lot of apps to recover in NM store, NM prints these two 
> lines for each app, which gets annoying.
> {code}
> 2015-10-13 01:58:40,277 WARN  logaggregation.LogAggregationService 
> (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root 
> Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: 
> [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple 
> users.
> 336 2015-10-13 01:58:40,277 WARN  logaggregation.AppLogAggregatorImpl 
> (AppLogAggregatorImpl.java:(182)) - rollingMonitorInterval is set as 
> -1. The log rolling mornitoring interval is disabled. The logs will be 
> aggregated after this application is finished.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5112) Excessive log warnings on NM recovery

[Jian He (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jian He updated YARN-5112:
--
Attachment: YARN-5112.patch

> Excessive log warnings on NM recovery
> -
>
> Key: YARN-5112
> URL: https://issues.apache.org/jira/browse/YARN-5112
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Jian He
>Assignee: Jian He
> Attachments: YARN-5112.patch
>
>
> When there are a lot of apps to recover in NM store, NM prints these two 
> lines for each app, which gets annoying.
> {code}
> 2015-10-13 01:58:40,277 WARN  logaggregation.LogAggregationService 
> (LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root 
> Log Dir [/app-logs] already exist, but with incorrect permissions. Expected: 
> [rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple 
> users.
> 336 2015-10-13 01:58:40,277 WARN  logaggregation.AppLogAggregatorImpl 
> (AppLogAggregatorImpl.java:(182)) - rollingMonitorInterval is set as 
> -1. The log rolling mornitoring interval is disabled. The logs will be 
> aggregated after this application is finished.
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Daniel Templeton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated YARN-4464:
---
Attachment: YARN-4664.002.patch

Added the yarn-defaults.xml change.

> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch, YARN-4664.002.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4987) Read cache concurrency issue between read and evict in EntityGroupFS timeline store

[Li Lu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Li Lu updated YARN-4987:

Attachment: YARN-4987-trunk.002.patch

Thanks [~djp] for the review! I addressed your comments in the latest patch. 
Also, after some offline discussion with [~hitesh], I added one more unit test 
that can simulate the concurrency issue and make sure the fix is right. 

> Read cache concurrency issue between read and evict in EntityGroupFS timeline 
> store 
> 
>
> Key: YARN-4987
> URL: https://issues.apache.org/jira/browse/YARN-4987
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Li Lu
>Assignee: Li Lu
>Priority: Critical
> Attachments: YARN-4987-trunk.001.patch, YARN-4987-trunk.002.patch
>
>
> To handle concurrency issues, key value based timeline storage may return 
> null on reads that are concurrent to service stop. This is actually caused by 
> a concurrency issue between cache reads and evicts. Specifically, if the 
> storage is being read when it gets evicted, the storage may turn into null. 
> EntityGroupFS timeline store needs to handle this case gracefully. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290090#comment-15290090
 ] 

Arun Suresh commented on YARN-5110:
---

Most of the test failures look un-related and are being tracked by YARN-5107 
and YARN-5091. TestMiniYarnClusterNodeUtilization works fine locally..

LGTM, +1 pending another Jenkins run

> Minor fixes and refactoring for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Attachment: YARN-5110.002.patch

> Minor fixes and refactoring for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch, YARN-5110.002.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4308) ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290083#comment-15290083
 ] 

Daniel Templeton commented on YARN-4308:


I'd go one step further and ask that the comments be a javadoc comments *and* 
that the javadoc comments be complete javadoc for the methods, i.e. add javadoc 
comments for the methods that are missing it.

> ContainersAggregated CPU resource utilization reports negative usage in first 
> few heartbeats
> 
>
> Key: YARN-4308
> URL: https://issues.apache.org/jira/browse/YARN-4308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4308.patch, 0002-YARN-4308.patch, 
> 0003-YARN-4308.patch
>
>
> NodeManager reports ContainerAggregated CPU resource utilization as -ve value 
> in first few heartbeats cycles. I added a new debug print and received below 
> values from heartbeats.
> {noformat}
> INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  ContainersResource Utilization : CpuTrackerUsagePercent : -1.0 
> INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:ContainersResource
>  Utilization :  CpuTrackerUsagePercent : 198.94598
> {noformat}
> Its better we send 0 as CPU usage rather than sending a negative values in 
> heartbeats eventhough its happening in only first few heartbeats.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5093) created time shows 0 in most REST output

[Vrushali C (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290080#comment-15290080
 ] 

Vrushali C commented on YARN-5093:
--

Alright, I chatted with Sangjin offline. 

I think we can put in a specific check for created time in the 
HBaseTimelineWriterImpl#storeInfo function to write it out only if it is set.

I suggest defining a constant value such a final static long NO_VALUE_SET which 
is set to -1 (or some such value which makes it clear that it's not set yet). 
The TimelineEntity constructor can then initialize the created time member to 
NO_VALUE_SET. That way, in the writer code, we can check if the value of 
created time is equal to NO_VALUE_SET or not. If not, we can write it out. 

Let me know if you want to offload this jira. 

> created time shows 0 in most REST output
> 
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as 
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow 
> run page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5107) TestContainerMetrics fails

[Arun Suresh (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290070#comment-15290070
 ] 

Arun Suresh commented on YARN-5107:
---

+1, looks good..

> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: YARN-5107.01.patch
>
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
>   Time elapsed: 0.022 sec  <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsFlow(TestContainerMetrics.java:53)
> testContainerMetricsLimit(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
>   Time elapsed: 0.001 sec  <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsLimit(TestContainerMetrics.java:93)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4878) Expose scheduling policy and max running apps over JMX for Yarn queues

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-4878:
---
Attachment: YARN-4878.003.patch

Upload patch 003 for [~rchiang]'s comments. 

> Expose scheduling policy and max running apps over JMX for Yarn queues
> --
>
> Key: YARN-4878
> URL: https://issues.apache.org/jira/browse/YARN-4878
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-4878.001.patch, YARN-4878.002.patch, 
> YARN-4878.003.patch
>
>
> There are two things that are not currently visible over JMX: the current 
> scheduling policy for a queue, and the number of max running apps. It would 
> be great if these could be exposed over JMX as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5107) TestContainerMetrics fails

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290054#comment-15290054
 ] 

Daniel Templeton commented on YARN-5107:


LGTM.  +1 (non-binding).

What I want to know is how this test has been failing consistently for 16 
months without anyone noticing...

> TestContainerMetrics fails
> --
>
> Key: YARN-5107
> URL: https://issues.apache.org/jira/browse/YARN-5107
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: test
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: YARN-5107.01.patch
>
>
> {noformat}
> testContainerMetricsFlow(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
>   Time elapsed: 0.022 sec  <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsFlow(TestContainerMetrics.java:53)
> testContainerMetricsLimit(org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics)
>   Time elapsed: 0.001 sec  <<< ERROR!
> org.mockito.exceptions.misusing.WrongTypeOfReturnValue: 
> TestContainerMetrics cannot be returned by register()
> register() should return MetricsSink
>   at 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics.testContainerMetricsLimit(TestContainerMetrics.java:93)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4878) Expose scheduling policy and max running apps over JMX for Yarn queues

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290053#comment-15290053
 ] 

Yufei Gu commented on YARN-4878:


Thanks [~rchiang] for reviewing. 

1. Nice point.
2. Since function {{forqueue}} is unrelated to this JIRA, should we just leave 
it?

> Expose scheduling policy and max running apps over JMX for Yarn queues
> --
>
> Key: YARN-4878
> URL: https://issues.apache.org/jira/browse/YARN-4878
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-4878.001.patch, YARN-4878.002.patch
>
>
> There are two things that are not currently visible over JMX: the current 
> scheduling policy for a queue, and the number of max running apps. It would 
> be great if these could be exposed over JMX as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4957) Add getNewReservation in ApplicationClientProtocol

[Sean Po (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Po updated YARN-4957:
--
Attachment: YARN-4957.v7.patch

YARN-4957.v7.patch addresses Carlo's comments and fixes the checkstyle and 
javadoc issues. Two check-style issues with regards to method length will 
remain - these are pre-existing, and a refactor will be out of the scope of 
this JIRA. 

> Add getNewReservation in ApplicationClientProtocol
> --
>
> Key: YARN-4957
> URL: https://issues.apache.org/jira/browse/YARN-4957
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: applications, client, resourcemanager
>Affects Versions: 2.8.0
>Reporter: Subru Krishnan
>Assignee: Sean Po
>  Labels: api-breaking
> Attachments: YARN-4957.v0.patch, YARN-4957.v1.patch, 
> YARN-4957.v2.patch, YARN-4957.v3.patch, YARN-4957.v4.patch, 
> YARN-4957.v5.1.patch, YARN-4957.v5.2.patch, YARN-4957.v5.patch, 
> YARN-4957.v7.patch
>
>
> Currently submitReservation returns a ReservationId if sucessful. This JIRA 
> propose adding a getNewReservation in ApplicationClientProtocol for the 
> following reasons:
>   * Prevent zombie reservations in the face of client and/or network failures 
> post submitReservation
>   * Align reservation submission with application submission



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Arun Suresh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Arun Suresh updated YARN-5110:
--
Description: 
* Couple of Logs can be moved to debug rather than info
* Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
address, not just the hostname.
* Minor refactoring w.r.t method attribute names

  was:
* Couple of Logs can be moved to debug rather than info
* Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
address, not just the hostname.
* Minor refactoring


> Minor fixes and refactoring for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring w.r.t method attribute names



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5112) Excessive log warnings on NM recovery

[Jian He (JIRA)]

Jian He created YARN-5112:
-

 Summary: Excessive log warnings on NM recovery
 Key: YARN-5112
 URL: https://issues.apache.org/jira/browse/YARN-5112
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jian He
Assignee: Jian He


When there are a lot of apps to recover in NM store, NM prints these two lines 
for each app, which gets annoying.

{code}
2015-10-13 01:58:40,277 WARN  logaggregation.LogAggregationService 
(LogAggregationService.java:verifyAndCreateRemoteLogDir(195)) - Remote Root Log 
Dir [/app-logs] already exist, but with incorrect permissions. Expected: 
[rwxrwxrwt], Found: [rwxrwxrwx]. The cluster may have problems with multiple 
users.
336 2015-10-13 01:58:40,277 WARN  logaggregation.AppLogAggregatorImpl 
(AppLogAggregatorImpl.java:(182)) - rollingMonitorInterval is set as -1. 
The log rolling mornitoring interval is disabled. The logs will be aggregated 
after this application is finished.
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15290019#comment-15290019
 ] 

Hadoop QA commented on YARN-5110:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s 
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s 
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 
0s {color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s 
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 10s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s 
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
32s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s 
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s 
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s 
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 24s 
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: patch 
generated 1 new + 59 unchanged - 0 fixed = 60 total (was 59) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 30s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s 
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s 
{color} | {color:green} hadoop-yarn-server-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 32s {color} 
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 20s {color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 6m 54s {color} 
| {color:red} hadoop-yarn-server-tests in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 
16s {color} | {color:green} Patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 78m 0s {color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | 
hadoop.yarn.server.nodemanager.containermanager.monitor.TestContainerMetrics |
|   | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.TestClientRMTokens |
|   | hadoop.yarn.server.resourcemanager.TestAMAuthorization |
|   | hadoop.yarn.server.TestContainerManagerSecurity |
|   | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:2c91fd8 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12804781/YARN-5110.001.patch |
| JIRA Issue | YARN-5110 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 422e8c38091b 3.13.0-36-lowlatency #63-Ubuntu SMP 

[jira] [Updated] (YARN-1942) Many of ConverterUtils methods need to have public interfaces

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-1942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-1942:
-
Attachment: YARN-1942.6.patch

Attached ver.6 patch addressed more warnings.

> Many of ConverterUtils methods need to have public interfaces
> -
>
> Key: YARN-1942
> URL: https://issues.apache.org/jira/browse/YARN-1942
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api
>Affects Versions: 2.4.0
>Reporter: Thomas Graves
>Assignee: Wangda Tan
>Priority: Critical
> Attachments: YARN-1942.1.patch, YARN-1942.2.patch, YARN-1942.3.patch, 
> YARN-1942.4.patch, YARN-1942.5.patch, YARN-1942.6.patch
>
>
> ConverterUtils has a bunch of functions that are useful to application 
> masters.   It should either be made public or we make some of the utilities 
> in it public or we provide other external apis for application masters to 
> use.  Note that distributedshell and MR are both using these interfaces. 
> For instance the main use case I see right now is for getting the application 
> attempt id within the appmaster:
> String containerIdStr =
>   System.getenv(Environment.CONTAINER_ID.name());
> ConverterUtils.toContainerId
> ContainerId containerId = ConverterUtils.toContainerId(containerIdStr);
>   ApplicationAttemptId applicationAttemptId =
>   containerId.getApplicationAttemptId();
> I don't see any other way for the application master to get this information. 
>  If there is please let me know.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4957) Add getNewReservation in ApplicationClientProtocol

[Sean Po (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289950#comment-15289950
 ] 

Sean Po commented on YARN-4957:
---

Are the classes "@Stable"? the methods are marked "@Unstable"... I am asking as 
I am not sure how we should use stable/unstable at the class vs method level.

> Stable and Unstable refers to the Stability of the interface. If we expect 
> the interface to change, then it should be annotated Unstable - otherwise, 
> annotate it Stable.

Why YARNClient.createReservation() in stead of 
YARNClient.createNewReservationId()?

> YarnClient.createReservation because I wanted this API to mirror the 
> application API, for consistency. In YarnClient, the method used to create an 
> application ID is YarnClient.createApplication.

The submitReservation is idempotent if we submit exactly the same 
ReservationDefinition, but will reject if I try to reuse the reservationId with 
same Id but a different ReservationDefinition

> That's exactly it. I have updated the error message in the next patch.

In the "ReservationSubmissionResponse" should we retain the ReservationId? I 
think might make the implementation on the client easier at times (if multiple 
reservations are submitted asynchronsouly or something like that, it helps 
retain the info of "who got accepted"). For now invocations are synch, but we 
might want to change it later. Thoughts?

> I tend to agree with you, but I think it is better that the Reservation and 
> Application APIs are consistent with one another. Currently in 
> SubmitApplicationResponse, no ApplicationId is returned to the client.

You have copy and pasted code in TestYarnClient and TestClientRMService, 
consider (if possible for packaging etc..) to put this in a shared testbase 
class.

> I agree that it a shared test base will be good, but it is broader than this 
> JIRA. Perhaps we can create another JIRA for test refactoring.

What do "list" return if I have obtained a ReservationId, but never used it to 
submit a reservation?

> List will not return the new ReservationId.

Why in the tests do you use a mock "generateReservationId(int)" instead of 
using the createReservation API?

> This is a mistake - will update this in the next patch.



> Add getNewReservation in ApplicationClientProtocol
> --
>
> Key: YARN-4957
> URL: https://issues.apache.org/jira/browse/YARN-4957
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: applications, client, resourcemanager
>Affects Versions: 2.8.0
>Reporter: Subru Krishnan
>Assignee: Sean Po
>  Labels: api-breaking
> Attachments: YARN-4957.v0.patch, YARN-4957.v1.patch, 
> YARN-4957.v2.patch, YARN-4957.v3.patch, YARN-4957.v4.patch, 
> YARN-4957.v5.1.patch, YARN-4957.v5.2.patch, YARN-4957.v5.patch
>
>
> Currently submitReservation returns a ReservationId if sucessful. This JIRA 
> propose adding a getNewReservation in ApplicationClientProtocol for the 
> following reasons:
>   * Prevent zombie reservations in the face of client and/or network failures 
> post submitReservation
>   * Align reservation submission with application submission



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5093) created time shows 0 in most REST output

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289917#comment-15289917
 ] 

Sangjin Lee commented on YARN-5093:
---

It is getting written. It's that subsequent writes for that entity are 
overwriting it to 0.

> created time shows 0 in most REST output
> 
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as 
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow 
> run page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Ray Chiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289911#comment-15289911
 ] 

Ray Chiang commented on YARN-4464:
--

Nope.  Thanks for the explanation.

> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289906#comment-15289906
 ] 

Karthik Kambatla commented on YARN-4464:


bq. If we change the default value to 0
The zero is only for completed applications. We need the store to store all 
running applications so they can be recovered. So, the ZK connection is still 
required. Am I missing something? 

> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4999) [YARN-3368] Fix licensing issues in new Yarn-ui

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289904#comment-15289904
 ] 

Wangda Tan commented on YARN-4999:
--

+1, thanks [~sunilg]

> [YARN-3368] Fix licensing issues in new Yarn-ui
> ---
>
> Key: YARN-4999
> URL: https://issues.apache.org/jira/browse/YARN-4999
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: YARN-4999-YARN-3368.1.patch, YARN-4999-YARN-3368.2.patch
>
>
> Few more jars used in new YARN web-ui needs to be updated in LICENSE.txt



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4494) Recover completed apps asynchronously

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289900#comment-15289900
 ] 

Karthik Kambatla commented on YARN-4494:


I am okay with asynchronous recovery so long as it is transparent to clients. 
That is, requesting details about a completed app should return the same values 
if the recovery was synchronous. This would also apply to queries for apps of a 
particular type, start/finish time etc. 

It might be possible for the client calls to synchronously wait for recovery to 
finish, but I get the feeling it might not be all that straight-forward. Given 
the option of a low number of completed applications and ATSv2 being around the 
corner, my recommendation would be to not look at asynchronous recovery. 

> Recover completed apps asynchronously
> -
>
> Key: YARN-4494
> URL: https://issues.apache.org/jira/browse/YARN-4494
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Reporter: Jun Gong
>Assignee: Jun Gong
>
> With RM HA enabled, when recovering apps, recover completed apps 
> asynchronously.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289895#comment-15289895
 ] 

Sangjin Lee commented on YARN-5109:
---

I spoke with [~jrottinghuis] offline about this. Initially we were thinking 
that we should encode those characters even in the case of bytes (essentially 
creating {{Separator.joinEncoded(byte[]...)}} and removing the raw 
{{Separator.join()}} method), but we are realizing that won't work.

The key here is that we not only need to handle those separator characters 
("=", "!", etc.) but also *preserve the ordering*. For example, suppose we have 
two timestamps ({{ts1}} and {{ts2}}) where {{ts1 < ts2}}. And assume {{ts2}} 
has a separator character in it. If we blindly encoded the separator character, 
we could easily violate {{ts1 < ts2}} once they are written. This would break 
all sorts of things, including range scans.

My proposal is this. In almost all of these cases, the structure of the data 
we're storing and parsing is known strongly, whether it is the row key or the 
column qualifier. The problem with the current parsing is it uses solely 
splitting by separator. We should use the full data structure it knows already 
to parse correctly.

For example, if we know that the structure is (string)=(timestamp)=(string), we 
can parse the first string, and then take the next 8 bytes *without splitting 
again* as we know it's a timestamp anyway and convert it into the long number, 
and take the last token after that. We should be able to follow the same idea 
in all cases.

Thoughts? [~varun_saxena], let me know if you'd like to take a stab at that 
idea, or I should.

> timestamps are stored unencoded causing parse errors
> 
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
>  Labels: yarn-2928-1st-milestone
>
> When we store timestamps (for example as part of the row key or part of the 
> column name for an event), the bytes are used as is without any encoding. If 
> the byte value happens to contain a separator character we use (e.g. "!" or 
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
>  incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event) 
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event 
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4882) Change the log level to DEBUG for recovering completed applications

[Karthik Kambatla (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289885#comment-15289885
 ] 

Karthik Kambatla commented on YARN-4882:


Thanks for your inputs, [~jlowe]. I would say, let us start with debug logging 
success cases. If we ever run into a case of needing logs for success cases, 
let us add a separate logger. 

> Change the log level to DEBUG for recovering completed applications
> ---
>
> Key: YARN-4882
> URL: https://issues.apache.org/jira/browse/YARN-4882
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Rohith Sharma K S
>Assignee: Daniel Templeton
>
> I think for recovering completed applications no need to log as INFO, rather 
> it can be made it as DEBUG.  The problem seen from large cluster is if any 
> issue happens during RM start up and continuously switching , then  RM logs 
> are filled with most with recovering applications only. 
> There are 6 lines are logged for 1 applications as I shown in below logs, 
> then consider RM default value for max-completed applications is 10K. So for 
> each switch 10K*6=60K lines will be added which is not useful I feel.
> {noformat}
> 2016-03-01 10:20:59,077 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAppManager: Default priority 
> level is set to application:application_1456298208485_21507
> 2016-03-01 10:20:59,094 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: Recovering 
> app: application_1456298208485_21507 with 1 attempts and final state = 
> FINISHED
> 2016-03-01 10:20:59,100 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> Recovering attempt: appattempt_1456298208485_21507_01 with final state: 
> FINISHED
> 2016-03-01 10:20:59,107 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl: 
> appattempt_1456298208485_21507_01 State change from NEW to FINISHED
> 2016-03-01 10:20:59,111 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.RMAppImpl: 
> application_1456298208485_21507 State change from NEW to FINISHED
> 2016-03-01 10:20:59,112 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=rohith   
> OPERATION=Application Finished - Succeeded  TARGET=RMAppManager 
> RESULT=SUCCESS  APPID=application_1456298208485_21507
> {noformat}
> The main problem is missing important information's from the logs before RM 
> unstable. Even though log roll back is 50 or 100, in a short period all these 
> logs will be rolled out and all the logs contains only RM switching 
> information that too recovering applications!!. 
> I suggest at least completed applications recovery should be logged as DEBUG.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5001) Aggregated Logs root directory is created with wrong group if nonexistent

[Ray Chiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289882#comment-15289882
 ] 

Ray Chiang commented on YARN-5001:
--

+1 (nonbinding).  Looks like a simple enough change.

> Aggregated Logs root directory is created with wrong group if nonexistent 
> --
>
> Key: YARN-5001
> URL: https://issues.apache.org/jira/browse/YARN-5001
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.0
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: yarn5001.001.patch
>
>
> The directory /tmp/logs, where the aggregated logs go, is supposed to be read 
> by JHS. But if it is not created beforehand, it will be created by 
> NodeManager with the group being the superuser set in HDFS.  Files created 
> under this directory will then inherit the supergroup as their groups. This 
> leads to JHS to fail to read the container logs files under that directory if 
> JHS is not running as a user that belongs to superuser.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4844) Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource

[Wangda Tan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wangda Tan updated YARN-4844:
-
Attachment: YARN-4844.10.patch

Attached ver.10 patch, removed all YARN/MR internal usages of {{int 
getMemory()}}

> Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource
> 
>
> Key: YARN-4844
> URL: https://issues.apache.org/jira/browse/YARN-4844
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: api
>Reporter: Wangda Tan
>Assignee: Wangda Tan
>Priority: Blocker
> Attachments: YARN-4844.1.patch, YARN-4844.10.patch, 
> YARN-4844.2.patch, YARN-4844.3.patch, YARN-4844.4.patch, YARN-4844.5.patch, 
> YARN-4844.6.patch, YARN-4844.7.patch, YARN-4844.8.branch-2.patch, 
> YARN-4844.8.patch, YARN-4844.9.branch, YARN-4844.9.branch-2.patch
>
>
> We use int32 for memory now, if a cluster has 10k nodes, each node has 210G 
> memory, we will get a negative total cluster memory.
> And another case that easier overflows int32 is: we added all pending 
> resources of running apps to cluster's total pending resources. If a 
> problematic app requires too much resources (let's say 1M+ containers, each 
> of them has 3G containers), int32 will be not enough.
> Even if we can cap each app's pending request, we cannot handle the case that 
> there're many running apps, each of them has capped but still significant 
> numbers of pending resources.
> So we may possibly need to add getMemoryLong/getVirtualCoreLong to 
> o.a.h.y.api.records.Resource.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4866) FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-4866:
---
Attachment: YARN-4866.005.patch

Thanks [~kasha] for reviewing. 

I upload a new patch for all suggestions.

> FairScheduler: AMs can consume all vcores leading to a livelock when using 
> FAIR policy
> --
>
> Key: YARN-4866
> URL: https://issues.apache.org/jira/browse/YARN-4866
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: fairscheduler
>Affects Versions: 2.8.0
>Reporter: Karthik Kambatla
>Assignee: Yufei Gu
> Attachments: YARN-4866.001.patch, YARN-4866.002.patch, 
> YARN-4866.003.patch, YARN-4866.004.patch, YARN-4866.005.patch
>
>
> The maxAMShare uses the queue's policy for enforcing limits. When using FAIR 
> policy, this considers only memory. If there are fewer vcores on the cluster, 
> the AMs can end up taking all the vcores leading to a livelock. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5110) Minor fixes and refactoring for Distributed Scheduling

[Konstantinos Karanasos (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Konstantinos Karanasos updated YARN-5110:
-
Attachment: YARN-5110.001.patch

Attaching an intermediate version of the patch just to kick off Jenkins.

> Minor fixes and refactoring for Distributed Scheduling
> --
>
> Key: YARN-5110
> URL: https://issues.apache.org/jira/browse/YARN-5110
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Konstantinos Karanasos
> Attachments: YARN-5110.001.patch
>
>
> * Couple of Logs can be moved to debug rather than info
> * Minor fix required in {{OpportunisticContainerAllocator}} : Fix 
> ContainerTokenIdentifier::getNmHostAddress() to return the complete host 
> address, not just the hostname.
> * Minor refactoring



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4766) NM should not aggregate logs older than the retention policy

[Ray Chiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4766?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289743#comment-15289743
 ] 

Ray Chiang commented on YARN-4766:
--

Minor nit: Rename noMoreRetentionLogFiles to obsoleteRetentionLogFIles.  
Similar rename for getNoMoreRetentionLogFiles().

> NM should not aggregate logs older than the retention policy
> 
>
> Key: YARN-4766
> URL: https://issues.apache.org/jira/browse/YARN-4766
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: log-aggregation, nodemanager
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: yarn4766.001.patch, yarn4766.002.patch, 
> yarn4766.003.patch, yarn4766.004.patch, yarn4766.004.patch
>
>
> When a log aggregation fails on the NM the information is for the attempt is 
> kept in the recovery DB. Log aggregation can fail for multiple reasons which 
> are often related to HDFS space or permissions.
> On restart the recovery DB is read and if an application attempt needs its 
> logs aggregated, the files are scheduled for aggregation without any checks. 
> The log files could be older than the retention limit in which case we should 
> not aggregate them but immediately mark them for deletion from the local file 
> system.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5093) created time shows 0 in most REST output

[Vrushali C (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289720#comment-15289720
 ] 

Vrushali C commented on YARN-5093:
--

Related jira for "correctly set createdTime and remove modifiedTime when 
publishing entities" :  https://issues.apache.org/jira/browse/YARN-4238

The patch in the jira above should have been populating the created time at 
entity creation. Did this get "lost" in the merge? IIRC, entity creation time 
has to be populated when the entity is created. Is it getting overwritten to 
become null? 

The writer cannot distinguish between a real 0 and non-existing value. If at 
all that is to be done at the writer side, then we need to initialize numeric 
members with something like "NON_EXISTENT_VALUE" constant which could be set to 
-1 or something.


> created time shows 0 in most REST output
> 
>
> Key: YARN-5093
> URL: https://issues.apache.org/jira/browse/YARN-5093
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Critical
>  Labels: yarn-2928-1st-milestone
>
> When querying the REST API, I find that the created time value is returned as 
> "0" for most of the output. It includes:
> - flow activity and flow runs in the flow activity page
> - apps in the application page
> - entities in the entity page
> For example, in the flow activity page,
> {noformat}
> {
> metrics: [ ],
> events: [ ],
> id: "yarn_cluster/146335680/sjlee@ds-date",
> type: "YARN_FLOW_ACTIVITY",
> createdtime: 0,
> flowruns: [
> {
> metrics: [ ],
> events: [ ],
> id: "sjlee@ds-date/1463435661428",
> type: "YARN_FLOW_RUN",
> createdtime: 0,
> info: {
> SYSTEM_INFO_FLOW_VERSION: "1",
> SYSTEM_INFO_FLOW_RUN_ID: 1463435661428,
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> ],
> info: {
> SYSTEM_INFO_CLUSTER: "yarn_cluster",
> UID: "yarn_cluster!sjlee!ds-date",
> SYSTEM_INFO_FLOW_NAME: "ds-date",
> SYSTEM_INFO_DATE: 146335680,
> SYSTEM_INFO_USER: "sjlee"
> },
> isrelatedto: { },
> relatesto: { }
> }
> {noformat}
> The only page that appears to show the proper created time value is the flow 
> run page.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Ray Chiang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ray Chiang updated YARN-4464:
-
Target Version/s: 3.0.0-alpha1

> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4464) default value of yarn.resourcemanager.state-store.max-completed-applications should lower.

[Ray Chiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289707#comment-15289707
 ] 

Ray Chiang commented on YARN-4464:
--

Agree about making this trunk-only.

One thought.  If we change the default value to 0, it would be good to verify 
that we never actually create a local state store file or open a ZK connection. 
 It would be wasteful to actually do any file/network open if we're going to 
send nothing.

> default value of yarn.resourcemanager.state-store.max-completed-applications 
> should lower.
> --
>
> Key: YARN-4464
> URL: https://issues.apache.org/jira/browse/YARN-4464
> Project: Hadoop YARN
>  Issue Type: Wish
>  Components: resourcemanager
>Reporter: KWON BYUNGCHANG
>Assignee: Daniel Templeton
>Priority: Blocker
> Attachments: YARN-4464.001.patch
>
>
> my cluster has 120 nodes.
> I configured RM Restart feature.
> {code}
> yarn.resourcemanager.recovery.enabled=true
> yarn.resourcemanager.store.class=org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore
> yarn.resourcemanager.fs.state-store.uri=/system/yarn/rmstore
> {code}
> unfortunately I did not configure 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> so that property configured default value 10,000.
> I have restarted RM due to changing another configuartion.
> I expected that RM restart immediately.
> recovery process was very slow.  I have waited about 20min.  
> realize missing 
> {{yarn.resourcemanager.state-store.max-completed-applications}}.
> its default value is very huge.  
> need to change lower value or document notice on [RM Restart 
> page|http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5076) YARN web interfaces lack XFS protection

[Jonathan Maron (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jonathan Maron updated YARN-5076:
-
Attachment: YARN-5076.003.patch

[~vvasudev] - please review and see if this addresses the changes requested.  
Thanks!

> YARN web interfaces lack XFS protection
> ---
>
> Key: YARN-5076
> URL: https://issues.apache.org/jira/browse/YARN-5076
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager, timelineserver
>Reporter: Jonathan Maron
>Assignee: Jonathan Maron
> Attachments: YARN-5076.002.patch, YARN-5076.003.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are web interfaces in YARN that do not provide protection against cross 
> frame scripting 
> (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet).  
> HADOOP-13008 provides a common filter for addressing this vulnerability, so 
> this filter should be integrated into the YARN web interfaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4878) Expose scheduling policy and max running apps over JMX for Yarn queues

[Ray Chiang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289697#comment-15289697
 ] 

Ray Chiang commented on YARN-4878:
--

Some minor nitpicks:

- Change the metric description from "Maximum applications number" to "Maximum 
number of applications"
- For readability, I'd recommend changing this style:

{noformat}
  public synchronized
  static FSQueueMetrics forQueue(String queueName, Queue parent,
  boolean enableUserMetrics, Configuration conf) {
{noformat}

   to this:

{noformat}
  public static synchronized FSQueueMetrics forQueue(String queueName,
  Queue parent, boolean enableUserMetrics, Configuration conf) {
{noformat}

  In general, it's best to have the code readable, not best diff compatibility.


> Expose scheduling policy and max running apps over JMX for Yarn queues
> --
>
> Key: YARN-4878
> URL: https://issues.apache.org/jira/browse/YARN-4878
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-4878.001.patch, YARN-4878.002.patch
>
>
> There are two things that are not currently visible over JMX: the current 
> scheduling policy for a queue, and the number of max running apps. It would 
> be great if these could be exposed over JMX as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5109) timestamps are stored unencoded causing parse errors

[Sangjin Lee (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289571#comment-15289571
 ] 

Sangjin Lee commented on YARN-5109:
---

These are the all the cases of "naked" joins:
{noformat}org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper.getColumnQualifier(byte[],
 byte[])
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper.getColumnQualifier(byte[],
 long)
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper.getColumnQualifier(byte[],
 String)
org.apache.hadoop.yarn.server.timelineservice.storage.common.ColumnHelper.getCompoundColumnQualifierBytes(String,
 byte[]...)
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowActivityRowKey.getRowKey(String,
 long, String, String)
org.apache.hadoop.yarn.server.timelineservice.storage.entity.EntityRowKey.getRowKey(String,
 String, String, Long, String, String, String)
org.apache.hadoop.yarn.server.timelineservice.storage.application.ApplicationRowKey.getRowKey(String,
 String, String, Long, String)
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowRunRowKey.getRowKey(String,
 String, String, Long)
org.apache.hadoop.yarn.server.timelineservice.storage.apptoflow.AppToFlowRowKey.getRowKey(String,
 String)
org.apache.hadoop.yarn.server.timelineservice.storage.flow.FlowActivityRowKey.getRowKeyPrefix(String,
 long)
org.apache.hadoop.yarn.server.timelineservice.storage.entity.EntityRowKey.getRowKeyPrefix(String,
 String, String, Long, String, String)
org.apache.hadoop.yarn.server.timelineservice.storage.entity.EntityRowKey.getRowKeyPrefix(String,
 String, String, Long, String)
org.apache.hadoop.yarn.server.timelineservice.storage.application.ApplicationRowKey.getRowKeyPrefix(String,
 String, String, Long)
org.apache.hadoop.yarn.server.timelineservice.storage.application.ApplicationRowKey.getRowKeyPrefix(String,
 String, String)
{noformat}

> timestamps are stored unencoded causing parse errors
> 
>
> Key: YARN-5109
> URL: https://issues.apache.org/jira/browse/YARN-5109
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Affects Versions: YARN-2928
>Reporter: Sangjin Lee
>Assignee: Varun Saxena
>Priority: Blocker
>  Labels: yarn-2928-1st-milestone
>
> When we store timestamps (for example as part of the row key or part of the 
> column name for an event), the bytes are used as is without any encoding. If 
> the byte value happens to contain a separator character we use (e.g. "!" or 
> "="), it causes a parse failure when we read it.
> I came across this while looking into this error in the timeline reader:
> {noformat}
> 2016-05-17 21:28:38,643 WARN 
> org.apache.hadoop.yarn.server.timelineservice.storage.common.TimelineStorageUtils:
>  incorrectly formatted column name: it will be discarded
> {noformat}
> I traced the data that was causing this, and the column name (for the event) 
> was the following:
> {noformat}
> i:e!YARN_RM_CONTAINER_CREATED=\x7F\xFF\xFE\xABDY=\x99=YARN_CONTAINER_ALLOCATED_HOST
> {noformat}
> Note that the column name is supposed to be of the format (event 
> id)=(timestamp)=(event info key). However, observe the timestamp portion:
> {noformat}
> \x7F\xFF\xFE\xABDY=\x99
> {noformat}
> The presence of the separator ("=") causes the parse error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5055) max per user can be larger than max per queue

[Eric Badger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Badger updated YARN-5055:
--
Attachment: YARN-5055.001.patch

Attaching patch that limits the max per user application limit to the max queue 
application limit. I tested this and saw the change reflected in the UI as well 
as changing the relevant unit tests. 

> max per user can be larger than max per queue
> -
>
> Key: YARN-5055
> URL: https://issues.apache.org/jira/browse/YARN-5055
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacityscheduler, resourcemanager
>Affects Versions: 2.7.2
>Reporter: Jason Lowe
>Assignee: Eric Badger
>Priority: Minor
> Attachments: YARN-5055.001.patch
>
>
> If user limit and/or user limit factor are >100% then the calculated maximum 
> values per user can exceed the maximum for the queue.  For example, maximum 
> AM resource per user could exceed maximum AM resource for the entire queue, 
> or max applications per user could be larger than max applications for the 
> queue.  The per-user values should be capped by the per queue values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4876) [Phase 1] Decoupled Init / Destroy of Containers from Start / Stop

[Varun Vasudev (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289656#comment-15289656
 ] 

Varun Vasudev commented on YARN-4876:
-

Thanks for the patch [~marco.rabozzi]! Can you please take a look at the 
pre-commit build output above? It looks like the patch doesn't apply cleanly on 
trunk. 

Question about the patch - there is an allowsRestart() function that's being 
used. Can you please clarify what that means - is this referring to if the 
container can be re-started by the AM or the NM?

> [Phase 1] Decoupled Init / Destroy of Containers from Start / Stop
> --
>
> Key: YARN-4876
> URL: https://issues.apache.org/jira/browse/YARN-4876
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Marco Rabozzi
> Attachments: YARN-4876-design-doc.pdf, YARN-4876.01.patch
>
>
> Introduce *initialize* and *destroy* container API into the 
> *ContainerManagementProtocol* and decouple the actual start of a container 
> from the initialization. This will allow AMs to re-start a container without 
> having to lose the allocation.
> Additionally, if the localization of the container is associated to the 
> initialize (and the cleanup with the destroy), This can also be used by 
> applications to upgrade a Container by *re-initializing* with a new 
> *ContainerLaunchContext*



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4751) In 2.7, Labeled queue usage not shown properly in capacity scheduler UI

[Eric Payne (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eric Payne updated YARN-4751:
-
Attachment: YARN-4751-branch-2.7.005.patch

Thanks, [~Naganarasimha]. You are correct, most of the findbugs and checkstyle 
warnings are related to this patch. I am uploading new patch addressing them:
{{YARN-4751-branch-2.7.005.patch}}


> In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
> ---
>
> Key: YARN-4751
> URL: https://issues.apache.org/jira/browse/YARN-4751
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, yarn
>Affects Versions: 2.7.3
>Reporter: Eric Payne
>Assignee: Eric Payne
> Attachments: 2.7 CS UI No BarGraph.jpg, 
> YARH-4752-branch-2.7.001.patch, YARH-4752-branch-2.7.002.patch, 
> YARN-4751-branch-2.7.003.patch, YARN-4751-branch-2.7.004.patch, 
> YARN-4751-branch-2.7.005.patch
>
>
> In 2.6 and 2.7, the capacity scheduler UI does not have the queue graphs 
> separated by partition. When applications are running on a labeled queue, no 
> color is shown in the bar graph, and several of the "Used" metrics are zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-5111) YARN container system metrics are not aggregated to application

[Sangjin Lee (JIRA)]

Sangjin Lee created YARN-5111:
-

 Summary: YARN container system metrics are not aggregated to 
application
 Key: YARN-5111
 URL: https://issues.apache.org/jira/browse/YARN-5111
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: timelineserver
Affects Versions: YARN-2928
Reporter: Sangjin Lee
Priority: Critical


It appears that the container system metrics (CPU and memory) are not being 
aggregated onto the application.

I definitely see container system metrics when I query for YARN_CONTAINER. 
However, there is no corresponding metrics on the parent application.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2877) Extend YARN to support distributed scheduling

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289535#comment-15289535
 ] 

Wangda Tan commented on YARN-2877:
--

An additional note:

Junping and I investigated PB compatible cases when adding new fields to middle 
of a proto.

Let's say:
PB1Proto:
{code}
  optional int32 w = 1;  
  optional int32 x = 2;
  optional int32 z = 4;
{code} 

PB2Proto:
{code}
  optional int32 w = 1;  
  optional int32 x = 2;
  optional int32 y = 3;
  optional int32 z = 4;
{code}

PB2Proto can read PB1Proto without any issue.

So next time we should not update sequence of fields in trunk/branch-2, what we 
need to do is to make sure fields of protos across branches has same id.

> Extend YARN to support distributed scheduling
> -
>
> Key: YARN-2877
> URL: https://issues.apache.org/jira/browse/YARN-2877
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: nodemanager, resourcemanager
>Reporter: Sriram Rao
>Assignee: Konstantinos Karanasos
> Attachments: distributed-scheduling-design-doc_v1.pdf
>
>
> This is an umbrella JIRA that proposes to extend YARN to support distributed 
> scheduling.  Briefly, some of the motivations for distributed scheduling are 
> the following:
> 1. Improve cluster utilization by opportunistically executing tasks otherwise 
> idle resources on individual machines.
> 2. Reduce allocation latency.  Tasks where the scheduling time dominates 
> (i.e., task execution time is much less compared to the time required for 
> obtaining a container from the RM).
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5077) Fix FSLeafQueue#getFairShare() for queues with weight 0.0

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289516#comment-15289516
 ] 

Yufei Gu commented on YARN-5077:


Thanks [~kasha] for reviewing. 

To filter out the non-active queue is done by {{handleFixedFairShares}}, which 
is invoked by {{computeSharesInternal}}. But {{handleFixedFairShares}} does 
more than that, it also filters out zero-weight queue, and that's one of place 
we want to fix in this JIRA. If there are all zero-weight active queues, we 
should get them in instead of removing them. So I should know if there is no 
non-zero-weight active queue before {{handleFixedFairShares}}, which is done by 
func {{noNonZeroWeightActive}}.

I totally agree we should avoid double negation name, but it seems reasonable 
here. Of course, the name could be called {{allWeightsZero}} after function 
{{handleFixedFairShares}}, so I modify the name after that as your suggestions. 

I assume you said YARN-5106. It is fair enough. 


> Fix FSLeafQueue#getFairShare() for queues with weight 0.0
> -
>
> Key: YARN-5077
> URL: https://issues.apache.org/jira/browse/YARN-5077
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5077.001.patch, YARN-5077.002.patch
>
>
> 1) When a queue's weight is set to 0.0, FSLeafQueue#getFairShare() returns 
>  
> 2) When a queue's weight is nonzero, FSLeafQueue#getFairShare() returns 
> 
> In case 1), that means no container ever gets allocated for an AM because 
> from the viewpoint of the RM, there is never any headroom to allocate a 
> container on that queue.
> For example, we have a pool with the following weights: 
> - root.dev 0.0 
> - root.product 1.0
> The root.dev is a best effort pool and should only get resources if 
> root.product is not running. In our tests, with no jobs running under 
> root.product, jobs started in root.dev queue stay stuck in ACCEPT phase and 
> never start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5077) Fix FSLeafQueue#getFairShare() for queues with weight 0.0

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5077:
---
Attachment: YARN-5077.003.patch

> Fix FSLeafQueue#getFairShare() for queues with weight 0.0
> -
>
> Key: YARN-5077
> URL: https://issues.apache.org/jira/browse/YARN-5077
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5077.001.patch, YARN-5077.002.patch, 
> YARN-5077.003.patch
>
>
> 1) When a queue's weight is set to 0.0, FSLeafQueue#getFairShare() returns 
>  
> 2) When a queue's weight is nonzero, FSLeafQueue#getFairShare() returns 
> 
> In case 1), that means no container ever gets allocated for an AM because 
> from the viewpoint of the RM, there is never any headroom to allocate a 
> container on that queue.
> For example, we have a pool with the following weights: 
> - root.dev 0.0 
> - root.product 1.0
> The root.dev is a best effort pool and should only get resources if 
> root.product is not running. In our tests, with no jobs running under 
> root.product, jobs started in root.dev queue stay stuck in ACCEPT phase and 
> never start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5077) Fix FSLeafQueue#getFairShare() for queues with weight 0.0

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5077:
---
Attachment: (was: YARN-5077.003.patch)

> Fix FSLeafQueue#getFairShare() for queues with weight 0.0
> -
>
> Key: YARN-5077
> URL: https://issues.apache.org/jira/browse/YARN-5077
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5077.001.patch, YARN-5077.002.patch
>
>
> 1) When a queue's weight is set to 0.0, FSLeafQueue#getFairShare() returns 
>  
> 2) When a queue's weight is nonzero, FSLeafQueue#getFairShare() returns 
> 
> In case 1), that means no container ever gets allocated for an AM because 
> from the viewpoint of the RM, there is never any headroom to allocate a 
> container on that queue.
> For example, we have a pool with the following weights: 
> - root.dev 0.0 
> - root.product 1.0
> The root.dev is a best effort pool and should only get resources if 
> root.product is not running. In our tests, with no jobs running under 
> root.product, jobs started in root.dev queue stay stuck in ACCEPT phase and 
> never start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5077) Fix FSLeafQueue#getFairShare() for queues with weight 0.0

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-5077:
---
Attachment: YARN-5077.003.patch

> Fix FSLeafQueue#getFairShare() for queues with weight 0.0
> -
>
> Key: YARN-5077
> URL: https://issues.apache.org/jira/browse/YARN-5077
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-5077.001.patch, YARN-5077.002.patch
>
>
> 1) When a queue's weight is set to 0.0, FSLeafQueue#getFairShare() returns 
>  
> 2) When a queue's weight is nonzero, FSLeafQueue#getFairShare() returns 
> 
> In case 1), that means no container ever gets allocated for an AM because 
> from the viewpoint of the RM, there is never any headroom to allocate a 
> container on that queue.
> For example, we have a pool with the following weights: 
> - root.dev 0.0 
> - root.product 1.0
> The root.dev is a best effort pool and should only get resources if 
> root.product is not running. In our tests, with no jobs running under 
> root.product, jobs started in root.dev queue stay stuck in ACCEPT phase and 
> never start.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289507#comment-15289507
 ] 

Junping Du commented on YARN-5076:
--

Yes. Varun's list above is exactly what I prefer. Please go ahead to update. 
Also, SAMEORIGIN as default make more sense also. Thanks!

> YARN web interfaces lack XFS protection
> ---
>
> Key: YARN-5076
> URL: https://issues.apache.org/jira/browse/YARN-5076
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager, timelineserver
>Reporter: Jonathan Maron
>Assignee: Jonathan Maron
> Attachments: YARN-5076.002.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are web interfaces in YARN that do not provide protection against cross 
> frame scripting 
> (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet).  
> HADOOP-13008 provides a common filter for addressing this vulnerability, so 
> this filter should be integrated into the YARN web interfaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4308) ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289509#comment-15289509
 ] 

Naganarasimha G R commented on YARN-4308:
-

Thanks for the patch [~sunilg], Patch seems to be simple and straight fwd, just 
on nit , can the comment be a javadoc comment if not the link is of no use .


> ContainersAggregated CPU resource utilization reports negative usage in first 
> few heartbeats
> 
>
> Key: YARN-4308
> URL: https://issues.apache.org/jira/browse/YARN-4308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4308.patch, 0002-YARN-4308.patch, 
> 0003-YARN-4308.patch
>
>
> NodeManager reports ContainerAggregated CPU resource utilization as -ve value 
> in first few heartbeats cycles. I added a new debug print and received below 
> values from heartbeats.
> {noformat}
> INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  ContainersResource Utilization : CpuTrackerUsagePercent : -1.0 
> INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:ContainersResource
>  Utilization :  CpuTrackerUsagePercent : 198.94598
> {noformat}
> Its better we send 0 as CPU usage rather than sending a negative values in 
> heartbeats eventhough its happening in only first few heartbeats.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4832) NM side resource value should get updated if change applied in RM side

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated YARN-4832:
-
Target Version/s: 2.8.0, 2.7.4  (was: 2.8.0)

> NM side resource value should get updated if change applied in RM side
> --
>
> Key: YARN-4832
> URL: https://issues.apache.org/jira/browse/YARN-4832
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: nodemanager, resourcemanager
>Reporter: Junping Du
>Assignee: Junping Du
>Priority: Critical
> Fix For: 2.8.0
>
> Attachments: YARN-4832-addendum.patch, YARN-4832-branch-2.patch, 
> YARN-4832-demo.patch, YARN-4832-v2.patch, YARN-4832-v3.patch, YARN-4832.patch
>
>
> Now, if we execute CLI to update node (single or multiple) resource in RM 
> side, NM will not receive any notification. It doesn't affect resource 
> scheduling but will make resource usage metrics reported by NM a bit weird. 
> We should sync up new resource between RM and NM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-4308) ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-4308:
--
Attachment: 0003-YARN-4308.patch

Updating new patch as per the discussion. Kindly help to check the same.

> ContainersAggregated CPU resource utilization reports negative usage in first 
> few heartbeats
> 
>
> Key: YARN-4308
> URL: https://issues.apache.org/jira/browse/YARN-4308
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager
>Affects Versions: 2.7.1
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-YARN-4308.patch, 0002-YARN-4308.patch, 
> 0003-YARN-4308.patch
>
>
> NodeManager reports ContainerAggregated CPU resource utilization as -ve value 
> in first few heartbeats cycles. I added a new debug print and received below 
> values from heartbeats.
> {noformat}
> INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:
>  ContainersResource Utilization : CpuTrackerUsagePercent : -1.0 
> INFO 
> org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl:ContainersResource
>  Utilization :  CpuTrackerUsagePercent : 198.94598
> {noformat}
> Its better we send 0 as CPU usage rather than sending a negative values in 
> heartbeats eventhough its happening in only first few heartbeats.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289440#comment-15289440
 ] 

Hudson commented on YARN-4925:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #9816 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9816/])
YARN-4925. ContainerRequest in AMRMClient, application should be able to 
(naganarasimha_gr: rev f04c81c9ce93512bc714531a7731debbe6b794ce)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java


> ContainerRequest in AMRMClient, application should be able to specify 
> nodes/racks together with nodeLabelExpression
> ---
>
> Key: YARN-4925
> URL: https://issues.apache.org/jira/browse/YARN-4925
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Fix For: 2.8.0
>
> Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch
>
>
> Currently with nodelabel AMRMClient will not be able to specify nodelabels 
> with Node/Rack requests.For application like spark NODE_LOCAL requests cannot 
> be asked with label expression.
> As per the check in  {{AMRMClientImpl#checkNodeLabelExpression}}
> {noformat}
> // Don't allow specify node label against ANY request
> if ((containerRequest.getRacks() != null && 
> (!containerRequest.getRacks().isEmpty()))
> || 
> (containerRequest.getNodes() != null && 
> (!containerRequest.getNodes().isEmpty( {
>   throw new InvalidContainerRequestException(
>   "Cannot specify node label with rack and node");
> }
> {noformat}
> {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of 
> OFF-SWITCH. 
> The above check is not required for ContainerRequest ask /cc [~wangda] thank 
> you for confirming



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289412#comment-15289412
 ] 

Naganarasimha G R edited comment on YARN-4925 at 5/18/16 5:44 PM:
--

Committed it to branch 2.8, branch 2 & trunk, Thanks for working on this patch 
[~bibinchundatt] and [~sunilg] for the review.


was (Author: naganarasimha):
Committed it to branch 2.8, branch 2 & Thanks for working on this patch 

> ContainerRequest in AMRMClient, application should be able to specify 
> nodes/racks together with nodeLabelExpression
> ---
>
> Key: YARN-4925
> URL: https://issues.apache.org/jira/browse/YARN-4925
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch
>
>
> Currently with nodelabel AMRMClient will not be able to specify nodelabels 
> with Node/Rack requests.For application like spark NODE_LOCAL requests cannot 
> be asked with label expression.
> As per the check in  {{AMRMClientImpl#checkNodeLabelExpression}}
> {noformat}
> // Don't allow specify node label against ANY request
> if ((containerRequest.getRacks() != null && 
> (!containerRequest.getRacks().isEmpty()))
> || 
> (containerRequest.getNodes() != null && 
> (!containerRequest.getNodes().isEmpty( {
>   throw new InvalidContainerRequestException(
>   "Cannot specify node label with rack and node");
> }
> {noformat}
> {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of 
> OFF-SWITCH. 
> The above check is not required for ContainerRequest ask /cc [~wangda] thank 
> you for confirming



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289412#comment-15289412
 ] 

Naganarasimha G R commented on YARN-4925:
-

Committed it to branch 2.8, branch 2 & Thanks for working on this patch 

> ContainerRequest in AMRMClient, application should be able to specify 
> nodes/racks together with nodeLabelExpression
> ---
>
> Key: YARN-4925
> URL: https://issues.apache.org/jira/browse/YARN-4925
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
> Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch
>
>
> Currently with nodelabel AMRMClient will not be able to specify nodelabels 
> with Node/Rack requests.For application like spark NODE_LOCAL requests cannot 
> be asked with label expression.
> As per the check in  {{AMRMClientImpl#checkNodeLabelExpression}}
> {noformat}
> // Don't allow specify node label against ANY request
> if ((containerRequest.getRacks() != null && 
> (!containerRequest.getRacks().isEmpty()))
> || 
> (containerRequest.getNodes() != null && 
> (!containerRequest.getNodes().isEmpty( {
>   throw new InvalidContainerRequestException(
>   "Cannot specify node label with rack and node");
> }
> {noformat}
> {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of 
> OFF-SWITCH. 
> The above check is not required for ContainerRequest ask /cc [~wangda] thank 
> you for confirming



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4913) Yarn logs should take a -out option to write to a directory

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289401#comment-15289401
 ] 

Hudson commented on YARN-4913:
--

SUCCESS: Integrated in Hadoop-trunk-Commit #9815 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/9815/])
YARN-4913. Yarn logs should take a -out option to write to a directory. 
(vvasudev: rev ef1757790d89cc72f88f5330761b1c8901c59e94)
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/cli/TestLogsCLI.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
* 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/logaggregation/LogCLIHelpers.java


> Yarn logs should take a -out option to write to a directory
> ---
>
> Key: YARN-4913
> URL: https://issues.apache.org/jira/browse/YARN-4913
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-4913.1.patch, YARN-4913.2.patch, YARN-4913.3.patch, 
> YARN-4913.4.patch, YARN-4913.5.1.patch, YARN-4913.5.patch, YARN-4913.6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4957) Add getNewReservation in ApplicationClientProtocol

[Carlo Curino (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289393#comment-15289393
 ] 

Carlo Curino commented on YARN-4957:


[~seanpo03], please address the checkstyle/Javadoc issues... and remind us why 
it is ok to commit with tests not passing (e.g., refer to respective jiras to 
fix them). 

Below a list of nits/mostly minor stuff (some of which is just taste or 
questions):

Also I notice in the comments of the API we leak the current implementation by 
saying "The ResourceManager responds with a new, monotonically increasing 
ReservationId"... I would just say is unique. We might want later on to return 
non monotonically increasing reservationIds, and definitely not have people 
rely on this.

Please check comments for small typos/oddities (e.g., "@param request request 
to get a new ReservationId", "to submit an reservation")

Are the classes "@Stable"? the methods are marked "@Unstable"... I am asking as 
I am not sure how we should use stable/unstable at the class vs method level. 

Why YARNClient.createReservation() in stead of 
YARNClient.createNewReservationId()?

Just a question to make sure I follow correctly: The submitReservation is 
idempotent if we submit exactly the same ReservationDefinition, but will reject 
if I try to reuse the reservationId with same Id but a different 
ReservationDefinition. If the user intend to modify prior submission it should 
use update. The error message could be made more informative, but also stating 
the ReservationDefinition is different (consider to update existing 
reservationId or submit with a new reservation)

Are the comments and code of "ReservationSubmissionResponse" consistent?

In the "ReservationSubmissionResponse" should we retain the ReservationId? I 
think might make the implementation on the client easier at times (if multiple 
reservations are submitted asynchronsouly or something like that, it helps 
retain the info of "who got accepted"). For now invocations are synch, but we 
might want to change it later. Thoughts?

You have copy and pasted code in TestYarnClient and TestClientRMService, 
consider (if possible for packaging etc..) to put this in a shared testbase 
class.

What do "list" return if I have obtained a ReservationId, but never used it to 
submit a reservation?

Why in the tests do you use a mock "generateReservationId(int)" instead of 
using the createReservation API? 

In documentation I would not talk to the reader directly (e.g., "With the New 
Reservation API, you can obtain a reservation-id ") use impersonal forms if 
possible (unless the rest of docs are written that way).

> Add getNewReservation in ApplicationClientProtocol
> --
>
> Key: YARN-4957
> URL: https://issues.apache.org/jira/browse/YARN-4957
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: applications, client, resourcemanager
>Affects Versions: 2.8.0
>Reporter: Subru Krishnan
>Assignee: Sean Po
>  Labels: api-breaking
> Attachments: YARN-4957.v0.patch, YARN-4957.v1.patch, 
> YARN-4957.v2.patch, YARN-4957.v3.patch, YARN-4957.v4.patch, 
> YARN-4957.v5.1.patch, YARN-4957.v5.2.patch, YARN-4957.v5.patch
>
>
> Currently submitReservation returns a ReservationId if sucessful. This JIRA 
> propose adding a getNewReservation in ApplicationClientProtocol for the 
> following reasons:
>   * Prevent zombie reservations in the face of client and/or network failures 
> post submitReservation
>   * Align reservation submission with application submission



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-4913) Yarn logs should take a -out option to write to a directory

[Xuan Gong (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuan Gong resolved YARN-4913.
-
   Resolution: Fixed
Fix Version/s: 2.9.0

> Yarn logs should take a -out option to write to a directory
> ---
>
> Key: YARN-4913
> URL: https://issues.apache.org/jira/browse/YARN-4913
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-4913.1.patch, YARN-4913.2.patch, YARN-4913.3.patch, 
> YARN-4913.4.patch, YARN-4913.5.1.patch, YARN-4913.5.patch, YARN-4913.6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4913) Yarn logs should take a -out option to write to a directory

[Xuan Gong (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289382#comment-15289382
 ] 

Xuan Gong commented on YARN-4913:
-

Thanks for the review and commit. Varun.

> Yarn logs should take a -out option to write to a directory
> ---
>
> Key: YARN-4913
> URL: https://issues.apache.org/jira/browse/YARN-4913
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Fix For: 2.9.0
>
> Attachments: YARN-4913.1.patch, YARN-4913.2.patch, YARN-4913.3.patch, 
> YARN-4913.4.patch, YARN-4913.5.1.patch, YARN-4913.5.patch, YARN-4913.6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4913) Yarn logs should take a -out option to write to a directory

[Varun Vasudev (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289376#comment-15289376
 ] 

Varun Vasudev commented on YARN-4913:
-

Committed to trunk and branch-2. Thanks [~xgong]!

> Yarn logs should take a -out option to write to a directory
> ---
>
> Key: YARN-4913
> URL: https://issues.apache.org/jira/browse/YARN-4913
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-4913.1.patch, YARN-4913.2.patch, YARN-4913.3.patch, 
> YARN-4913.4.patch, YARN-4913.5.1.patch, YARN-4913.5.patch, YARN-4913.6.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-4002) make ResourceTrackerService.nodeHeartbeat more concurrent

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289372#comment-15289372
 ] 

Jian He commented on YARN-4002:
---

ok, let's keep it, I think the lock can be removed ?

> make ResourceTrackerService.nodeHeartbeat more concurrent
> -
>
> Key: YARN-4002
> URL: https://issues.apache.org/jira/browse/YARN-4002
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: Hong Zhiguo
>Assignee: Hong Zhiguo
>Priority: Critical
> Attachments: 0001-YARN-4002.patch, YARN-4002-lockless-read.patch, 
> YARN-4002-rwlock-v2.patch, YARN-4002-rwlock-v2.patch, 
> YARN-4002-rwlock-v3-rebase.patch, YARN-4002-rwlock-v3.patch, 
> YARN-4002-rwlock-v4.patch, YARN-4002-rwlock-v5.patch, YARN-4002-rwlock.patch, 
> YARN-4002-v0.patch
>
>
> We have multiple RPC threads to handle NodeHeartbeatRequest from NMs. By 
> design the method ResourceTrackerService.nodeHeartbeat should be concurrent 
> enough to scale for large clusters.
> But we have a "BIG" lock in NodesListManager.isValidNode which I think it's 
> unnecessary.
> First, the fields "includes" and "excludes" of HostsFileReader are only 
> updated on "refresh nodes".  All RPC threads handling node heartbeats are 
> only readers.  So RWLock could be used to  alow concurrent access by RPC 
> threads.
> Second, since he fields "includes" and "excludes" of HostsFileReader are 
> always updated by "reference assignment", which is atomic in Java, the reader 
> side lock could just be skipped.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection

[Jonathan Maron (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289362#comment-15289362
 ] 

Jonathan Maron commented on YARN-5076:
--

[~djp] - if you can confirm that the settings [~vvasudev] lists above are a 
reflection of the way you'd like to see the configuration I can make the change 
- I have no real strong objection.  I can look into making the default 
SAMEORIGIN.

> YARN web interfaces lack XFS protection
> ---
>
> Key: YARN-5076
> URL: https://issues.apache.org/jira/browse/YARN-5076
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager, timelineserver
>Reporter: Jonathan Maron
>Assignee: Jonathan Maron
> Attachments: YARN-5076.002.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are web interfaces in YARN that do not provide protection against cross 
> frame scripting 
> (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet).  
> HADOOP-13008 provides a common filter for addressing this vulnerability, so 
> this filter should be integrated into the YARN web interfaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5076) YARN web interfaces lack XFS protection

[Varun Vasudev (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15289269#comment-15289269
 ] 

Varun Vasudev commented on YARN-5076:
-

Ah I think I misunderstood what you were saying. If I understand you correctly, 
you want the following the configs -

# yarn.webapp.xfs-filter.enabled
# yarn.resourcemanager.webapp.xframe-options
# yarn.nodemanager.webapp.xframe-options
# yarn.timeline-service.webapp.xframe-options

Correct?

> YARN web interfaces lack XFS protection
> ---
>
> Key: YARN-5076
> URL: https://issues.apache.org/jira/browse/YARN-5076
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: nodemanager, resourcemanager, timelineserver
>Reporter: Jonathan Maron
>Assignee: Jonathan Maron
> Attachments: YARN-5076.002.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> There are web interfaces in YARN that do not provide protection against cross 
> frame scripting 
> (https://www.owasp.org/index.php/Clickjacking_Defense_Cheat_Sheet).  
> HADOOP-13008 provides a common filter for addressing this vulnerability, so 
> this filter should be integrated into the YARN web interfaces.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org