[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[Jian He (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989772#comment-15989772
 ] 

Jian He commented on YARN-1197:
---

[~mingma], I don't think fair scheduler support this as of now.
It'll be great if framework like Reef can start using it.

> Support changing resources of an allocated container
> 
>
> Key: YARN-1197
> URL: https://issues.apache.org/jira/browse/YARN-1197
> Project: Hadoop YARN
>  Issue Type: Task
>  Components: api, graceful, nodemanager, resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Wangda Tan
> Attachments: YARN-1197_Design.2015.06.24.pdf, 
> YARN-1197_Design.2015.07.07.pdf, YARN-1197_Design.2015.08.21.pdf, 
> YARN-1197_Design.pdf, YARN-1197 old-design-docs-patches-for-reference.zip
>
>
> The current YARN resource management logic assumes resource allocated to a 
> container is fixed during the lifetime of it. When users want to change a 
> resource 
> of an allocated container the only way is releasing it and allocating a new 
> container with expected size.
> Allowing run-time changing resources of an allocated container will give us 
> better control of resource usage in application side



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989751#comment-15989751
 ] 

Hadoop QA commented on YARN-5331:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
14s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
4s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 in trunk has 8 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
25s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 42 unchanged - 1 fixed = 42 total (was 43) {color} 
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red}  0m 
19s{color} | {color:red} 
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager
 generated 1 new + 879 unchanged - 1 fixed = 880 total (was 880) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 39m  
3s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
19s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m  7s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-5331 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12865642/YARN-5331.009.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux e3b0d21800e7 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / 19a7e94 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs | 
https://builds.apache.org/job/PreCommit-YARN-Build/15778/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
 |
| javadoc | 
https://builds.apache.org/job/PreCommit-YARN-Build/15778/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/15778/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[Tobin Baker (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989685#comment-15989685
 ] 

Tobin Baker commented on YARN-1197:
---

I believe [Apache REEF|http://reef.apache.org/] is interested in this feature.

> Support changing resources of an allocated container
> 
>
> Key: YARN-1197
> URL: https://issues.apache.org/jira/browse/YARN-1197
> Project: Hadoop YARN
>  Issue Type: Task
>  Components: api, graceful, nodemanager, resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Wangda Tan
> Attachments: YARN-1197_Design.2015.06.24.pdf, 
> YARN-1197_Design.2015.07.07.pdf, YARN-1197_Design.2015.08.21.pdf, 
> YARN-1197_Design.pdf, YARN-1197 old-design-docs-patches-for-reference.zip
>
>
> The current YARN resource management logic assumes resource allocated to a 
> container is fixed during the lifetime of it. When users want to change a 
> resource 
> of an allocated container the only way is releasing it and allocating a new 
> container with expected size.
> Allowing run-time changing resources of an allocated container will give us 
> better control of resource usage in application side



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-1197) Support changing resources of an allocated container

[Ming Ma (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-1197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989682#comment-15989682
 ] 

Ming Ma commented on YARN-1197:
---

Thanks for the feature! Is the fair scheduler support available? Also wonder if 
any framework plans to use the feature.

> Support changing resources of an allocated container
> 
>
> Key: YARN-1197
> URL: https://issues.apache.org/jira/browse/YARN-1197
> Project: Hadoop YARN
>  Issue Type: Task
>  Components: api, graceful, nodemanager, resourcemanager
>Affects Versions: 2.1.0-beta
>Reporter: Wangda Tan
> Attachments: YARN-1197_Design.2015.06.24.pdf, 
> YARN-1197_Design.2015.07.07.pdf, YARN-1197_Design.2015.08.21.pdf, 
> YARN-1197_Design.pdf, YARN-1197 old-design-docs-patches-for-reference.zip
>
>
> The current YARN resource management logic assumes resource allocated to a 
> container is fixed during the lifetime of it. When users want to change a 
> resource 
> of an allocated container the only way is releasing it and allocating a new 
> container with expected size.
> Allowing run-time changing resources of an allocated container will give us 
> better control of resource usage in application side



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6538) Inter Queue preemption is not happening when DRF is configured

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-6538:
--
Description: 
Cluster capacity of . Here memory is more and vcores 
are less. If applications have more demand, vcores might be exhausted. 
Inter queue preemption ideally has to be kicked in once vcores is over 
utilized. However preemption is not happening.

Analysis:
In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, 
{code}
// assign all cluster resources until no more demand, or no resources are
// left
while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant,
unassigned, Resources.none())) {
{code}
 will loop even when vcores are 0 (because memory is still +ve). Hence we are 
having more vcores in idealAssigned which cause no-preemption cases.

  was:
Cluster capacity of . Here memory is more and vcores 
are less. If applications have more demand, vcores might be exhausted. 
Inter queue preemption ideally has to be kicked in once vcores is over 
utilized. However preemption is not happening.

Analysis:
In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, 
{code}
// assign all cluster resources until no more demand, or no resources are
// left
while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant,
unassigned, Resources.none())) {
{code}
 will loop even when vcores are 0. Hence we are having more vcores in 
idealAssigned which cause no-preemption cases.


> Inter Queue preemption is not happening when DRF is configured
> --
>
> Key: YARN-6538
> URL: https://issues.apache.org/jira/browse/YARN-6538
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>
> Cluster capacity of . Here memory is more and vcores 
> are less. If applications have more demand, vcores might be exhausted. 
> Inter queue preemption ideally has to be kicked in once vcores is over 
> utilized. However preemption is not happening.
> Analysis:
> In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, 
> {code}
> // assign all cluster resources until no more demand, or no resources are
> // left
> while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant,
> unassigned, Resources.none())) {
> {code}
>  will loop even when vcores are 0 (because memory is still +ve). Hence we are 
> having more vcores in idealAssigned which cause no-preemption cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem

[Sangeetha Abdu Jyothi (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989605#comment-15989605
 ] 

Sangeetha Abdu Jyothi commented on YARN-5331:
-

Thanks for the comments, [~subru]. 

1) I have addressed this in the latest patch.

2) getMaxPeriodicCapacity and getMinimumCapacityInInterval perform different 
functions. getMaxPeriodicCapacity finds the maximum at periodic offsets without 
considering the values in the interval between these time instants. On the 
other hand, getMinimumCapacityInInterval returns the minimum in the given 
interval. Hence I have renamed the function to getMaximumPeriodicCapacity.

3) Since this will lead to changes in the remaining JIRAs under YARN-5326, it 
might be helpful to make this a separate JIRA.  


> Extend RLESparseResourceAllocation with period for supporting recurring 
> reservations in YARN ReservationSystem
> --
>
> Key: YARN-5331
> URL: https://issues.apache.org/jira/browse/YARN-5331
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Sangeetha Abdu Jyothi
>  Labels: oct16-medium
> Attachments: YARN-5331.001.patch, YARN-5331.002.patch, 
> YARN-5331.003.patch, YARN-5331.004.patch, YARN-5331.005.patch, 
> YARN-5331.006.patch, YARN-5331.007.patch, YARN-5331.008.patch, 
> YARN-5331.009.patch
>
>
> YARN-5326 proposes adding native support for recurring reservations in the 
> YARN ReservationSystem. This JIRA is a sub-task to add a 
> PeriodicRLESparseResourceAllocation. Please refer to the design doc in the 
> parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem

[Sangeetha Abdu Jyothi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sangeetha Abdu Jyothi updated YARN-5331:

Attachment: YARN-5331.009.patch

> Extend RLESparseResourceAllocation with period for supporting recurring 
> reservations in YARN ReservationSystem
> --
>
> Key: YARN-5331
> URL: https://issues.apache.org/jira/browse/YARN-5331
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Sangeetha Abdu Jyothi
>  Labels: oct16-medium
> Attachments: YARN-5331.001.patch, YARN-5331.002.patch, 
> YARN-5331.003.patch, YARN-5331.004.patch, YARN-5331.005.patch, 
> YARN-5331.006.patch, YARN-5331.007.patch, YARN-5331.008.patch, 
> YARN-5331.009.patch
>
>
> YARN-5326 proposes adding native support for recurring reservations in the 
> YARN ReservationSystem. This JIRA is a sub-task to add a 
> PeriodicRLESparseResourceAllocation. Please refer to the design doc in the 
> parent JIRA for details.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989599#comment-15989599
 ] 

Wangda Tan commented on YARN-2113:
--

[~eepayne]/[~sunilg] here's my understanding of the two policies: 

1) Priority first:
App1 can preempt App2 if:

When App1.priority > App2.priority
- User of app1 not reached user-limit

When App1.priority = App2.priority
- app1.user != app2.user
- User of app1 not reached user-limit 
- User of app2 beyond user-limit.

2) User limit first:
App1 can preempt app2 if:
- app1.user != app2.user
- User of app1 not reached user-limit 
- User of app2 beyond user-limit.

For both:
- No matter if {{app1.user}} is outside of {{#100/MULP}} or not, no change to 
preemption behavior. (To the example in 
https://issues.apache.org/jira/browse/YARN-2113?focusedCommentId=15951222=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15951222,
 if policy is PRIORITY_FIRST, preemption is allowed but if policy is 
USERLIMIT_FIRST, preemption is not allowed).

If I understand Jason's comment correctly: 
https://issues.apache.org/jira/browse/YARN-2113?focusedCommentId=15951538=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15951538,
 above behavior should support both scenarios via different policies: 
USERLIMIT_FIRST don't allow any preemption to an user who is below user limit; 
PRIORITY_FIRST allows preemption happen if one app has higher priority than 
another.

> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sunil G
> Attachments: IntraQueue Preemption-Impact Analysis.pdf, 
> TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
>  YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, 
> YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch, 
> YARN-2113.0007.patch, YARN-2113.0008.patch, YARN-2113.0009.patch, 
> YARN-2113.0010.patch, YARN-2113.0011.patch, YARN-2113.0012.patch, 
> YARN-2113.0013.patch, YARN-2113.apply.onto.0012.ericp.patch, 
> YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989540#comment-15989540
 ] 

Daniel Templeton commented on YARN-2962:


Since it's a compatible change that doesn't mess with existing properties, I 
went ahead and pulled it into branch-2.  I'm running on not a lot of sleep 
right now, though, so I'd feel better if someone double-checked me on the 
sensibility of that. :)

> ZKRMStateStore: Limit the number of znodes under a znode
> 
>
> Key: YARN-2962
> URL: https://issues.apache.org/jira/browse/YARN-2962
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-2962.006.patch, YARN-2962.007.patch, 
> YARN-2962.008.patch, YARN-2962.008.patch, YARN-2962.009.patch, 
> YARN-2962.010.patch, YARN-2962.011.patch, YARN-2962.01.patch, 
> YARN-2962.04.patch, YARN-2962.05.patch, YARN-2962.2.patch, YARN-2962.3.patch
>
>
> We ran into this issue where we were hitting the default ZK server message 
> size configs, primarily because the message had too many znodes even though 
> they individually they were all small.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode

[Daniel Templeton (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Templeton updated YARN-2962:
---
Target Version/s: 2.9.0, 3.0.0-alpha3  (was: 3.0.0-alpha3)

> ZKRMStateStore: Limit the number of znodes under a znode
> 
>
> Key: YARN-2962
> URL: https://issues.apache.org/jira/browse/YARN-2962
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Varun Saxena
>Priority: Critical
> Fix For: 2.9.0, 3.0.0-alpha3
>
> Attachments: YARN-2962.006.patch, YARN-2962.007.patch, 
> YARN-2962.008.patch, YARN-2962.008.patch, YARN-2962.009.patch, 
> YARN-2962.010.patch, YARN-2962.011.patch, YARN-2962.01.patch, 
> YARN-2962.04.patch, YARN-2962.05.patch, YARN-2962.2.patch, YARN-2962.3.patch
>
>
> We ran into this issue where we were hitting the default ZK server message 
> size configs, primarily because the message had too many znodes even though 
> they individually they were all small.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6375) App level aggregation should not consider metric values reported in the previous aggregation cycle

[Vrushali C (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989521#comment-15989521
 ] 

Vrushali C commented on YARN-6375:
--

Hi [~varun_saxena]

I noticed some checkstyle issues in the new test code in the patch. Will it be 
okay for you to fix those? 

{code}
[ERROR] 
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[218]
 (indentation) Indentation: 'method def modifier' have incorrect indentation 
level 9, expected level should be one of the following: 6, 8, 10.
[ERROR] 
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[220]
 (indentation) Indentation: 'method def' child have incorrect indentation level 
11, expected level should be one of the following: 8, 10, 12.
[ERROR] 
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[222]
 (indentation) Indentation: 'method def rcurly' have incorrect indentation 
level 9, expected level should be one of the following: 6, 8, 10.
[ERROR] 
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[229,19]
 (whitespace) NoWhitespaceBefore: ';' is preceded with whitespace.
[ERROR] 
src/test/java/org/apache/hadoop/yarn/server/timelineservice/collector/TestTimelineCollector.java:[263,19]
 (whitespace) NoWhitespaceBefore: ';' is preceded with whitespace.
{code}

thanks
Vrushali

> App level aggregation should not consider metric values reported in the 
> previous aggregation cycle
> --
>
> Key: YARN-6375
> URL: https://issues.apache.org/jira/browse/YARN-6375
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: timelineserver
>Reporter: Varun Saxena
>Assignee: Varun Saxena
> Attachments: YARN-6375-YARN-5355.01.patch
>
>
> Currently app level aggregation is done every 15 seconds.
> And we consider last reported metric value for each entity belonging to an 
> app for aggregation.
> We however merely update the corresponding metric values for the entity on 
> put. We never remove the entries.
> But it is possible that multiple entities finish during lifetime of an 
> application. We however continue to consider them till the end.
> We should however not consider metric values of entities unless reported 
> within the 15 second period.
> Consider containers. For a long running app, several containers would start 
> and end at various times during the lifetime of an app.
> To consider metrics for all the containers throughout the lifetime of app, 
> hence wont be correct.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated YARN-6536:
-
Fix Version/s: (was: 2.8.1)
   2.8.2

> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
>Assignee: Jason Lowe
> Fix For: 2.9.0, 3.0.0-alpha3, 2.8.2
>
> Attachments: YARN-6536.001.patch
>
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989458#comment-15989458
 ] 

Hudson commented on YARN-2962:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11652 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11652/])
YARN-2962. ZKRMStateStore: Limit the number of znodes under a znode (templedf: 
rev 2e52789edf68016e7a3f450164f8bd3d8e6cb210)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/RMStateStoreTestBase.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/TestZKRMStateStore.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMStoreCommands.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java


> ZKRMStateStore: Limit the number of znodes under a znode
> 
>
> Key: YARN-2962
> URL: https://issues.apache.org/jira/browse/YARN-2962
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-2962.006.patch, YARN-2962.007.patch, 
> YARN-2962.008.patch, YARN-2962.008.patch, YARN-2962.009.patch, 
> YARN-2962.010.patch, YARN-2962.011.patch, YARN-2962.01.patch, 
> YARN-2962.04.patch, YARN-2962.05.patch, YARN-2962.2.patch, YARN-2962.3.patch
>
>
> We ran into this issue where we were hitting the default ZK server message 
> size configs, primarily because the message had too many znodes even though 
> they individually they were all small.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989438#comment-15989438
 ] 

Hudson commented on YARN-6536:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11651 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11651/])
YARN-6536. TestAMRMClient.testAMRMClientWithSaslEncryption fails (epayne: rev 
fdf51923c81e5fb221758297605271139dc9)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java


> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
>Assignee: Jason Lowe
> Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3
>
> Attachments: YARN-6536.001.patch
>
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2962) ZKRMStateStore: Limit the number of znodes under a znode

[Daniel Templeton (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989426#comment-15989426
 ] 

Daniel Templeton commented on YARN-2962:


+1 on the latest patch.

> ZKRMStateStore: Limit the number of znodes under a znode
> 
>
> Key: YARN-2962
> URL: https://issues.apache.org/jira/browse/YARN-2962
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: resourcemanager
>Affects Versions: 2.6.0
>Reporter: Karthik Kambatla
>Assignee: Varun Saxena
>Priority: Critical
> Attachments: YARN-2962.006.patch, YARN-2962.007.patch, 
> YARN-2962.008.patch, YARN-2962.008.patch, YARN-2962.009.patch, 
> YARN-2962.010.patch, YARN-2962.011.patch, YARN-2962.01.patch, 
> YARN-2962.04.patch, YARN-2962.05.patch, YARN-2962.2.patch, YARN-2962.3.patch
>
>
> We ran into this issue where we were hitting the default ZK server message 
> size configs, primarily because the message had too many znodes even though 
> they individually they were all small.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989401#comment-15989401
 ] 

Eric Payne commented on YARN-6536:
--

Thanks [~jlowe]
+1

> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
>Assignee: Jason Lowe
> Attachments: YARN-6536.001.patch
>
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6457) Allow custom SSL configuration to be supplied in WebApps

[Sanjay M Pujare (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989356#comment-15989356
 ] 

Sanjay M Pujare commented on YARN-6457:
---

[~haibochen] thinking more about it, I can see how this can be made to work. I 
am thinking the following code in WebAppUtils.loadSslConfiguration(Builder, 
Configuration) will do the trick:

if (sslConf == null) {
  sslConf = new Configuration(false);
  sslConf.addResource(YarnConfiguration.YARN_SSL_SERVER_RESOURCE_DEFAULT);
} else {
  String customSslServerConf = sslConf.get("hadoop.ssl.server.conf");  // 
TODO define the string in YarnConfigruation or elsewhere
  if (customSslServerConf != null) {
  sslConf.addResource(new Path(customSslServerConf));
  } else {
  
sslConf.addResource(YarnConfiguration.YARN_SSL_SERVER_RESOURCE_DEFAULT);
  }
}

Let me know if this is what you had in mind.

> Allow custom SSL configuration to be supplied in WebApps
> 
>
> Key: YARN-6457
> URL: https://issues.apache.org/jira/browse/YARN-6457
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: webapp, yarn
>Reporter: Sanjay M Pujare
>Assignee: Sanjay M Pujare
>   Original Estimate: 96h
>  Remaining Estimate: 96h
>
> Currently a custom SSL store cannot be passed on to WebApps which forces the 
> embedded web-server to use the default keystore set up in ssl-server.xml for 
> the whole Hadoop cluster. There are cases where the Hadoop app needs to use 
> its own/custom keystore.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6522) Make SLS JSON input file format simple and scalable

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6522:
---
Description: 
SLS input format is verbose, and it doesn't scale out. We can improve it in 
these ways:
# We need to configure tasks one by one if there are more than one task in a 
job, which means the job configuration usually includes lots of redundant 
items. To specify the number of task for task configuration will solve this 
issue.
# Container host is useful for locality testing. It is obnoxious to specify 
container host for each task for tests unrelated to locality. We would like to 
make it optional.
# For most tests, we don't care about job.id. Make it optional and generated 
automatically by default.
# job.finish.ms doesn't make sense, just remove it.
# container type and container priority should be optional as well. 

  was:
SLS input format is verbose, and it doesn't scale out. We can improve it in 
these ways:
# We need to configure tasks one by one if there are more than one task in a 
job, which means the job configuration usually includes lots of redundant 
items. Specify the number task for a task configuration will solve this issue.
# Container host is useful for locality testing. It is obnoxious to specify 
container host for each task for tests unrelated to locality. We would like to 
make it optional.
# For most tests, we don't care about job.id. Make it optional and generated 
automatically by default.
# job.finish.ms doesn't make sense, just remove it.
# container type and container priority should be optional as well. 


> Make SLS JSON input file format simple and scalable
> ---
>
> Key: YARN-6522
> URL: https://issues.apache.org/jira/browse/YARN-6522
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler-load-simulator
>Affects Versions: 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6522.001.patch
>
>
> SLS input format is verbose, and it doesn't scale out. We can improve it in 
> these ways:
> # We need to configure tasks one by one if there are more than one task in a 
> job, which means the job configuration usually includes lots of redundant 
> items. To specify the number of task for task configuration will solve this 
> issue.
> # Container host is useful for locality testing. It is obnoxious to specify 
> container host for each task for tests unrelated to locality. We would like 
> to make it optional.
> # For most tests, we don't care about job.id. Make it optional and generated 
> automatically by default.
> # job.finish.ms doesn't make sense, just remove it.
> # container type and container priority should be optional as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6522) Make SLS JSON input file format simple and scalable

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989320#comment-15989320
 ] 

Yufei Gu edited comment on YARN-6522 at 4/28/17 7:20 PM:
-

Uploaded patch v1. It largely simplifies the job configuration and be 
compatible with the existing format.
This is a job configuration for FS minshare preemption test. Here is what it 
looks like before patch:
{code}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 6,
"job.queue.name" : "default",
"job.id" : "job_1",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 6,
"job.queue.name" : "default",
"job.id" : "job_11",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : 

[jira] [Comment Edited] (YARN-6522) Make SLS JSON input file format simple and scalable

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989320#comment-15989320
 ] 

Yufei Gu edited comment on YARN-6522 at 4/28/17 7:17 PM:
-

Uploaded patch v1. It largely simplifies the job configuration. 
Here is a job configuration for FS minshare preemption test. What it looks like 
before patch:
{code}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 6,
"job.queue.name" : "default",
"job.id" : "job_1",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 6,
"job.queue.name" : "default",
"job.id" : "job_11",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,

[jira] [Commented] (YARN-6522) Make SLS JSON input file format simple and scalable

[Yufei Gu (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989320#comment-15989320
 ] 

Yufei Gu commented on YARN-6522:


Uploaded patch v1. It largely simplifies the job configuration. 
Here is a job configuration for minshare preemption test. What it looks like 
before patch:
{code}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 6,
"job.queue.name" : "default",
"job.id" : "job_1",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
} ]
}
{
"am.type" : "mapreduce",
"job.start.ms" : 0,
"job.end.ms" : 6,
"job.queue.name" : "default",
"job.id" : "job_11",
"job.user" : "default",
"job.tasks" : [ {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6664,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node2",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node3",
"container.start.ms" : 6665,
"container.end.ms" : 6,
"container.priority" : 20,
"container.type" : "map"
}, {
"container.host" : "/default-rack/node1",
"container.start.ms" : 6665,
"container.end.ms" : 6,

[jira] [Updated] (YARN-6522) Make SLS JSON input file format simple and scalable

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6522:
---
Affects Version/s: 3.0.0-alpha2

> Make SLS JSON input file format simple and scalable
> ---
>
> Key: YARN-6522
> URL: https://issues.apache.org/jira/browse/YARN-6522
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler-load-simulator
>Affects Versions: 3.0.0-alpha2
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6522.001.patch
>
>
> SLS input format is verbose, and it doesn't scale out. We can improve it in 
> these ways:
> # We need to configure tasks one by one if there are more than one task in a 
> job, which means the job configuration usually includes lots of redundant 
> items. Specify the number task for a task configuration will solve this issue.
> # Container host is useful for locality testing. It is obnoxious to specify 
> container host for each task for tests unrelated to locality. We would like 
> to make it optional.
> # For most tests, we don't care about job.id. Make it optional and generated 
> automatically by default.
> # job.finish.ms doesn't make sense, just remove it.
> # container type and container priority should be optional as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6522) Make SLS JSON input file format simple and scalable

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6522:
---
Attachment: YARN-6522.001.patch

> Make SLS JSON input file format simple and scalable
> ---
>
> Key: YARN-6522
> URL: https://issues.apache.org/jira/browse/YARN-6522
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler-load-simulator
>Reporter: Yufei Gu
>Assignee: Yufei Gu
> Attachments: YARN-6522.001.patch
>
>
> SLS input format is verbose, and it doesn't scale out. We can improve it in 
> these ways:
> # We need to configure tasks one by one if there are more than one task in a 
> job, which means the job configuration usually includes lots of redundant 
> items. Specify the number task for a task configuration will solve this issue.
> # Container host is useful for locality testing. It is obnoxious to specify 
> container host for each task for tests unrelated to locality. We would like 
> to make it optional.
> # For most tests, we don't care about job.id. Make it optional and generated 
> automatically by default.
> # job.finish.ms doesn't make sense, just remove it.
> # container type and container priority should be optional as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6522) Make SLS JSON input file format simple and scalable

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6522:
---
Summary: Make SLS JSON input file format simple and scalable  (was: Make 
SLS JSON input file format simpler and more powerful)

> Make SLS JSON input file format simple and scalable
> ---
>
> Key: YARN-6522
> URL: https://issues.apache.org/jira/browse/YARN-6522
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler-load-simulator
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> SLS input format is verbose, and it doesn't scale out. We can improve it in 
> these ways:
> # We need to configure tasks one by one if there are more than one task in a 
> job, which means the job configuration usually includes lots of redundant 
> items. Specify the number task for a task configuration will solve this issue.
> # Container host is useful for locality testing. It is obnoxious to specify 
> container host for each task for tests unrelated to locality. We would like 
> to make it optional.
> # For most tests, we don't care about job.id. Make it optional and generated 
> automatically by default.
> # job.finish.ms doesn't make sense, just remove it.
> # container type and container priority should be optional as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6522) Make SLS JSON input file format simpler and more powerful

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6522:
---
Description: 
SLS input format is verbose, and it doesn't scale out. We can improve it in 
these ways:
# We need to configure tasks one by one if there are more than one task in a 
job, which means the job configuration usually includes lots of redundant 
items. Specify the number task for a task configuration will solve this issue.
# Container host is useful for locality testing. It is obnoxious to specify 
container host for each task for tests unrelated to locality. We would like to 
make it optional.
# For most tests, we don't care about job.id. Make it optional and generated 
automatically by default.
# job.finish.ms doesn't make sense, just remove it.
# container type and container priority should be optional as well. 

  was:Container host is useful for locality testing. It is obnoxious to specify 
container host for each task for tests unrelated to locality. We would like to 
make it optional.


> Make SLS JSON input file format simpler and more powerful
> -
>
> Key: YARN-6522
> URL: https://issues.apache.org/jira/browse/YARN-6522
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler-load-simulator
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> SLS input format is verbose, and it doesn't scale out. We can improve it in 
> these ways:
> # We need to configure tasks one by one if there are more than one task in a 
> job, which means the job configuration usually includes lots of redundant 
> items. Specify the number task for a task configuration will solve this issue.
> # Container host is useful for locality testing. It is obnoxious to specify 
> container host for each task for tests unrelated to locality. We would like 
> to make it optional.
> # For most tests, we don't care about job.id. Make it optional and generated 
> automatically by default.
> # job.finish.ms doesn't make sense, just remove it.
> # container type and container priority should be optional as well. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6522) Make SLS JSON input file format simpler and more powerful

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6522:
---
Summary: Make SLS JSON input file format simpler and more powerful  (was: 
Container host should be optional in SLS JSON input file format)

> Make SLS JSON input file format simpler and more powerful
> -
>
> Key: YARN-6522
> URL: https://issues.apache.org/jira/browse/YARN-6522
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler-load-simulator
>Reporter: Yufei Gu
>Assignee: Yufei Gu
>
> Container host is useful for locality testing. It is obnoxious to specify 
> container host for each task for tests unrelated to locality. We would like 
> to make it optional.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6472) Improve Java sandbox regex

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989282#comment-15989282
 ] 

Hudson commented on YARN-6472:
--

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11649 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11649/])
YARN-6472. Improve Java sandbox regex (gphillips via rkanter) (rkanter: rev 
68e45f554b6cf7d56fca6bbf6e89dc4f55fdc716)
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/JavaSandboxLinuxContainerRuntime.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java
* (edit) 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java


> Improve Java sandbox regex
> --
>
> Key: YARN-6472
> URL: https://issues.apache.org/jira/browse/YARN-6472
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Greg Phillips
> Fix For: 3.0.0-alpha3
>
> Attachments: YARN-6472.001.patch, YARN-6472.002.patch, 
> YARN-6472.003.patch
>
>
> I set the sandbox to enforcing mode. Unfortunately I was able to break out of 
> the sandbox running native code with the following command:
> {code}
> cmd = "$JAVA_HOME/bin/java %s -Xmx825955249 
> org.apache.hadoop.yarn.applications.helloworld.HelloWorld `touch 
> ../../helloworld`" + \
>   " 1>/AppMaster.stdout 2>/AppMaster.stderr"
> $ ls .../nm-local-dir/usercache/root/appcache/
> helloworld
> {code}
> Also, if I am not using sandboxes, could we create the nm-sandbox-policies 
> directory (empty) lazily?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6538) Inter Queue preemption is not happening when DRF is configured

[Yufei Gu (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yufei Gu updated YARN-6538:
---
Component/s: capacity scheduler

> Inter Queue preemption is not happening when DRF is configured
> --
>
> Key: YARN-6538
> URL: https://issues.apache.org/jira/browse/YARN-6538
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacity scheduler, scheduler preemption
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>
> Cluster capacity of . Here memory is more and vcores 
> are less. If applications have more demand, vcores might be exhausted. 
> Inter queue preemption ideally has to be kicked in once vcores is over 
> utilized. However preemption is not happening.
> Analysis:
> In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, 
> {code}
> // assign all cluster resources until no more demand, or no resources are
> // left
> while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant,
> unassigned, Resources.none())) {
> {code}
>  will loop even when vcores are 0. Hence we are having more vcores in 
> idealAssigned which cause no-preemption cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6538) Inter Queue preemption is not happening when DRF is configured

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989261#comment-15989261
 ] 

Sunil G commented on YARN-6538:
---

cc/ [~leftnoteasy]

> Inter Queue preemption is not happening when DRF is configured
> --
>
> Key: YARN-6538
> URL: https://issues.apache.org/jira/browse/YARN-6538
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: scheduler preemption
>Affects Versions: 2.8.0
>Reporter: Sunil G
>Assignee: Sunil G
>
> Cluster capacity of . Here memory is more and vcores 
> are less. If applications have more demand, vcores might be exhausted. 
> Inter queue preemption ideally has to be kicked in once vcores is over 
> utilized. However preemption is not happening.
> Analysis:
> In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, 
> {code}
> // assign all cluster resources until no more demand, or no resources are
> // left
> while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant,
> unassigned, Resources.none())) {
> {code}
>  will loop even when vcores are 0. Hence we are having more vcores in 
> idealAssigned which cause no-preemption cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Created] (YARN-6538) Inter Queue preemption is not happening when DRF is configured

[Sunil G (JIRA)]

Sunil G created YARN-6538:
-

 Summary: Inter Queue preemption is not happening when DRF is 
configured
 Key: YARN-6538
 URL: https://issues.apache.org/jira/browse/YARN-6538
 Project: Hadoop YARN
  Issue Type: Bug
  Components: scheduler preemption
Affects Versions: 2.8.0
Reporter: Sunil G
Assignee: Sunil G


Cluster capacity of . Here memory is more and vcores 
are less. If applications have more demand, vcores might be exhausted. 
Inter queue preemption ideally has to be kicked in once vcores is over 
utilized. However preemption is not happening.

Analysis:
In {{AbstractPreemptableResourceCalculator.computeFixpointAllocation}}, 
{code}
// assign all cluster resources until no more demand, or no resources are
// left
while (!orderedByNeed.isEmpty() && Resources.greaterThan(rc, totGuarant,
unassigned, Resources.none())) {
{code}
 will loop even when vcores are 0. Hence we are having more vcores in 
idealAssigned which cause no-preemption cases.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-679) add an entry point that can start any Yarn service

[Hudson (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989255#comment-15989255
 ] 

Hudson commented on YARN-679:
-

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11648 (See 
[https://builds.apache.org/job/Hadoop-trunk-Commit/11648/])
YARN-679. Add an entry point that can start any Yarn service. (junping_du: rev 
373bb4931fb392e3ca6bfd78992887e5a405e186)
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLaunchException.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceLauncher.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/StoppingInStartLaunchableService.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/StringConstructorOnlyService.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/AbstractLaunchableService.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceShutdownHook.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/LauncherArguments.java
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/BreakableService.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitUtil.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/IrqHandler.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ExitCodeProvider.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/RunningService.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceInterruptHandling.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/FailInInitService.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/InitInConstructorLaunchableService.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/ExceptionInExecuteLaunchableService.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/LaunchableRunningService.java
* (edit) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/test/GenericTestUtils.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/NoArgsAllowedService.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/NullBindLaunchableService.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/StringUtils.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/FailingStopInStartService.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/LauncherExitCodes.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/FailInStartService.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/ServiceLauncher.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceConf.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/GenericOptionsParser.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/InterruptEscalator.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/testservices/FailInConstructorService.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/ExitTrackingServiceLauncher.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/HadoopUncaughtExceptionHandler.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceLauncherInnerMethods.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/TestServiceLauncherCreationFailures.java
* (edit) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceStateException.java
* (add) 
hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/service/launcher/AbstractServiceLauncherTestBase.java
* (add) 
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/launcher/LaunchableService.java
* (add) 

[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue

[Sunil G (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989244#comment-15989244
 ] 

Sunil G commented on YARN-2113:
---

I guess there is some mis understanding. I ll pull out 
USERLIMIT_FIRST/PRIORITY_FIRST checks first. So it ll be helpful to bring out 
which all scenarios we will be covering when we configure  USERLIMIT_FIRST or 
PRIORITY_FIRST. And we could handle in other patch if needed post that 
discussion.

> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sunil G
> Attachments: IntraQueue Preemption-Impact Analysis.pdf, 
> TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
>  YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, 
> YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch, 
> YARN-2113.0007.patch, YARN-2113.0008.patch, YARN-2113.0009.patch, 
> YARN-2113.0010.patch, YARN-2113.0011.patch, YARN-2113.0012.patch, 
> YARN-2113.0013.patch, YARN-2113.apply.onto.0012.ericp.patch, 
> YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6472) Improve Java sandbox regex

[Robert Kanter (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989238#comment-15989238
 ] 

Robert Kanter commented on YARN-6472:
-

+1

> Improve Java sandbox regex
> --
>
> Key: YARN-6472
> URL: https://issues.apache.org/jira/browse/YARN-6472
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Miklos Szegedi
>Assignee: Greg Phillips
> Attachments: YARN-6472.001.patch, YARN-6472.002.patch, 
> YARN-6472.003.patch
>
>
> I set the sandbox to enforcing mode. Unfortunately I was able to break out of 
> the sandbox running native code with the following command:
> {code}
> cmd = "$JAVA_HOME/bin/java %s -Xmx825955249 
> org.apache.hadoop.yarn.applications.helloworld.HelloWorld `touch 
> ../../helloworld`" + \
>   " 1>/AppMaster.stdout 2>/AppMaster.stderr"
> $ ls .../nm-local-dir/usercache/root/appcache/
> helloworld
> {code}
> Also, if I am not using sandboxes, could we create the nm-sandbox-policies 
> directory (empty) lazily?



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue

[Eric Payne (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989225#comment-15989225
 ] 

Eric Payne commented on YARN-2113:
--

[~sunilg], I'm sorry to say that the latest patch (YARN-2113.0013.patch) makes 
things worse for USERLIMIT_FIRST mode. Not only does it allow the ((100/MULP) + 
1) user to preempt other users down to 1 container, it also oscillates. In 
fact, even in PRIORITY_FIRST mode, it preempts users below their user limit, 
even when #users < (100/MULP).

I suggest that we separate the USERLIMIT/PRIORITY_FIRST code into a separate 
JIRA. Can you easily revert all USERLIMIT_FIRST/PRIORITY_FIRST changes? I would 
suggest using YARN-2113.0012.patch as a basis because YARN-2113.0013.patch has 
problems even in the PRIORITY_FIRST case.

> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sunil G
> Attachments: IntraQueue Preemption-Impact Analysis.pdf, 
> TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
>  YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, 
> YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch, 
> YARN-2113.0007.patch, YARN-2113.0008.patch, YARN-2113.0009.patch, 
> YARN-2113.0010.patch, YARN-2113.0011.patch, YARN-2113.0012.patch, 
> YARN-2113.0013.patch, YARN-2113.apply.onto.0012.ericp.patch, 
> YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-5894) fixed license warning caused by de.ruedigermoeller:fst:jar:2.24

[Junping Du (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Junping Du updated YARN-5894:
-
Fix Version/s: (was: 2.9.0)

> fixed license warning caused by de.ruedigermoeller:fst:jar:2.24
> ---
>
> Key: YARN-5894
> URL: https://issues.apache.org/jira/browse/YARN-5894
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Blocker
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: YARN-5894.00.patch, YARN-5894.01.patch
>
>
> The artifact de.ruedigermoeller:fst:jar:2.24, that ApplicationHistoryService 
> depends on,  shows its license being LGPL 2.1 in our license checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5894) fixed license warning caused by de.ruedigermoeller:fst:jar:2.24

[Junping Du (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989183#comment-15989183
 ] 

Junping Du commented on YARN-5894:
--

Thanks Robert!

> fixed license warning caused by de.ruedigermoeller:fst:jar:2.24
> ---
>
> Key: YARN-5894
> URL: https://issues.apache.org/jira/browse/YARN-5894
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Blocker
> Fix For: 2.8.1, 3.0.0-alpha3
>
> Attachments: YARN-5894.00.patch, YARN-5894.01.patch
>
>
> The artifact de.ruedigermoeller:fst:jar:2.24, that ApplicationHistoryService 
> depends on,  shows its license being LGPL 2.1 in our license checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Resolved] (YARN-5894) fixed license warning caused by de.ruedigermoeller:fst:jar:2.24

[Robert Kanter (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-5894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Kanter resolved YARN-5894.
-
   Resolution: Fixed
Fix Version/s: 2.8.1
   2.9.0

Sure thing - committed to branch-2, branch-2.8, and branch-2.8.1!

> fixed license warning caused by de.ruedigermoeller:fst:jar:2.24
> ---
>
> Key: YARN-5894
> URL: https://issues.apache.org/jira/browse/YARN-5894
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 3.0.0-alpha1
>Reporter: Haibo Chen
>Assignee: Haibo Chen
>Priority: Blocker
> Fix For: 2.9.0, 2.8.1, 3.0.0-alpha3
>
> Attachments: YARN-5894.00.patch, YARN-5894.01.patch
>
>
> The artifact de.ruedigermoeller:fst:jar:2.24, that ApplicationHistoryService 
> depends on,  shows its license being LGPL 2.1 in our license checking.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5534) Allow whitelisted volume mounts

[Shane Kumpf (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989104#comment-15989104
 ] 

Shane Kumpf commented on YARN-5534:
---

[~luhuichun] [~tangzhankun] - We're close on this one. Would you like me to 
take the lead on this and get it wrapped up? Thanks!

> Allow whitelisted volume mounts 
> 
>
> Key: YARN-5534
> URL: https://issues.apache.org/jira/browse/YARN-5534
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: luhuichun
>Assignee: luhuichun
> Attachments: YARN-5534.001.patch, YARN-5534.002.patch
>
>
> Introduction 
> Mounting files or directories from the host is one way of passing 
> configuration and other information into a docker container. 
> We could allow the user to set a list of mounts in the environment of 
> ContainerLaunchContext (e.g. /dir1:/targetdir1,/dir2:/targetdir2). 
> These would be mounted read-only to the specified target locations. This has 
> been resolved in YARN-4595
> 2.Problem Definition
> Bug mounting arbitrary volumes into a Docker container can be a security risk.
> 3.Possible solutions
> one approach to provide safe mounts is to allow the cluster administrator to 
> configure a set of parent directories as white list mounting directories.
>  Add a property named yarn.nodemanager.volume-mounts.white-list, when 
> container executor do mount checking, only the allowed directories or 
> sub-directories can be mounted. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-3854) Add localization support for docker images

[Shane Kumpf (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989102#comment-15989102
 ] 

Shane Kumpf commented on YARN-3854:
---

@luhuichun [~tangzhankun] - Thanks for your efforts here. I'd really like to 
get this wrapped up and would be happy to take this over. Let me know if that 
would be ok?

> Add localization support for docker images
> --
>
> Key: YARN-3854
> URL: https://issues.apache.org/jira/browse/YARN-3854
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-3854-branch-2.8.001.patch, 
> YARN-3854_Localization_support_for_Docker_image_v1.pdf, 
> YARN-3854_Localization_support_for_Docker_image_v2.pdf, 
> YARN-3854_Localization_support_for_Docker_image_v3.pdf
>
>
> We need the ability to localize docker images when those images aren't 
> already available locally. There are various approaches that could be used 
> here with different trade-offs/issues : image archives on HDFS + docker load 
> ,  docker pull during the localization phase or (automatic) docker pull 
> during the run/launch phase. 
> We also need the ability to clean-up old/stale, unused images. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Eric Badger (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989080#comment-15989080
 ] 

Eric Badger commented on YARN-6536:
---

+1 this makes sense to me. Not all allocations necessarily have to come back in 
a single iteration, so checking the aggregate allocations would be better than 
checking the most recent allocation.

> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
>Assignee: Jason Lowe
> Attachments: YARN-6536.001.patch
>
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989044#comment-15989044
 ] 

Hadoop QA commented on YARN-6536:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  0m 
30s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client in 
trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
14s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  0m 
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 
17s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
17s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6536 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12865563/YARN-6536.001.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux c206944de38a 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / cb672a4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs | 
https://builds.apache.org/job/PreCommit-YARN-Build/15775/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/15775/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client |
| Console output | 
https://builds.apache.org/job/PreCommit-YARN-Build/15775/console |
| Powered by | Apache Yetus 0.5.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>

[jira] [Updated] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated YARN-6536:
-
Attachment: YARN-6536.001.patch

I don't have time to rework the test to stop using a full minicluster, but 
here's a quick patch that should solve the sporadic failure that was reported.

> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
> Attachments: YARN-6536.001.patch
>
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Assigned] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reassigned YARN-6536:


Assignee: Jason Lowe
Target Version/s: 2.8.2

> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
>Assignee: Jason Lowe
> Attachments: YARN-6536.001.patch
>
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6520) Fix warnings from Spotbugs in hadoop-yarn-client

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988900#comment-15988900
 ] 

Weiwei Yang commented on YARN-6520:
---

Thanks [~Naganarasimha] :)

> Fix warnings from Spotbugs in hadoop-yarn-client
> 
>
> Key: YARN-6520
> URL: https://issues.apache.org/jira/browse/YARN-6520
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6520.001.patch
>
>
> There 2 findbugs warning in hadoop-yarn-client since switched to spotbugs
> # Possible exposure of partially initialized object in 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken()
> # Useless condition: it's known that isAppFinished == false at this point
> See more info from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Weiwei Yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988893#comment-15988893
 ] 

Weiwei Yang commented on YARN-6519:
---

Thanks [~Naganarasimha], got a clean result this time, thanks for the help. 
Please let me know if this looks good to you.

> Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager
> 
>
> Key: YARN-6519
> URL: https://issues.apache.org/jira/browse/YARN-6519
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6519.001.patch, YARN-6519.002.patch
>
>
> There is 8 findbugs warning in hadoop-yarn-server-timelineservice since 
> switched to spotbugs
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager$1.compare(CSQueue,
>  CSQueue) incorrectly handles float value
> # org.apache.hadoop.yarn.server.resourcemanager.scheduler.NodeType.index 
> field is public and mutable
> # 
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.EMPTY_CONTAINER_LIST
>  is a mutable collection which should be package protected
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.queueMetrics
>  is a mutable collection
> # 
> org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy.cleanupStaledPreemptionCandidates(long)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.transferStateFromAttempt(RMAppAttempt)
>  makes inefficient use of keySet iterator instead of entrySet iterator
> # 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FSSchedulerNode.cleanupPreemptionList()
>  makes inefficient use of keySet iterator instead of entrySet iterator
> See more from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe updated YARN-6536:
-
Affects Version/s: 2.8.1

> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6536) TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently

[Jason Lowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988891#comment-15988891
 ] 

Jason Lowe commented on YARN-6536:
--

This is the assertion that is failing:
{code}
assertEquals(2, amClient.release.size());
{code}

The problem is that the code above is doing this:
{code}
while (allocatedContainerCount < containersRequestedAny
&& iterationsLeft-- > 0) {
  AllocateResponse allocResponse = amClient.allocate(0.1f);
  assertEquals(0, amClient.ask.size());
  assertEquals(0, amClient.release.size());
  
  assertEquals(nodeCount, amClient.getClusterNodeCount());
  allocatedContainerCount += allocResponse.getAllocatedContainers().size();
  for(Container container : allocResponse.getAllocatedContainers()) {
ContainerId rejectContainerId = container.getId();
releases.add(rejectContainerId);
amClient.releaseAssignedContainer(rejectContainerId);
  }
[...]
{code}

If it takes more than one iteration to get two containers allocated then 
amClient.release will be 1 instead of 2 and the test will fail.  Part of the 
problem here is that it's using a full-blown minicluster and doing hacky 
100msec sleeps in hopes that the node heartbeats in the interim.  Having the 
test wield something like a MockRM and MockNM that it can explicitly control 
the heartbeats relative to allocate calls would make this test more 
deterministic (and faster).

Minimally the test should be checking {{releases.size()}} to see if all of the 
containers were released across the iterations rather than 
{{amClient.release.size()}}.

> TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently
> 
>
> Key: YARN-6536
> URL: https://issues.apache.org/jira/browse/YARN-6536
> Project: Hadoop YARN
>  Issue Type: Bug
>Affects Versions: 2.8.1
>Reporter: Eric Badger
>
> {noformat}
> java.lang.AssertionError: expected:<2> but was:<1>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:743)
>   at org.junit.Assert.assertEquals(Assert.java:118)
>   at org.junit.Assert.assertEquals(Assert.java:555)
>   at org.junit.Assert.assertEquals(Assert.java:542)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAllocation(TestAMRMClient.java:1005)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.registerAndAllocate(TestAMRMClient.java:703)
>   at 
> org.apache.hadoop.yarn.client.api.impl.TestAMRMClient.testAMRMClientWithSaslEncryption(TestAMRMClient.java:675)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Updated] (YARN-6481) Yarn top shows negative container number in FS

[Tao Jie (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tao Jie updated YARN-6481:
--
Attachment: YARN-6481.001.patch

> Yarn top shows negative container number in FS
> --
>
> Key: YARN-6481
> URL: https://issues.apache.org/jira/browse/YARN-6481
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>  Labels: newbie
> Attachments: YARN-6481.001.patch
>
>
> yarn top shows negative container numbers and they didn't change even they 
> were supposed to.
> {code}
> NodeManager(s): 2 total, 2 active, 0 unhealthy, 0 decommissioned, 0 lost, 0 
> rebooted
> Queue(s) Applications: 0 running, 12 submitted, 0 pending, 12 completed, 0 
> killed, 0 failed
> Queue(s) Mem(GB): 0 available, 0 allocated, 0 pending, 0 reserved
> Queue(s) VCores: 0 available, 0 allocated, 0 pending, 0 reserved
> Queue(s) Containers: -2 allocated, -2 pending, -2 reserved
>   APPLICATIONID USER TYPE  QUEUE   #CONT  
> #RCONT  VCORES RVC
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6481) Yarn top shows negative container number in FS

[Tao Jie (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988861#comment-15988861
 ] 

Tao Jie commented on YARN-6481:
---

When generating QueueStatistics instance in FSQueue,  metrics about containers 
are missed.
[~yufeigu][~kasha], I uploaded a patch, would you give it a review?

> Yarn top shows negative container number in FS
> --
>
> Key: YARN-6481
> URL: https://issues.apache.org/jira/browse/YARN-6481
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: yarn
>Affects Versions: 2.9.0
>Reporter: Yufei Gu
>  Labels: newbie
> Attachments: YARN-6481.001.patch
>
>
> yarn top shows negative container numbers and they didn't change even they 
> were supposed to.
> {code}
> NodeManager(s): 2 total, 2 active, 0 unhealthy, 0 decommissioned, 0 lost, 0 
> rebooted
> Queue(s) Applications: 0 running, 12 submitted, 0 pending, 12 completed, 0 
> killed, 0 failed
> Queue(s) Mem(GB): 0 available, 0 allocated, 0 pending, 0 reserved
> Queue(s) VCores: 0 available, 0 allocated, 0 pending, 0 reserved
> Queue(s) Containers: -2 allocated, -2 pending, -2 reserved
>   APPLICATIONID USER TYPE  QUEUE   #CONT  
> #RCONT  VCORES RVC
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988707#comment-15988707
 ] 

Hadoop QA commented on YARN-6519:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
28s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
24s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m 
17s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 in trunk has 8 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
32s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 274 unchanged - 4 fixed = 274 total (was 278) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 
24s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 0 new + 0 unchanged - 8 fixed = 0 total (was 8) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 43m 
10s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch 
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
23s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 42s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6519 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12864889/YARN-6519.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 891cfbb4674d 3.13.0-106-generic #153-Ubuntu SMP Tue Dec 6 
15:44:32 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / cb672a4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs | 
https://builds.apache.org/job/PreCommit-YARN-Build/15773/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-YARN-Build/15773/testReport/ |
| modules | C: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 U: 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 

[jira] [Commented] (YARN-6519) Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988700#comment-15988700
 ] 

Hadoop QA commented on YARN-6519:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
13s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
4s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 in trunk has 8 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
26s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 0 new + 275 unchanged - 4 fixed = 275 total (was 279) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
9s{color} | {color:green} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 generated 0 new + 0 unchanged - 8 fixed = 0 total (was 8) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 38m 32s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
21s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 54s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-6519 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12864889/YARN-6519.002.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 7114973033a3 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / cb672a4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs | 
https://builds.apache.org/job/PreCommit-YARN-Build/15774/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
 |
| unit | 
https://builds.apache.org/job/PreCommit-YARN-Build/15774/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt

[jira] [Commented] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue

[Hadoop QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988680#comment-15988680
 ] 

Hadoop QA commented on YARN-2113:
-

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 
17s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 3 new or modified test 
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  0m 
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
17s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red}  1m  
3s{color} | {color:red} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
 in trunk has 8 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
31s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  
0m 24s{color} | {color:orange} 
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
 The patch generated 21 new + 181 unchanged - 3 fixed = 202 total (was 184) 
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  0m 
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green}  0m 
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m 
 0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m  
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  0m 
19s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 39m 57s{color} 
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 
20s{color} | {color:green} The patch does not generate ASF License warnings. 
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 63m 17s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.server.resourcemanager.TestRMRestart |
|   | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-2113 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12865516/YARN-2113.0013.patch |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  
unit  findbugs  checkstyle  |
| uname | Linux 26dfb2352bfc 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20 
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh 
|
| git revision | trunk / cb672a4 |
| Default Java | 1.8.0_121 |
| findbugs | v3.1.0-RC1 |
| findbugs | 
https://builds.apache.org/job/PreCommit-YARN-Build/15772/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html
 |
| checkstyle | 
https://builds.apache.org/job/PreCommit-YARN-Build/15772/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
 |
| unit | 

[jira] [Updated] (YARN-2113) Add cross-user preemption within CapacityScheduler's leaf-queue

[Sunil G (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-2113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sunil G updated YARN-2113:
--
Attachment: YARN-2113.0013.patch

[~eepayne]
Thank you very much..

I have some more thoughts to add here. To address the problem which you have 
mentioned, we are already kind of handling it in 
{{FifoIntraQueuePreemptionPlugin#validateOutSameAppPriorityFromDemand}}
{code}
+// Ideally if any application has a higher priority, then it can
+// force to preempt any lower priority app from any user. However
+// if admin enforces user-limit over priority, preemption module
+// will not choose lower priority apps from usre's who are not yet
+// met its user-limit.
+TempUserPerPartition tmpUser = usersPerPartition
+.get(apps[lPriority].getUser());
+if ((!apps[hPriority].getUser().equals(apps[lPriority].getUser()))
+&& (tmpUser.isUserLimitReached(rc, cluster) == false)
+&& (intraQueuePreemptionOrder
+.equals(IntraQueuePreemptionOrder.USERLIMIT_FIRST))) {
+  continue;
+}
{code}

This indicates that we ll not preempt resources from other users who are still 
under its userlimit. But given priority_first order, resources could be 
preempted from such users.

I guess the [latest issue mentioned by 
you|https://issues.apache.org/jira/browse/YARN-2113?focusedCommentId=15985032=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15985032],
 is caused only because of logic added to find the FAT container which will 
bring down the usage under UL. To identify fat container, i think we must 
iterate through all containers.

Without the logic added to find FAT container, priority based preemption will 
really work eventhough we set order as userlimit-first. Hence in my latest 
patch, I was trying to handle this specific case step by step.

# In first pass of preemption, we could find that one user "user1" has 25GB of 
used resource and UL is 20GB (after calculation). Given demand from other 
users, we have to preempt 4GB (avoiding last container). But there could be a 
high priority app within "user1" who has more demand. But we wont consider this 
in round 1.
# In second or further passes, once user-limit is normalized, we could preempt 
for that high priority app within "user1" itself. This is possible by the Step 
3) which is mentioned in my earlier comment and v12 patch was handling same.

Attaching v13 patch with better refactoring and test cases. Please share your 
thoughts?

> Add cross-user preemption within CapacityScheduler's leaf-queue
> ---
>
> Key: YARN-2113
> URL: https://issues.apache.org/jira/browse/YARN-2113
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: scheduler
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Sunil G
> Attachments: IntraQueue Preemption-Impact Analysis.pdf, 
> TestNoIntraQueuePreemptionIfBelowUserLimitAndDifferentPrioritiesWithExtraUsers.txt,
>  YARN-2113.0001.patch, YARN-2113.0002.patch, YARN-2113.0003.patch, 
> YARN-2113.0004.patch, YARN-2113.0005.patch, YARN-2113.0006.patch, 
> YARN-2113.0007.patch, YARN-2113.0008.patch, YARN-2113.0009.patch, 
> YARN-2113.0010.patch, YARN-2113.0011.patch, YARN-2113.0012.patch, 
> YARN-2113.0013.patch, YARN-2113.apply.onto.0012.ericp.patch, 
> YARN-2113.v0.patch
>
>
> Preemption today only works across queues and moves around resources across 
> queues per demand and usage. We should also have user-level preemption within 
> a queue, to balance capacity across users in a predictable manner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Comment Edited] (YARN-6521) Yarn Log-aggregation Transform Enable not to Spam the NameNode

[zhangyubiao (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988453#comment-15988453
 ] 

zhangyubiao edited comment on YARN-6521 at 4/28/17 10:19 AM:
-

Thanks for all. Thanks for [~rkanter],[~djp]. I will try the HadoopArchiveLogs.

By the way I see that YARN-431 Stabilize YARN application log-handling. But 
None of them basically solve the containers increase  making too many small 
files and causing a problem for the NN at once. 

 If we don't enable  log aggregation and can provide "yarn logs" to get logs.  
The problem will gone forever.  
The thing I face is that we can't get the app and contianerReport  relationship 
 when the app finish(Spark). 


was (Author: piaoyu zhang):
Thanks for all. Thanks for [~rkanter],[~djp]. I will try the HadoopArchiveLogs.

By the way I see that YARN-431 Stabilize YARN application log-handling. But 
None of them basically solve the containers increase  making too many small 
files and causing a problem for the NN at once. 

 If we don't enable  log aggregation and can provide "yarn logs" to get logs.  
The problem will gone forever.  
The thing I face is that we can't get the app and contianerReport  relationship 
 when the app finish. 

> Yarn Log-aggregation Transform Enable not to Spam the NameNode
> --
>
> Key: YARN-6521
> URL: https://issues.apache.org/jira/browse/YARN-6521
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: zhangyubiao
>
> Nowdays We have large cluster,with the apps incr and container incr ,We split 
> an  namespace to store applogs. 
> But the  log-aggregation /tmp/app-logs incr also make the namespace responce 
> slow.  
> We want to chang yarn.log-aggregation-enable true -> false,But Transform the 
> the yarn log cli service also can get the app-logs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6409) RM does not blacklist node for AM launch failures

[Rohith Sharma K S (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988555#comment-15988555
 ] 

Rohith Sharma K S commented on YARN-6409:
-

bq. Do you mean to say there is NO way to remove blacklisted nodes from AM 
container?
Yup

bq. RM will often try the subsequent AM launch attempts on the same node and 
fail the application eventually,
Do you mean NM is unresponsive but NM heartbeats are sent to RM? 

> RM does not blacklist node for AM launch failures
> -
>
> Key: YARN-6409
> URL: https://issues.apache.org/jira/browse/YARN-6409
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: resourcemanager
>Affects Versions: 3.0.0-alpha2
>Reporter: Haibo Chen
>Assignee: Haibo Chen
> Attachments: YARN-6409.00.patch, YARN-6409.01.patch
>
>
> Currently, node blacklisting upon AM failures only handles failures that 
> happen after AM container is launched (see 
> RMAppAttemptImpl.shouldCountTowardsNodeBlacklisting()).  However, AM launch 
> can also fail if the NM, where the AM container is allocated, goes 
> unresponsive.  Because it is not handled, scheduler may continue to allocate 
> AM containers on that same NM for the following app attempts. 
> {code}
> Application application_1478721503753_0870 failed 2 times due to Error 
> launching appattempt_1478721503753_0870_02. Got exception: 
> java.io.IOException: Failed on local exception: java.io.IOException: 
> java.net.SocketTimeoutException: 6 millis timeout while waiting for 
> channel to be ready for read. ch : java.nio.channels.SocketChannel[connected 
> local=/17.111.179.113:46702 remote=*.me.com/17.111.178.125:8041]; Host 
> Details : local host is: "*.me.com/17.111.179.113"; destination host is: 
> "*.me.com":8041; 
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) 
> at org.apache.hadoop.ipc.Client.call(Client.java:1475) 
> at org.apache.hadoop.ipc.Client.call(Client.java:1408) 
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
>  
> at com.sun.proxy.$Proxy86.startContainers(Unknown Source) 
> at 
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.startContainers(ContainerManagementProtocolPBClientImpl.java:96)
>  
> at sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  
> at java.lang.reflect.Method.invoke(Method.java:497) 
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
>  
> at 
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
>  
> at com.sun.proxy.$Proxy87.startContainers(Unknown Source) 
> at 
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.launch(AMLauncher.java:120)
>  
> at 
> org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:256)
>  
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  
> at java.lang.Thread.run(Thread.java:745) 
> Caused by: java.io.IOException: java.net.SocketTimeoutException: 6 millis 
> timeout while waiting for channel to be ready for read. ch : 
> java.nio.channels.SocketChannel[connected local=/17.111.179.113:46702 
> remote=*.me.com/17.111.178.125:8041] 
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:687) 
> at java.security.AccessController.doPrivileged(Native Method) 
> at javax.security.auth.Subject.doAs(Subject.java:422) 
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
>  
> at 
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:650)
>  
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:738) 
> at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375) 
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1524) 
> at org.apache.hadoop.ipc.Client.call(Client.java:1447) 
> ... 15 more 
> Caused by: java.net.SocketTimeoutException: 6 millis timeout while 
> waiting for channel to be ready for read. ch : 
> java.nio.channels.SocketChannel[connected local=/17.111.179.113:46702 
> remote=*.me.com/17.111.178.125:8041] 
> at 
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164) 
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) 
> at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) 
> at java.io.FilterInputStream.read(FilterInputStream.java:133) 
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) 
> at 

[jira] [Commented] (YARN-6521) Yarn Log-aggregation Transform Enable not to Spam the NameNode

[zhangyubiao (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988453#comment-15988453
 ] 

zhangyubiao commented on YARN-6521:
---

Thanks for all. Thanks for [~rkanter],[~djp]. I will try the HadoopArchiveLogs.

By the way I see that YARN-431 Stabilize YARN application log-handling. But 
None of them basically solve the containers increase  making too many small 
files and causing a problem for the NN at once. 

 If we don't enable  log aggregation and can provide "yarn logs" to get logs.  
The problem will gone forever.  
The thing I face is that we can't get the app and contianerReport  relationship 
 when the app finish. 

> Yarn Log-aggregation Transform Enable not to Spam the NameNode
> --
>
> Key: YARN-6521
> URL: https://issues.apache.org/jira/browse/YARN-6521
> Project: Hadoop YARN
>  Issue Type: Improvement
>Reporter: zhangyubiao
>
> Nowdays We have large cluster,with the apps incr and container incr ,We split 
> an  namespace to store applogs. 
> But the  log-aggregation /tmp/app-logs incr also make the namespace responce 
> slow.  
> We want to chang yarn.log-aggregation-enable true -> false,But Transform the 
> the yarn log cli service also can get the app-logs.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5983) [Umbrella] Support for FPGA as a Resource in YARN

[Zhankun Tang (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988445#comment-15988445
 ] 

Zhankun Tang commented on YARN-5983:


[~devaraj.k], Thanks a lot for the review.
1.
{code:xml}
The scheduler only considers non-exclusive resource. The exclusive resources may
have extra attributes needs to be matched when scheduling. Not just simply add 
or
reduce a number. For instance, in our PoC, a FPGA slot in one node may already
have one IP flashed so that the scheduler should try to match this IP attribute 
to
reuse it.
{code}
{quote}
If you are passing all the attributes of the FPGA resources to RM scheduler, 
why do you want to have the NM side resource management? Can you give some 
details about the attributes passing to the RM and details maintain by the NM 
side resource management in abstract terms?
{quote}
Zhankun --> The RM maintains all attributes of FPGA resources is an ideal 
solution which needs to extend YARN-3926 scheduler implementation. But in this 
design proposal, we'd prefer to keep YARN-3926 RM scheduler unchanged(which 
only consider the count of resource) and add NM a local FPGA resource handler 
which will mainly:
a). Schedule the FPGA resource request(2 MCP for instance) into real FPGA device
b). Handle FPGA IP download and re-configure of specific slot
The key part of this proposal is similar to GPU design(YARN-6223) but add the 
vendor framework part.
2.
{quote}
Does this need to be done for each container, Can it be done one time during 
the cluster installation?
{quote}
Zhankun: --> Good question. Actually the admin can do the IP download and 
configuration of FPGA one time to the cluster manually. But if an application 
wants to use a new FPGA IP that needs re-configuration, a good way of doing 
these may be in YARN itself. This automation in YARN can potentially increase 
the FPGA utilization(not good as the ideal RM scheduler) if we have large 
amount of FPGA devices.
And these work should be done for each container, although we can skip it if an 
FPGA slot already matches IP requirements.
3. 
{quote}
Can FPGA slots share my multiple containers? How do we prevent if any 
container(Non FPGA allocated container)/application try to use the FPGA 
resources which are not allocated to them?
{quote}
Zhankun: --> We can utilize cgroups/Docker to do FPGA slot isolation to avoid 
invalid access. But I guess there may be requirements that don't want isolation 
in order to maximize device utilization( the user may have a daemon managing 
requests from multiple containers to FPGA IP), we can provide option for user 
to disable isolation?
4.
{quote}
Any changes to ContainerExecutor, how does the application code running in the 
container come to know about the allocated FPGA resource to access/use the FPFA?
{quote}
Zhankun:--> It should be no changes to both DefaultContainerExecutor and 
LinuxContainerExecutor. Knowing about which FPGA resource to use depends on the 
vendor specific SDK I think. For instance, container application code may 
utilize Intel AAL SDK and the latter will know which FPGA slot to use (maybe 
based on some environment setting).
5.
{quote}
What are the configurations user to need to configure for the application to 
use FPGA resources?
{quote}
Zhankun: --> For YARN configurations before cluster start, the admin need to 
define different type of FPGA resources and its count. When a user wants to 
submit the application, it needs to provide two inputs:
a). the required FPGA resource for container request (AM will use this to 
request FPGA container from RM)
b). the ID/name of the required FPGA IP (can be set in container's environment 
variable for container run-time to use)


And great thanks again for the questions and remind me some points uncertain. 
I'll provide a patch to make it more clear later.

> [Umbrella] Support for FPGA as a Resource in YARN
> -
>
> Key: YARN-5983
> URL: https://issues.apache.org/jira/browse/YARN-5983
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
> Attachments: YARN-5983-Support-FPGA-resource-on-NM-side_v1.pdf
>
>
> As various big data workload running on YARN, CPU will no longer scale 
> eventually and heterogeneous systems will become more important. ML/DL is a 
> rising star in recent years, applications focused on these areas have to 
> utilize GPU or FPGA to boost performance. Also, hardware vendors such as 
> Intel also invest in such hardware. It is most likely that FPGA will become 
> popular in data centers like CPU in the near future.
> So YARN as a resource managing and scheduling system, would be great to 
> evolve to support this. This JIRA proposes FPGA to be a first-class citizen. 
> The changes roughly includes:
> 

[jira] [Updated] (YARN-6369) [YARN-3368] Refactor of yarn-app and yarn-app-attempt pages in YARN-UI

[Akhil PB (JIRA)]

 [ 
https://issues.apache.org/jira/browse/YARN-6369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Akhil PB updated YARN-6369:
---
Summary: [YARN-3368] Refactor of yarn-app and yarn-app-attempt pages in 
YARN-UI  (was: [YARN-3368] Redesign of app attempt page in YARN-UI)

> [YARN-3368] Refactor of yarn-app and yarn-app-attempt pages in YARN-UI
> --
>
> Key: YARN-6369
> URL: https://issues.apache.org/jira/browse/YARN-6369
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: yarn-ui-v2
>Reporter: Akhil PB
>Assignee: Akhil PB
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-5983) [Umbrella] Support for FPGA as a Resource in YARN

[Devaraj K (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-5983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988319#comment-15988319
 ] 

Devaraj K commented on YARN-5983:
-

Thanks [~tangzhankun] and [~zyluo] for the design doc and hardwork, 
[~leftnoteasy] for the discussion.

1.
{code:xml}
The scheduler only considers non-exclusive resource. The exclusive resources may
have extra attributes needs to be matched when scheduling. Not just simply add 
or
reduce a number. For instance, in our PoC, a FPGA slot in one node may already
have one IP flashed so that the scheduler should try to match this IP attribute 
to
reuse it.
{code}

If you are passing all the attributes of the FPGA resources to RM scheduler, 
why do you want to have the NM side resource management? Can you give some 
details about the attributes passing to the RM and details maintain by the NM 
side resource management in abstract terms? 

2. {code:xml}
 Device resource needs additional preparation and isolation before container 
launch.
For instance, FPGA device may need to download an IP file from a repo then 
flash to
an allocated FPGA slot.
{code}
Does this need to be done for each container, Can it be done one time during 
the cluster installation?

3. Can FPGA slots share my multiple containers? How do we prevent if any 
container(Non FPGA allocated container)/application try to use the FPGA 
resources which are not allocated to them?

4. Any changes to ContainerExecutor, how does the application code running in 
the container come to know about the allocated FPGA resource to access/use the 
FPFA?

5. What are the configurations user to need to configure for the application to 
use FPGA resources?


> [Umbrella] Support for FPGA as a Resource in YARN
> -
>
> Key: YARN-5983
> URL: https://issues.apache.org/jira/browse/YARN-5983
> Project: Hadoop YARN
>  Issue Type: New Feature
>  Components: yarn
>Reporter: Zhankun Tang
>Assignee: Zhankun Tang
> Attachments: YARN-5983-Support-FPGA-resource-on-NM-side_v1.pdf
>
>
> As various big data workload running on YARN, CPU will no longer scale 
> eventually and heterogeneous systems will become more important. ML/DL is a 
> rising star in recent years, applications focused on these areas have to 
> utilize GPU or FPGA to boost performance. Also, hardware vendors such as 
> Intel also invest in such hardware. It is most likely that FPGA will become 
> popular in data centers like CPU in the near future.
> So YARN as a resource managing and scheduling system, would be great to 
> evolve to support this. This JIRA proposes FPGA to be a first-class citizen. 
> The changes roughly includes:
> 1. FPGA resource detection and heartbeat
> 2. Scheduler changes
> 3. FPGA related preparation and isolation before launch container
> We know that YARN-3926 is trying to extend current resource model. But still 
> we can leave some FPGA related discussion here



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

[jira] [Commented] (YARN-6520) Fix warnings from Spotbugs in hadoop-yarn-client

[Naganarasimha G R (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-6520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15988282#comment-15988282
 ] 

Naganarasimha G R commented on YARN-6520:
-

Ohh Thanks [~cheersyang], you are right saw the console in detail and concluded 
the same... will commit the changes shortly.

> Fix warnings from Spotbugs in hadoop-yarn-client
> 
>
> Key: YARN-6520
> URL: https://issues.apache.org/jira/browse/YARN-6520
> Project: Hadoop YARN
>  Issue Type: Bug
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>  Labels: findbugs
> Attachments: YARN-6520.001.patch
>
>
> There 2 findbugs warning in hadoop-yarn-client since switched to spotbugs
> # Possible exposure of partially initialized object in 
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getTimelineDelegationToken()
> # Useless condition: it's known that isAppFinished == false at this point
> See more info from 
> [https://builds.apache.org/job/PreCommit-HADOOP-Build/12157/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client-warnings.html]



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org