[jira] [Commented] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15404644#comment-15404644 ] Subru Krishnan commented on YARN-4888: -- The checkstyle issue is to do with more than 7 parameters and the test case failure is unrelated. > Changes in RM container allocation for identifying resource-requests > explicitly > --- > > Key: YARN-4888 > URL: https://issues.apache.org/jira/browse/YARN-4888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch, > YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888-v4.patch, > YARN-4888-v5.patch, YARN-4888-v6.patch, YARN-4888.001.patch > > > YARN-4879 puts forward the notion of identifying allocate requests > explicitly. This JIRA is to track the changes in RM app scheduling data > structures to accomplish it. Please refer to the design doc in the parent > JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5307: - Attachment: YARN-5307-YARN-2915-v3.patch [~vvasudev], thanks for your feedback. Attaching patch (v3) that incorporates your comment. The names may not be exactly what you suggested as I have to tried to align with the final version of YARN-3662 which includes [~vinodkv]/[~leftnoteasy]'s feedback too. > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch, YARN-5307-YARN-2915-v3.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15403044#comment-15403044
]
Subru Krishnan edited comment on YARN-4888 at 8/1/16 11:39 PM:
---
Thanks [~leftnoteasy] for reviewing the patch. I have updated (v6) it with your
feedback.
I agree that the tests have commonality between _Capacity/Fair schedulers_ but
I added them individually as YARN-4805 has disabled parameterized test
execution. For e.g.: I did hit a minor bug in one of my early versions with FS
which the current {{ParameterizedSchedulerTestBase}} will not capture but the
existing tests in the patch will.
Cc: [~kasha], what are your thoughts as I see you worked on YARN-4805 and you
are aware of the changes here?
was (Author: subru):
Thanks [~leftnoteasy] for reviewing the patch. I have updated it with your
feedback.
I agree that the tests have commonality between _Capacity/Fair schedulers_ but
I added them individually as YARN-4805 has disabled parameterized test
execution. For e.g.: I did hit a minor bug in one of my early versions with FS
which the current {{ParameterizedSchedulerTestBase}} will not capture but the
existing tests in the patch will.
Cc: [~kasha], what are your thoughts as I see you worked on YARN-4805 and you
are aware of the changes here?
> Changes in RM container allocation for identifying resource-requests
> explicitly
> ---
>
> Key: YARN-4888
> URL: https://issues.apache.org/jira/browse/YARN-4888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch,
> YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888-v4.patch,
> YARN-4888-v5.patch, YARN-4888-v6.patch, YARN-4888.001.patch
>
>
> YARN-4879 puts forward the notion of identifying allocate requests
> explicitly. This JIRA is to track the changes in RM app scheduling data
> structures to accomplish it. Please refer to the design doc in the parent
> JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-4888:
-
Attachment: YARN-4888-v6.patch
Thanks [~leftnoteasy] for reviewing the patch. I have updated it with your
feedback.
I agree that the tests have commonality between _Capacity/Fair schedulers_ but
I added them individually as YARN-4805 has disabled parameterized test
execution. For e.g.: I did hit a minor bug in one of my early versions with FS
which the current {{ParameterizedSchedulerTestBase}} will not capture but the
existing tests in the patch will.
Cc: [~kasha], what are your thoughts as I see you worked on YARN-4805 and you
are aware of the changes here?
> Changes in RM container allocation for identifying resource-requests
> explicitly
> ---
>
> Key: YARN-4888
> URL: https://issues.apache.org/jira/browse/YARN-4888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch,
> YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888-v4.patch,
> YARN-4888-v5.patch, YARN-4888-v6.patch, YARN-4888.001.patch
>
>
> YARN-4879 puts forward the notion of identifying allocate requests
> explicitly. This JIRA is to track the changes in RM app scheduling data
> structures to accomplish it. Please refer to the design doc in the parent
> JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-4888:
-
Attachment: YARN-4888-v5.patch
Updating patch (v5) with new test
{{TestAppSchedulingInfo::testSchedulerRequestKeyOrdering}} as requested by
[~asuresh].
{{TestDirectoryCollection}} failure is unrelated to this patch.
> Changes in RM container allocation for identifying resource-requests
> explicitly
> ---
>
> Key: YARN-4888
> URL: https://issues.apache.org/jira/browse/YARN-4888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch,
> YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888-v4.patch,
> YARN-4888-v5.patch, YARN-4888.001.patch
>
>
> YARN-4879 puts forward the notion of identifying allocate requests
> explicitly. This JIRA is to track the changes in RM app scheduling data
> structures to accomplish it. Please refer to the design doc in the parent
> JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[ https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400253#comment-15400253 ] Subru Krishnan edited comment on YARN-5221 at 7/29/16 11:50 PM: It's a rather substantial patch :). I'll take a look at it and get back by Monday. >From an initial pass, I feel it will help if we can split the patch: 1. API changes to unify Container change requests. 2. Backend changes to affect the unified API. 3. NMStateStore and associated NMContainerStatus changes for RM failover. Thoughts [~asuresh]? was (Author: subru): It's a rather substantial patch :). I'll take a look at it and get back by Monday. > Expose UpdateResourceRequest API to allow AM to request for change in > container properties > -- > > Key: YARN-5221 > URL: https://issues.apache.org/jira/browse/YARN-5221 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5221.001.patch, YARN-5221.002.patch, > YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch, > YARN-5221.006.patch, YARN-5221.007.patch, YARN-5221.008.patch > > > YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease > of Container Resources after initial allocation. > YARN-5085 proposes to allow an AM to request for a change of Container > ExecutionType. > This JIRA proposes to unify both of the above into an Update Container API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5221) Expose UpdateResourceRequest API to allow AM to request for change in container properties
[ https://issues.apache.org/jira/browse/YARN-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400253#comment-15400253 ] Subru Krishnan commented on YARN-5221: -- It's a rather substantial patch :). I'll take a look at it and get back by Monday. > Expose UpdateResourceRequest API to allow AM to request for change in > container properties > -- > > Key: YARN-5221 > URL: https://issues.apache.org/jira/browse/YARN-5221 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5221.001.patch, YARN-5221.002.patch, > YARN-5221.003.patch, YARN-5221.004.patch, YARN-5221.005.patch, > YARN-5221.006.patch, YARN-5221.007.patch, YARN-5221.008.patch > > > YARN-1197 introduced APIs to allow an AM to request for Increase and Decrease > of Container Resources after initial allocation. > YARN-5085 proposes to allow an AM to request for a change of Container > ExecutionType. > This JIRA proposes to unify both of the above into an Update Container API. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-4888: - Attachment: YARN-4888-v4.patch Thanks [~asuresh] for the review, good catch on the allocationRequestId comparator. PFA (v4) patch that addresses your comments. > Changes in RM container allocation for identifying resource-requests > explicitly > --- > > Key: YARN-4888 > URL: https://issues.apache.org/jira/browse/YARN-4888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch, > YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888-v4.patch, > YARN-4888.001.patch > > > YARN-4879 puts forward the notion of identifying allocate requests > explicitly. This JIRA is to track the changes in RM app scheduling data > structures to accomplish it. Please refer to the design doc in the parent > JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15399797#comment-15399797 ] Subru Krishnan commented on YARN-3662: -- Thanks [~vinodkv]! FYI I deliberately kept the modifiers public as I mentioned [earlier|https://issues.apache.org/jira/browse/YARN-3662?focusedCommentId=15360676] but I am fine with adding it when required downstream. > Federation Membership State Store internal APIs > --- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch, > YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch, > YARN-3662-YARN-2915-v6.patch, YARN-3662-YARN-2915-v7.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-4888: - Attachment: YARN-4888-v3.patch Attaching v3 patch that fixes the test case failures and checkstyle issues. > Changes in RM container allocation for identifying resource-requests > explicitly > --- > > Key: YARN-4888 > URL: https://issues.apache.org/jira/browse/YARN-4888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch, > YARN-4888-v2.patch, YARN-4888-v3.patch, YARN-4888.001.patch > > > YARN-4879 puts forward the notion of identifying allocate requests > explicitly. This JIRA is to track the changes in RM app scheduling data > structures to accomplish it. Please refer to the design doc in the parent > JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) [Regression] QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Labels: regression (was: ) > [Regression] QueueCapacities not being updated for dynamic ReservationQueue > --- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Labels: regression > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues (ReservationQueue) to CapacityScheduler. The > QueueCapacities data structure was added subsequently but is not being > updated correctly for ReservationQueue. This JIRA tracks the changes required > to update QueueCapacities of ReservationQueue correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) [Regression] QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Summary: [Regression] QueueCapacities not being updated for dynamic ReservationQueue (was: QueueCapacities not being updated for dynamic ReservationQueue) > [Regression] QueueCapacities not being updated for dynamic ReservationQueue > --- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues (ReservationQueue) to CapacityScheduler. The > QueueCapacities data structure was added subsequently but is not being > updated correctly for ReservationQueue. This JIRA tracks the changes required > to update QueueCapacities of ReservationQueue correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Description: YARN-1707 added dynamic queues (ReservationQueue) to CapacityScheduler. The QueueCapacities data structure was added subsequently but is not being updated correctly for ReservationQueue. This JIRA tracks the changes required to update QueueCapacities of ReservationQueue correctly. (was: YARN-1707 added dynamic queues ( to CapacityScheduler. The QueueCapacities data structure was added subsequently but is not being updated correctly for ) > QueueCapacities not being updated for dynamic ReservationQueue > -- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues (ReservationQueue) to CapacityScheduler. The > QueueCapacities data structure was added subsequently but is not being > updated correctly for ReservationQueue. This JIRA tracks the changes required > to update QueueCapacities of ReservationQueue correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Description: YARN-1707 added dynamic queues ( to CapacityScheduler. The QueueCapacities data structure was added subsequently but is not being updated correctly for (was: YARN-1707 added dynamic queues to CapacityScheduler. The QueueCapacities data structure was added subsequently but is not being updated correctly for ) > QueueCapacities not being updated for dynamic ReservationQueue > -- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues ( to CapacityScheduler. The QueueCapacities > data structure was added subsequently but is not being updated correctly for -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Description: YARN-1707 added dynamic queues to CapacityScheduler. The QueueCapacities data structure was added subsequently but is not being updated correctly for (was: YARN-1707 added dynamic queues to CapacityScheduler. ) > QueueCapacities not being updated for dynamic ReservationQueue > -- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues to CapacityScheduler. The QueueCapacities data > structure was added subsequently but is not being updated correctly for -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Description: YARN-1707 added dynamic queues to CapacityScheduler. (was: After a reservation queue is allocated, and the plan is synchronized by the plan follower, we expect the user am resource limit to reflect this change if the appropriate configuration is set. Instead, the user am resource limit is always the same. As a result, multiple AM cannot run in parallel for small reservations. To reproduce this issue, create a reservation, and submit multiple applications against the new reservation.) > QueueCapacities not being updated for dynamic ReservationQueue > -- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > YARN-1707 added dynamic queues to CapacityScheduler. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) QueueCapacities not being updated for dynamic ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Summary: QueueCapacities not being updated for dynamic ReservationQueue (was: [Regression]User AM resource limit does not get updated for ReservationQueue) > QueueCapacities not being updated for dynamic ReservationQueue > -- > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > After a reservation queue is allocated, and the plan is synchronized by the > plan follower, we expect the user am resource limit to reflect this change if > the appropriate configuration is set. > Instead, the user am resource limit is always the same. As a result, multiple > AM cannot run in parallel for small reservations. > To reproduce this issue, create a reservation, and submit multiple > applications against the new reservation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5435) [Regression]User AM resource limit does not get updated for ReservationQueue
[ https://issues.apache.org/jira/browse/YARN-5435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5435: - Summary: [Regression]User AM resource limit does not get updated for ReservationQueue (was: User AM resource limit for does not get updated for ReservationQueue after execution of plan-follower.) > [Regression]User AM resource limit does not get updated for ReservationQueue > - > > Key: YARN-5435 > URL: https://issues.apache.org/jira/browse/YARN-5435 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, resourcemanager >Affects Versions: 2.8.0 >Reporter: Sean Po >Assignee: Sean Po > Attachments: YARN-5435.v1.patch, YARN-5435.v2.patch > > > After a reservation queue is allocated, and the plan is synchronized by the > plan follower, we expect the user am resource limit to reflect this change if > the appropriate configuration is set. > Instead, the user am resource limit is always the same. As a result, multiple > AM cannot run in parallel for small reservations. > To reproduce this issue, create a reservation, and submit multiple > applications against the new reservation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398455#comment-15398455
]
Subru Krishnan edited comment on YARN-4888 at 7/28/16 11:58 PM:
Clean patch (*v2*) with new tests for checking allocation request ids for both
{{Capacity/FairScheduler}} post rebase with YARN-5392.
The patch is complete and covers e2e scenarios except for:
1. Having multiple node/rack local requests for same *Priority* by using
different *AllocationRequestIds*. This is essentially the remaining work for
YARN-314.
2. Recreating *AllocationRequestId* for recovered *Containers* if RM fails
over. This might cause an issue only if RM fails over before notifying the AM
about newly allocated container(s). I have created YARN-5447 to track this.
was (Author: subru):
Clean patch with new tests for checking allocation request ids for both
{{Capacity/FairScheduler}} post rebase with YARN-5392.
The patch is complete and covers e2e scenarios except for:
1. Having multiple node/rack local requests for same *Priority* by using
different *AllocationRequestIds*. This is essentially the remaining work for
YARN-314.
2. Recreating *AllocationRequestId* for recovered *Containers* if RM fails
over. This might cause an issue only if RM fails over before notifying the AM
about newly allocated container(s). I have created YARN-5447 to track this.
> Changes in RM container allocation for identifying resource-requests
> explicitly
> ---
>
> Key: YARN-4888
> URL: https://issues.apache.org/jira/browse/YARN-4888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch,
> YARN-4888-v2.patch, YARN-4888.001.patch
>
>
> YARN-4879 puts forward the notion of identifying allocate requests
> explicitly. This JIRA is to track the changes in RM app scheduling data
> structures to accomplish it. Please refer to the design doc in the parent
> JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5447) Consider including allocationRequestId in NMContainerStatus to allow recovery in case of RM failover
[ https://issues.apache.org/jira/browse/YARN-5447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5447: - Description: We have added a mapping of the allocated container to the original request through YARN-4887/YARN-4888. There is a corner case in which the mapping will be lost, i.e. if RM fails over before notifying the AM about newly allocated container(s). This JIRA tracks the changes required to include the allocationRequestId in NMContainerStatus to allow recovery in case of RM failover. (was: We have added a mapping of the allocated container to the original request through YARN-4887/YARN-4888. This JIRA tracks the changes required to include the allocationRequestId in NMContainerStatus to allow recovery in case of RM failover.) > Consider including allocationRequestId in NMContainerStatus to allow recovery > in case of RM failover > > > Key: YARN-5447 > URL: https://issues.apache.org/jira/browse/YARN-5447 > Project: Hadoop YARN > Issue Type: Sub-task > Components: applications, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > > We have added a mapping of the allocated container to the original request > through YARN-4887/YARN-4888. There is a corner case in which the mapping will > be lost, i.e. if RM fails over before notifying the AM about newly allocated > container(s). This JIRA tracks the changes required to include the > allocationRequestId in NMContainerStatus to allow recovery in case of RM > failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM container allocation for identifying resource-requests explicitly
[ https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-4888: - Summary: Changes in RM container allocation for identifying resource-requests explicitly (was: Changes in RM AppSchedulingInfo for identifying resource-requests explicitly) > Changes in RM container allocation for identifying resource-requests > explicitly > --- > > Key: YARN-4888 > URL: https://issues.apache.org/jira/browse/YARN-4888 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch, > YARN-4888-v2.patch, YARN-4888.001.patch > > > YARN-4879 puts forward the notion of identifying allocate requests > explicitly. This JIRA is to track the changes in RM app scheduling data > structures to accomplish it. Please refer to the design doc in the parent > JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM AppSchedulingInfo for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-4888:
-
Attachment: YARN-4888-v2.patch
Clean patch with new tests for checking allocation request ids for both
{{Capacity/FairScheduler}} post rebase with YARN-5392.
The patch is complete and covers e2e scenarios except for:
1. Having multiple node/rack local requests for same *Priority* by using
different *AllocationRequestIds*. This is essentially the remaining work for
YARN-314.
2. Recreating *AllocationRequestId* for recovered *Containers* if RM fails
over. This might cause an issue only if RM fails over before notifying the AM
about newly allocated container(s). I have created YARN-5447 to track this.
> Changes in RM AppSchedulingInfo for identifying resource-requests explicitly
>
>
> Key: YARN-4888
> URL: https://issues.apache.org/jira/browse/YARN-4888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch,
> YARN-4888-v2.patch, YARN-4888.001.patch
>
>
> YARN-4879 puts forward the notion of identifying allocate requests
> explicitly. This JIRA is to track the changes in RM app scheduling data
> structures to accomplish it. Please refer to the design doc in the parent
> JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5447) Consider including allocationRequestId in NMContainerStatus to allow recovery in case of RM failover
Subru Krishnan created YARN-5447: Summary: Consider including allocationRequestId in NMContainerStatus to allow recovery in case of RM failover Key: YARN-5447 URL: https://issues.apache.org/jira/browse/YARN-5447 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Subru Krishnan We have added a mapping of the allocated container to the original request through YARN-4887/YARN-4888. This JIRA tracks the changes required to include the allocationRequestId in NMContainerStatus to allow recovery in case of RM failover. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398386#comment-15398386
]
Subru Krishnan edited comment on YARN-5203 at 7/28/16 11:13 PM:
I just committed this to trunk/branch-2.
Congrats [~ellenfkh] on your first patch :)!
Thanks to [~sunilg] for the thorough vetting and [~vvasudev] for his input.
was (Author: subru):
I just committed this to trunk/branch-2.
Congrats [~ellenfkh] on your first patch :)!
Thanks to [~sunilg] for the thorough whetting and [~vvasudev] for his input.
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Labels: api-breaking, bug, incompatible
> Fix For: 2.9.0
>
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch,
> YARN-5203.v2.patch, YARN-5203.v3.patch, YARN-5203.v4.patch, YARN-5203.v5.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398386#comment-15398386
]
Subru Krishnan edited comment on YARN-5203 at 7/28/16 11:12 PM:
I just committed this to trunk/branch-2.
Congrats [~ellenfkh] on your first patch :)!
Thanks to [~sunilg] for the thorough whetting and [~vvasudev] for his input.
was (Author: subru):
I just committed this to trunk/branch-2.
Congrats [~ellenfkh] on your first patch :)!
Thanks to [~sunilg] for the thorough wetting and [~vvasudev] for his input.
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Labels: api-breaking, bug, incompatible
> Fix For: 2.9.0
>
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch,
> YARN-5203.v2.patch, YARN-5203.v3.patch, YARN-5203.v4.patch, YARN-5203.v5.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[
https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398122#comment-15398122
]
Subru Krishnan commented on YARN-5327:
--
Thanks [~ajsangeetha] for the patch. It mostly looks good, a few minor comments:
* We need to add more meat to the Javadocs for
{{ReservationDefinition::get/setPeriodicity}} like how we distinguish
periodic/non-periodic, implicit priority in the event of replanning,
consistency of allocation, good till cancelled aspect etc.
* Can you add an explicit default value for *period* in {{yarn_protos.proto}}.
* Maybe it makes sense to add *period* to
{{ReservationDefinitionPBImpl::toString}}.
> API changes required to support recurring reservations in the YARN
> ReservationSystem
>
>
> Key: YARN-5327
> URL: https://issues.apache.org/jira/browse/YARN-5327
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Sangeetha Abdu Jyothi
> Attachments: YARN-5327.001.patch
>
>
> YARN-5326 proposes adding native support for recurring reservations in the
> YARN ReservationSystem. This JIRA is a sub-task to track the changes needed
> in ApplicationClientProtocol to accomplish it. Please refer to the design doc
> in the parent JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State Store internal APIs
[
https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-3662:
-
Attachment: YARN-3662-YARN-2915-v6.patch
Thanks [~vinodkv] for reviewing the patch. PFA v6 which addresses your feedback.
I had to leave the SC_ prefix in the enum as otherwise I get the following
error during protobuf compilation:
bq. yarn_server_federation_protos.proto:33:3: "hadoop.yarn.NEW" is already
defined in file "yarn_protos.proto".
I recall encountering this during YARN-1051 and also {{NodeStateProto}} uses a
NS_ prefix.
> Federation Membership State Store internal APIs
> ---
>
> Key: YARN-3662
> URL: https://issues.apache.org/jira/browse/YARN-3662
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3662-YARN-2915-v1.1.patch,
> YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch,
> YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch,
> YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch,
> YARN-3662-YARN-2915-v6.patch
>
>
> The Federation Application State encapsulates the information about the
> active RM of each sub-cluster that is participating in Federation. The
> information includes addresses for ClientRM, ApplicationMaster and Admin
> services along with the sub_cluster _capability_ which is currently defined
> by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for
> further details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5351) ResourceRequest should take ExecutionType into account during comparison
[
https://issues.apache.org/jira/browse/YARN-5351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396592#comment-15396592
]
Subru Krishnan commented on YARN-5351:
--
I hit this too, have created YARN-5441 with the fix. Thanks.
> ResourceRequest should take ExecutionType into account during comparison
>
>
> Key: YARN-5351
> URL: https://issues.apache.org/jira/browse/YARN-5351
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Konstantinos Karanasos
>Assignee: Konstantinos Karanasos
> Fix For: 2.9.0
>
> Attachments: YARN-5351.001.patch, YARN-5351.002.patch
>
>
> {{ExecutionTypeRequest}} should be taken into account in the {{compareTo}}
> method of {{ResourceRequest}}.
> Otherwise, in the {{ask}} list of the {{AMRMClientImpl}} we may incorrectly
> add pending container requests in the presence of both GUARANTEED and
> OPPORTUNISTIC containers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5441) Fixing minor Scheduler test case failures
[
https://issues.apache.org/jira/browse/YARN-5441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-5441:
-
Attachment: YARN-5441-v1.patch
Attaching a patch that sets import {{ExecutionTypeRequest}} whenever
{{ResourceRequest}} is created through _BuilderUtils_.
> Fixing minor Scheduler test case failures
> -
>
> Key: YARN-5441
> URL: https://issues.apache.org/jira/browse/YARN-5441
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-5441-v1.patch
>
>
> YARN-5351 added {{ExecutionTypeRequest}} to the {{ResourceRequest}}
> comparator but the ResourceRequest object created via utility methods in few
> tests like *TestFifoScheduler* and *TestCapacityScheduler* do not set
> ExecutionTypeRequest which results in null pointer exceptions. This JIRA
> proposes a simple fix by setting ExecutionTypeRequest in the utility methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15396519#comment-15396519 ] Subru Krishnan commented on YARN-3662: -- Thanks [~vvasudev] for the sign-off. You are right, this is for branch YARN-2915. I'll commit it, just waiting to hear from [~leftnoteasy] as he had some feedback which I have also addressed. > Federation Membership State Store internal APIs > --- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch, > YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5441) Fixing minor Scheduler test case failures
Subru Krishnan created YARN-5441:
Summary: Fixing minor Scheduler test case failures
Key: YARN-5441
URL: https://issues.apache.org/jira/browse/YARN-5441
Project: Hadoop YARN
Issue Type: Bug
Reporter: Subru Krishnan
Assignee: Subru Krishnan
YARN-5351 added {{ExecutionTypeRequest}} to the {{ResourceRequest}} comparator
but the ResourceRequest object created via utility methods in few tests like
*TestFifoScheduler* and *TestCapacityScheduler* do not set
ExecutionTypeRequest which results in null pointer exceptions. This JIRA
proposes a simple fix by setting ExecutionTypeRequest in the utility methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State Store internal APIs
[
https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-3662:
-
Attachment: YARN-3662-YARN-2915-v5.patch
Thanks [~vvasudev] for the clarifications. I have addressed your feedback in v5
of the patch.
Quite a few changes (:)) so I'll list only the important ones:
* Moved the {{FederationStore}} interfaces from _API_ to _store_ sub-package.
* Added request/response objects for all methods
* Renamed request/response objects to Get/Set... to align with YARN
convention.
* Other renames which you suggested.
I have left the _capability_ as string as it is more than a _resource_ - we
need the nodes in the cluster and utilization, i.e. why we currently use the
serialized string representation of _ClusterMetricsInfo_. We can update it
later if we find a better option.
I'll also update YARN-5307/YARN-3664 similarly and post the patches tomorrow.
This JIRA is more important as blocks both.
> Federation Membership State Store internal APIs
> ---
>
> Key: YARN-3662
> URL: https://issues.apache.org/jira/browse/YARN-3662
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3662-YARN-2915-v1.1.patch,
> YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch,
> YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch,
> YARN-3662-YARN-2915-v4.patch, YARN-3662-YARN-2915-v5.patch
>
>
> The Federation Application State encapsulates the information about the
> active RM of each sub-cluster that is participating in Federation. The
> information includes addresses for ClientRM, ApplicationMaster and Admin
> services along with the sub_cluster _capability_ which is currently defined
> by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for
> further details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5392) Replace use of Priority in the Scheduling infrastructure with an opaque ShedulerRequestKey
[
https://issues.apache.org/jira/browse/YARN-5392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394949#comment-15394949
]
Subru Krishnan commented on YARN-5392:
--
Thanks [~asuresh] for driving this and [~leftnoteasy]/[~kasha] for the reviews.
I'll rebase YARN-4888 shortly and post an updated patch.
> Replace use of Priority in the Scheduling infrastructure with an opaque
> ShedulerRequestKey
> --
>
> Key: YARN-5392
> URL: https://issues.apache.org/jira/browse/YARN-5392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Fix For: 2.9.0
>
> Attachments: YARN-5392.001.patch, YARN-5392.002.patch,
> YARN-5392.003.patch, YARN-5392.004.patch, YARN-5392.005.patch,
> YARN-5392.006.patch, YARN-5392.007.patch, YARN-5392.008.patch,
> YARN-5392.009.patch, YARN-5392.010.patch
>
>
> Based on discussions in YARN-4888, this jira proposes to replace the use of
> {{Priority}} in the Scheduler infrastructure (Scheduler, Queues, SchedulerApp
> / Node etc.) with a more opaque and extensible {{SchedulerKey}}.
> Note: Even though {{SchedulerKey}} will be used by the internal scheduling
> infrastructure, It will not be exposed to the Client or the AM. The
> SchdulerKey is meant to be an internal construct that is derived from
> attributes of the ResourceRequest / ApplicationSubmissionContext / Scheduler
> Configuration etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393076#comment-15393076
]
Subru Krishnan commented on YARN-5203:
--
Thanks [~ellenfkh] for the updated patch. It LGTM, just a few minor comments:
* Rename {{testUnmarshallApp}} --> {{testUnmarshallAppInfo}}
* Add an {{ExecutionTypeRequestInfo}} JAXB object, this will fix the findbugs
warning too
* It should be ok to have {{ResourceRequestInfo::resourceRequests}} as private
[~sunilg], can you please take a look too?
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Labels: api-breaking, bug
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch,
> YARN-5203.v2.patch, YARN-5203.v3.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392934#comment-15392934
]
Subru Krishnan commented on YARN-5203:
--
Thanks [~sunilg] for the validation. Your suggestions are good, I have updated
the labels accordingly.
We can revert {{AppInfo::getResourceRequestsInfo}} back to
{{AppInfo::getResourceRequests}} now.
[~ellenfkh], can you add a test case that marshalls the response directly to
{{AppInfo}} as I feel the lack of one is the root cause of the bug.
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Labels: api-breaking, bug
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch,
> YARN-5203.v2.patch, YARN-5203.v3.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4359) Update LowCost agents logic to take advantage of YARN-4358
[
https://issues.apache.org/jira/browse/YARN-4359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15392793#comment-15392793
]
Subru Krishnan commented on YARN-4359:
--
Thanks [~imenache]/[~jyaniv] for working on this as it's a critical perf
improvement.
I apologize for the delay in reviewing. Overall the patch looks good, just a
few comments:
* An equality check, i.e. _allocationEndTime != stageStartTime_ in
{{IterativePlanner::validateOrderNoGap}} should be more general than the
present inequality ones.
* The {{StageAllocatorLowCostAligned::getDurationInterval}} is quite dense.
Consider simplifying it, especially the corner cases and/or adding more code
comments.
* Nit: You can have a separate method for computing how many gangs can fit in
{{StageAllocatorLowCostAligned::getDurationInterval}}
About the unit tests, we should have one for each change we are introducing in
the patch:
* Primary enhancement - using RLE representation instead of time interval,
this should be covered by a no regression check on existing tests which is the
case.
* Adding the ability to allocate from left to right also; I see new tests for
this.
* I do not see one that covers the fix in
{{IterativePlanner::validateOrderNoGap}}.
* We should also add one to validate the corner cases in
{{StageAllocatorLowCostAligned::getDurationInterval}} .
> Update LowCost agents logic to take advantage of YARN-4358
> --
>
> Key: YARN-4359
> URL: https://issues.apache.org/jira/browse/YARN-4359
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler, fairscheduler, resourcemanager
>Reporter: Carlo Curino
>Assignee: Ishai Menache
> Attachments: YARN-4359.0.patch, YARN-4359.10.patch,
> YARN-4359.3.patch, YARN-4359.4.patch, YARN-4359.5.patch, YARN-4359.6.patch,
> YARN-4359.7.patch, YARN-4359.8.patch, YARN-4359.9.patch
>
>
> Given the improvements of YARN-4358, the LowCost agent should be improved to
> leverage this, and operate on RLESparseResourceAllocation (ideally leveraging
> the improvements of YARN-3454 to compute avaialable resources)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-5203:
-
Labels: api-breaking bug (was: api-change bug)
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Labels: api-breaking, bug
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch,
> YARN-5203.v2.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-5203:
-
Labels: api-change bug (was: )
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Labels: api-change, bug
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch,
> YARN-5203.v2.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388622#comment-15388622
]
Subru Krishnan commented on YARN-5203:
--
[~sunilg], I think present clients are all unmarshalling as Strings as
otherwise they would hit this issue (like we did). For those clients, there is
a potential problem if they do manually parsing of the internal
_resourceRequests_ [string
contents|https://issues.apache.org/jira/browse/YARN-5203?focusedCommentId=15370535].
I (obviously) feel that this is esoteric and moreover block any clients (like
Federation Router) clients that want to use JAXB and expect proper objects
back. So I feel we should be OK to ignore this scenario. Do this make sense?
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State Store internal APIs
[
https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388595#comment-15388595
]
Subru Krishnan commented on YARN-3662:
--
Thanks [~vvasudev] for reviewing the patches.
I think there is a misunderstanding on the scope of these APIs - Federation
does *not* add any public APIs (except for an Admin API which will be done in a
separate JIRA) as the design philosophy we have followed in Federation is to be
*transparent* to the applications. So the public APIs will be continue to be
the standard YARN APIs (ApplicationClient/Master protocols).
The Federation Store APIs are for analogous to {{RMStateStore}}. Subsequently
we decided:
* To have them in yarn-server and not in yarn-api. I will add a limited
audience of YARN to make it clear.
* Minimize the wrapper request/response classes as they cause more overhead.
I can add them if you still feel it's better to have them?
Regarding your rename suggestions, I'll update the method names accordingly. I
do have a question, I prefer to have Federation as the prefix for
request/response objects rather than the op (Get.../Set...) as it makes it
easier to filter and also align with the package hierarchy. Thoughts?
The title of the JIRAs might have been misleading so I updated them to call out
that these are internal APIs.
> Federation Membership State Store internal APIs
> ---
>
> Key: YARN-3662
> URL: https://issues.apache.org/jira/browse/YARN-3662
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3662-YARN-2915-v1.1.patch,
> YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch,
> YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch,
> YARN-3662-YARN-2915-v4.patch
>
>
> The Federation Application State encapsulates the information about the
> active RM of each sub-cluster that is participating in Federation. The
> information includes addresses for ClientRM, ApplicationMaster and Admin
> services along with the sub_cluster _capability_ which is currently defined
> by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for
> further details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5307) Federation Application State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5307: - Summary: Federation Application State Store internal APIs (was: Federation Application State APIs) > Federation Application State Store internal APIs > > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3664) Federation PolicyStore internal APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3664: - Summary: Federation PolicyStore internal APIs (was: Federation PolicyStore APIs) > Federation PolicyStore internal APIs > > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch, > YARN-3664-YARN-2915-v1.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State Store internal APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Summary: Federation Membership State Store internal APIs (was: Federation Membership State APIs) > Federation Membership State Store internal APIs > --- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch, > YARN-3662-YARN-2915-v4.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3927) Make the NodeManager's ContainerManager pluggable
[ https://issues.apache.org/jira/browse/YARN-3927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan resolved YARN-3927. -- Resolution: Workaround We ended up adding AMRMProxy as a first class service in NM. > Make the NodeManager's ContainerManager pluggable > - > > Key: YARN-3927 > URL: https://issues.apache.org/jira/browse/YARN-3927 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > > YARN-2884 proposes proxying all AM-RM communication for: > * perform distributed scheduling decisions (YARN-2877) > * throttling mis-behaving AMs > * mask the access to a federation of RMs (YARN-2915) > To enable all of the above, we are implementing the AMRMProxy as an extension > to NM's ContainerManagerImpl.This JIRA is for making the ContainerManager > pluggable so that we allow dynamically swap in of the AMRMProxy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5413) Create a proxy chain for ResourceManager Admin API in the Router
[ https://issues.apache.org/jira/browse/YARN-5413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5413: - Assignee: Giovanni Matteo Fumarola > Create a proxy chain for ResourceManager Admin API in the Router > > > Key: YARN-5413 > URL: https://issues.apache.org/jira/browse/YARN-5413 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > > As detailed in the proposal in the umbrella JIRA, we are introducing a new > component that routes client request to appropriate ResourceManager(s). This > JIRA tracks the creation of a proxy for ResourceManager Admin API in the > Router. This provides a placeholder for: > 1) throttling mis-behaving clients (YARN-1546) > 3) mask the access to multiple RMs (YARN-3659) > We are planning to follow the interceptor pattern like we did in YARN-2884 to > generalize the approach and have only dynamically coupling for Federation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5413) Create a proxy chain for ResourceManager Admin API in the Router
Subru Krishnan created YARN-5413: Summary: Create a proxy chain for ResourceManager Admin API in the Router Key: YARN-5413 URL: https://issues.apache.org/jira/browse/YARN-5413 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan As detailed in the proposal in the umbrella JIRA, we are introducing a new component that routes client request to appropriate ResourceManager(s). This JIRA tracks the creation of a proxy for ResourceManager Admin API in the Router. This provides a placeholder for: 1) throttling mis-behaving clients (YARN-1546) 3) mask the access to multiple RMs (YARN-3659) We are planning to follow the interceptor pattern like we did in YARN-2884 to generalize the approach and have only dynamically coupling for Federation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5412) Create a proxy chain for ResourceManager REST API in the Router
Subru Krishnan created YARN-5412: Summary: Create a proxy chain for ResourceManager REST API in the Router Key: YARN-5412 URL: https://issues.apache.org/jira/browse/YARN-5412 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Giovanni Matteo Fumarola As detailed in the proposal in the umbrella JIRA, we are introducing a new component that routes client request to appropriate ResourceManager(s). This JIRA tracks the creation of a proxy for ResourceManager REST API in the Router. This provides a placeholder for: 1) throttling mis-behaving clients (YARN-1546) 3) mask the access to multiple RMs (YARN-3659) We are planning to follow the interceptor pattern like we did in YARN-2884 to generalize the approach and have only dynamically coupling for Federation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5411) Create a proxy chain for ApplicationClientProtocol in the Router
[ https://issues.apache.org/jira/browse/YARN-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5411: - Description: As detailed in the proposal in the umbrella JIRA, we are introducing a new component that routes client request to appropriate ResourceManager(s). This JIRA tracks the creation of a proxy for ApplicationClientProtocol in the Router. This provides a placeholder for: 1) throttling mis-behaving clients (YARN-1546) 3) mask the access to multiple RMs (YARN-3659) We are planning to follow the interceptor pattern like we did in YARN-2884 to generalize the approach and have only dynamically coupling for Federation. was:As detailed in the proposal in the umbrella JIRA, we are introducing a new component that routes client request to appropriate ResourceManager(s). This JIRA tracks the creation of a proxy for ApplicationClientProtocol in the Router. > Create a proxy chain for ApplicationClientProtocol in the Router > > > Key: YARN-5411 > URL: https://issues.apache.org/jira/browse/YARN-5411 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > > As detailed in the proposal in the umbrella JIRA, we are introducing a new > component that routes client request to appropriate ResourceManager(s). This > JIRA tracks the creation of a proxy for ApplicationClientProtocol in the > Router. This provides a placeholder for: > 1) throttling mis-behaving clients (YARN-1546) > 3) mask the access to multiple RMs (YARN-3659) > We are planning to follow the interceptor pattern like we did in YARN-2884 to > generalize the approach and have only dynamically coupling for Federation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5411) Create a proxy chain for ApplicationClientProtocol in the Router
[ https://issues.apache.org/jira/browse/YARN-5411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5411: - Summary: Create a proxy chain for ApplicationClientProtocol in the Router (was: Create a proxy for ApplicationClientProtocol in the Router) > Create a proxy chain for ApplicationClientProtocol in the Router > > > Key: YARN-5411 > URL: https://issues.apache.org/jira/browse/YARN-5411 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Giovanni Matteo Fumarola > > As detailed in the proposal in the umbrella JIRA, we are introducing a new > component that routes client request to appropriate ResourceManager(s). This > JIRA tracks the creation of a proxy for ApplicationClientProtocol in the > Router. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5411) Create a proxy for ApplicationClientProtocol in the Router
Subru Krishnan created YARN-5411: Summary: Create a proxy for ApplicationClientProtocol in the Router Key: YARN-5411 URL: https://issues.apache.org/jira/browse/YARN-5411 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Giovanni Matteo Fumarola As detailed in the proposal in the umbrella JIRA, we are introducing a new component that routes client request to appropriate ResourceManager(s). This JIRA tracks the creation of a proxy for ApplicationClientProtocol in the Router. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5410) Bootstrap Router module
Subru Krishnan created YARN-5410: Summary: Bootstrap Router module Key: YARN-5410 URL: https://issues.apache.org/jira/browse/YARN-5410 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Giovanni Matteo Fumarola As detailed in the proposal in the umbrella JIRA, we are introducing a new component that routes client request to appropriate ResourceManager(s). This JIRA tracks the creation of a new sub-module for the Router. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2915) Enable YARN RM scale out via federation using multiple RM's
[ https://issues.apache.org/jira/browse/YARN-2915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-2915: - Attachment: YARN-Federation-Hadoop-Summit_final.pptx Attaching the slide deck from our Hadoop summit talk as that's a good primer for quick reference. > Enable YARN RM scale out via federation using multiple RM's > --- > > Key: YARN-2915 > URL: https://issues.apache.org/jira/browse/YARN-2915 > Project: Hadoop YARN > Issue Type: New Feature > Components: nodemanager, resourcemanager >Reporter: Sriram Rao >Assignee: Subru Krishnan > Attachments: FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf, > Federation-BoF.pdf, YARN-Federation-Hadoop-Summit_final.pptx, > Yarn_federation_design_v1.pdf, federation-prototype.patch > > > This is an umbrella JIRA that proposes to scale out YARN to support large > clusters comprising of tens of thousands of nodes. That is, rather than > limiting a YARN managed cluster to about 4k in size, the proposal is to > enable the YARN managed cluster to be elastically scalable. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5407) In-memory based implementation of the FederationApplicationStateStore
[ https://issues.apache.org/jira/browse/YARN-5407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5407: - Summary: In-memory based implementation of the FederationApplicationStateStore (was: In-memory based implementation of the FederationMembershipStateStore) > In-memory based implementation of the FederationApplicationStateStore > - > > Key: YARN-5407 > URL: https://issues.apache.org/jira/browse/YARN-5407 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Ellen Hui > > YARN-5307 defines the FederationApplicationStateStore API. This JIRA tracks > an in-memory based implementation which is useful for both single-box testing > and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5408) In-memory based implementation of the FederationPolicyStore
Subru Krishnan created YARN-5408: Summary: In-memory based implementation of the FederationPolicyStore Key: YARN-5408 URL: https://issues.apache.org/jira/browse/YARN-5408 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Ellen Hui YARN-3664 defines the FederationPolicyStore API. This JIRA tracks an in-memory based implementation which is useful for both single-box testing and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5407) In-memory based implementation of the FederationMembershipStateStore
Subru Krishnan created YARN-5407: Summary: In-memory based implementation of the FederationMembershipStateStore Key: YARN-5407 URL: https://issues.apache.org/jira/browse/YARN-5407 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Ellen Hui YARN-5307 defines the FederationApplicationStateStore API. This JIRA tracks an in-memory based implementation which is useful for both single-box testing and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5406) In-memory based implementation of the FederationMembershipStateStore
Subru Krishnan created YARN-5406: Summary: In-memory based implementation of the FederationMembershipStateStore Key: YARN-5406 URL: https://issues.apache.org/jira/browse/YARN-5406 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Ellen Hui YARN-3662 defines the FederationMembershipStateStore API. This JIRA tracks an in-memory based implementation which is useful for both single-box testing and for future unit tests that depend on the state store. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Attachment: YARN-3662-YARN-2915-v4.patch Minor update (v4) with fixes for checkstyle issues that can be addressed. > Federation Membership State APIs > > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch, > YARN-3662-YARN-2915-v4.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5390) Federation Subcluster Resolver
[ https://issues.apache.org/jira/browse/YARN-5390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5390: - Assignee: Ellen Hui > Federation Subcluster Resolver > -- > > Key: YARN-5390 > URL: https://issues.apache.org/jira/browse/YARN-5390 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Ellen Hui > > This JIRA tracks effort to create a mechanism to resolve nodes/racks resource > names to sub-cluster identifiers. This is needed by the federation policies > in YARN-5323, YARN-5324, YARN-5325 to operate correctly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3664) Federation PolicyStore APIs
[
https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-3664:
-
Attachment: YARN-3664-YARN-2915-v1.patch
Updated patch (v1) that incorporates [~leftnoteasy]'s
[feedback|https://issues.apache.org/jira/browse/YARN-3662?focusedCommentId=15375947]:
* Included {{FederationPolicyStore}} API class. I had missed it in previous
patch, good catch.
* Renamed record classes and updated Javadoc (with help from [~curino]) to
make it more understandable.
For more context, kindly refer to [~curino]'s
[summary|https://issues.apache.org/jira/browse/YARN-5323?focusedCommentId=15380907]
and associated policy patches in YARN-5323.
> Federation PolicyStore APIs
> ---
>
> Key: YARN-3664
> URL: https://issues.apache.org/jira/browse/YARN-3664
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3664-YARN-2915-v0.patch,
> YARN-3664-YARN-2915-v1.patch
>
>
> The federation Policy Store contains information about the capacity
> allocations made by users, their mapping to sub-clusters and the policies
> that each of the components (Router, AMRMPRoxy, RMs) should enforce
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5203) Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API
[
https://issues.apache.org/jira/browse/YARN-5203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385099#comment-15385099
]
Subru Krishnan commented on YARN-5203:
--
[~sunilg], thanks for taking a look and bringing up the important point on
compatibility.
IIUC unfortunately we cannot continue to have the raw RRs as JAXB will not be
able to unmarshal directly to {{AppInfo}} object. Today it works because
existing clients like UI deserialize directly to _String_ which should continue
to work even if we use JAXB object for RRs.
[~ellenfkh], can you kindly do a quick validation as you have a setup where you
already have been testing extensively. Thanks!
> Return ResourceRequest JAXB object in ResourceManager Cluster Applications
> REST API
> ---
>
> Key: YARN-5203
> URL: https://issues.apache.org/jira/browse/YARN-5203
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Subru Krishnan
>Assignee: Ellen Hui
> Attachments: YARN-5203.v0.patch, YARN-5203.v1.patch
>
>
> The ResourceManager Cluster Applications REST API returns {{ResourceRequest}}
> as String rather than a JAXB object. This prevents downstream tools like
> Federation Router (YARN-3659) that depend on the REST API to unmarshall the
> {{AppInfo}}. This JIRA proposes updating {{AppInfo}} to return a JAXB version
> of the {{ResourceRequest}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5392) Replace use of Priority in the Scheduling infrastructure with an opaque ShedulerKey
[
https://issues.apache.org/jira/browse/YARN-5392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385055#comment-15385055
]
Subru Krishnan commented on YARN-5392:
--
Thanks [~asuresh] for working on this.
I just have one question, should we have the {{SchedulerKey}} in addition to
{{Priority}}?
I feel {{Priority}} should be accessible directly as before outside of the
scheduler layers and the notion of {{SchedulerKey}} should be confined to the
scheduler (ideally should be transparent to other RM entities/services). An
extreme example would be that in future we could decide not to use {{Priority}}
as a {{SchedulerKey}} at some point in the future.
Overall the patch LGTM. Since I have been working very closely with you;
[~kasha]/[~leftnoteasy], can you guys take a look.
> Replace use of Priority in the Scheduling infrastructure with an opaque
> ShedulerKey
> ---
>
> Key: YARN-5392
> URL: https://issues.apache.org/jira/browse/YARN-5392
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Arun Suresh
> Attachments: YARN-5392.001.patch, YARN-5392.002.patch,
> YARN-5392.003.patch
>
>
> Based on discussions in YARN-4888, this jira proposes to replace the use of
> {{Priority}} in the Scheduler infrastructure (Scheduler, Queues, SchedulerApp
> / Node etc.) with a more opaque and extensible {{SchedulerKey}}.
> Note: Even though {{SchedulerKey}} will be used by the internal scheduling
> infrastructure, It will not be exposed to the Client or the AM. The
> SchdulerKey is meant to be an internal construct that is derived from
> attributes of the ResourceRequest / ApplicationSubmissionContext / Scheduler
> Configuration etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5350) Ensure LocalScheduler does not lose the sort order of allocatable nodes returned by the RM
[ https://issues.apache.org/jira/browse/YARN-5350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15385047#comment-15385047 ] Subru Krishnan commented on YARN-5350: -- [~asuresh], +1 for the fix. A minor feedback for the test: * it would be good to add a assertion on the total number of OPPORTUNISTIC containers. * add another check to ensure sort order is maintained by doing a second round of allocation and/or request for multiple OPPORTUNISTIC containers. Thanks. > Ensure LocalScheduler does not lose the sort order of allocatable nodes > returned by the RM > -- > > Key: YARN-5350 > URL: https://issues.apache.org/jira/browse/YARN-5350 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0 > > Attachments: YARN-5350.001.patch, YARN-5350.002.patch > > > The LocalScheduler receives an ordered list of nodes from the RM with each > allocate call. This list, which is used by the LocalScheduler to allocate > OPPORTUNISTIC containers, is sorted on the Nodes free capacity (queue length > / wait time). > Unfortunately, the LocalScheduler stores this list in a HashMap thereby > losing the sort order. > The trivial fix would be to replace the HashMap with a LinkedHashMap which > retains the insertion order. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Attachment: YARN-3662-YARN-2915-v3.01.patch Reattaching patch after rebasing _YARN-2915_ branch to pull in HADOOP-13342 > Federation Membership State APIs > > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch, > YARN-3662-YARN-2915-v3.01.patch, YARN-3662-YARN-2915-v3.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5307) Federation Application State APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5307: - Attachment: YARN-5307-YARN-2915-v2.patch Updated patch (v2) that incorporates [~leftnoteasy]'s feedback as described [here|https://issues.apache.org/jira/browse/YARN-3662]. > Federation Application State APIs > - > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch, > YARN-5307-YARN-2915-v2.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3662) Federation Membership State APIs
[
https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15383428#comment-15383428
]
Subru Krishnan edited comment on YARN-3662 at 7/19/16 1:16 AM:
---
Thanks [~leftnoteasy] for reviewing the APIs. I am attaching an updated patch
(v3) that addresses your concerns on membership API, specifically:
* Renaming {{FederationMembershipState --> FederationMembershipStateStore}}
as this is the equivalent of {{RMStateStore}} for Federation.
* Added an explicit {{FederationSubClusterHeartbeatRequest}}
* Renamed all the _address_ fields in {{FederationSubClusterInfo}} to
_serviceAddress_
I'll also update YARN-5307 by renaming {{FederationApplicationState -->
FederationApplicationStateStore}}.
Regarding your question about the Policy API, Carlo has uploaded the initial
version of the Policies in YARN-5323 with a
[summary|https://issues.apache.org/jira/browse/YARN-5323?focusedCommentId=15380907]
of how they will be used in Federation. We are updating the API Javadoc in
YARN-3664 to increase clarity.
was (Author: subru):
Thanks [~leftnoteasy] for reviewing the APIs. I am attaching an updated patch
(v3) that addresses your concerns on membership API, specifically:
* Renaming {{FederationMembershipState --> FederationMembershipStateStore}}
as this is the equivalent of {{RMStateStore}} for Federation.
* Added an explicit {{FederationSubClusterHeartbeatRequest}}
* Renamed all the _address_ fields in {{FederationSubClusterInfo}} to
_serviceAddress_
I'll also update YARN-5367 by renaming {{FederationApplicationState -->
FederationApplicationStateStore}}.
Regarding your question about the Policy API, Carlo has uploaded the initial
version of the Policies in YARN-5323 with a
[summary|https://issues.apache.org/jira/browse/YARN-5323?focusedCommentId=15380907]
of how they will be used in Federation. We are updating the API Javadoc in
YARN-3664 to increase clarity.
> Federation Membership State APIs
>
>
> Key: YARN-3662
> URL: https://issues.apache.org/jira/browse/YARN-3662
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3662-YARN-2915-v1.1.patch,
> YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch,
> YARN-3662-YARN-2915-v3.patch
>
>
> The Federation Application State encapsulates the information about the
> active RM of each sub-cluster that is participating in Federation. The
> information includes addresses for ClientRM, ApplicationMaster and Admin
> services along with the sub_cluster _capability_ which is currently defined
> by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for
> further details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State APIs
[
https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-3662:
-
Attachment: YARN-3662-YARN-2915-v3.patch
Thanks [~leftnoteasy] for reviewing the APIs. I am attaching an updated patch
(v3) that addresses your concerns on membership API, specifically:
* Renaming {{FederationMembershipState --> FederationMembershipStateStore}}
as this is the equivalent of {{RMStateStore}} for Federation.
* Added an explicit {{FederationSubClusterHeartbeatRequest}}
* Renamed all the _address_ fields in {{FederationSubClusterInfo}} to
_serviceAddress_
I'll also update YARN-5367 by renaming {{FederationApplicationState -->
FederationApplicationStateStore}}.
Regarding your question about the Policy API, Carlo has uploaded the initial
version of the Policies in YARN-5323 with a
[summary|https://issues.apache.org/jira/browse/YARN-5323?focusedCommentId=15380907]
of how they will be used in Federation. We are updating the API Javadoc in
YARN-3664 to increase clarity.
> Federation Membership State APIs
>
>
> Key: YARN-3662
> URL: https://issues.apache.org/jira/browse/YARN-3662
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-3662-YARN-2915-v1.1.patch,
> YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch,
> YARN-3662-YARN-2915-v3.patch
>
>
> The Federation Application State encapsulates the information about the
> active RM of each sub-cluster that is participating in Federation. The
> information includes addresses for ClientRM, ApplicationMaster and Admin
> services along with the sub_cluster _capability_ which is currently defined
> by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for
> further details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5261) Lease/Reclaim Extension to Yarn
[ https://issues.apache.org/jira/browse/YARN-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380427#comment-15380427 ] Subru Krishnan edited comment on YARN-5261 at 7/16/16 2:16 AM: --- [~y] thanks for the clarifications. I understand there are slight nuances for your use case but generally it's good to consolidate related efforts as otherwise we will have an explosion of specialized features in YARN (already see a trend). IIUC, this is what [~jlowe] and [~sunilg] are both alluding to. We do have the ability to reserve cluster resources as part of YARN-1051/YARN-2572/YARN-2573. Would that be sufficient or do you need to be able to do it at node level? If the latter is the case, you can consider achieving it in combination of reservation system with node labels (YARN-4193), intuition is each node that can be leased has a label (which can be it's name). Another option is to have LeaseManager run outside of YARN _initially_ and use the dynamic node resource configuration (YARN-291) to affect the change you want. Thoughts? was (Author: subru): @yuuu' thanks for the clarifications. I understand there are slight nuances for your use case but generally it's good to consolidate related efforts as otherwise we will have an explosion of specialized features in YARN (already see a trend). IIUC, this is what [~jlowe] and [~sunilg] are both alluding to. We do have the ability to reserve cluster resources as part of YARN-1051/YARN-2572/YARN-2573. Would that be sufficient or do you need to be able to do it at node level? If the latter is the case, you can consider achieving it in combination of reservation system with node labels (YARN-4193), intuition is each node that can be leased has a label (which can be it's name). Another option is to have LeaseManager run outside of YARN initially and use the dynamic node resource configuration (YARN-291) to affect the change you want. Thoughts? > Lease/Reclaim Extension to Yarn > --- > > Key: YARN-5261 > URL: https://issues.apache.org/jira/browse/YARN-5261 > Project: Hadoop YARN > Issue Type: New Feature > Components: scheduler >Reporter: Yu > Attachments: YARN-5261-1.patch, YARN-5261-2.patch, YARN-5261-3.patch, > Yarn-5261.pdf > > > In some clusters outside of Yarn, the machines resources are not fully > utilized, e.g., resource usage is quite low at night. > To better utilize the resources while keep the existing SLA of the cluster, > Lease/Reclaim Extension to Yarn is introduced. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5261) Lease/Reclaim Extension to Yarn
[ https://issues.apache.org/jira/browse/YARN-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15380427#comment-15380427 ] Subru Krishnan commented on YARN-5261: -- @yuuu' thanks for the clarifications. I understand there are slight nuances for your use case but generally it's good to consolidate related efforts as otherwise we will have an explosion of specialized features in YARN (already see a trend). IIUC, this is what [~jlowe] and [~sunilg] are both alluding to. We do have the ability to reserve cluster resources as part of YARN-1051/YARN-2572/YARN-2573. Would that be sufficient or do you need to be able to do it at node level? If the latter is the case, you can consider achieving it in combination of reservation system with node labels (YARN-4193), intuition is each node that can be leased has a label (which can be it's name). Another option is to have LeaseManager run outside of YARN initially and use the dynamic node resource configuration (YARN-291) to affect the change you want. Thoughts? > Lease/Reclaim Extension to Yarn > --- > > Key: YARN-5261 > URL: https://issues.apache.org/jira/browse/YARN-5261 > Project: Hadoop YARN > Issue Type: New Feature > Components: scheduler >Reporter: Yu > Attachments: YARN-5261-1.patch, YARN-5261-2.patch, YARN-5261-3.patch, > Yarn-5261.pdf > > > In some clusters outside of Yarn, the machines resources are not fully > utilized, e.g., resource usage is quite low at night. > To better utilize the resources while keep the existing SLA of the cluster, > Lease/Reclaim Extension to Yarn is introduced. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM AppSchedulingInfo for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-4888:
-
Attachment: YARN-4888-WIP.patch
Thanks [~asuresh] for taking a look at the v0 patch and updating it to add a
{{SchedulerKey}} to define a composite key for scheduler requests. This makes
the patch bigger (:)) but is more cleaner as we can now address YARN-314.
Currently we have {{Priority}} and {{allocationRequestId}} (will default to _0_
for backward compatibility) as the composite keys. This changes also enables
future extensions for the {{SchedulerKey}}.
> Changes in RM AppSchedulingInfo for identifying resource-requests explicitly
>
>
> Key: YARN-4888
> URL: https://issues.apache.org/jira/browse/YARN-4888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4888-WIP.patch, YARN-4888-v0.patch
>
>
> YARN-4879 puts forward the notion of identifying allocate requests
> explicitly. This JIRA is to track the changes in RM app scheduling data
> structures to accomplish it. Please refer to the design doc in the parent
> JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-4888) Changes in RM AppSchedulingInfo for identifying resource-requests explicitly
[
https://issues.apache.org/jira/browse/YARN-4888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Subru Krishnan updated YARN-4888:
-
Attachment: YARN-4888-v0.patch
I am uploading a WIP patch as the change is pretty involved. This patch adds an
_allocationRequestId_ key in {{AppSchedulingInfo.resourceRequestMap}} and maps
the allocated container against the _ResourceRequest_ which it satisfies.
I have got the existing test cases for _Capacity/Fair/Fifo_ schedulers working
*except* for the ones related to node labels. I am looking into those,
meanwhile it would be good to review the patch to make sure the approach I have
taken is sound as the changes affect core parts of RM.
I also need to add tests with _ResourceRequest_ that use the new
_allocationRequestId_ feature.
> Changes in RM AppSchedulingInfo for identifying resource-requests explicitly
>
>
> Key: YARN-4888
> URL: https://issues.apache.org/jira/browse/YARN-4888
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
> Attachments: YARN-4888-v0.patch
>
>
> YARN-4879 puts forward the notion of identifying allocate requests
> explicitly. This JIRA is to track the changes in RM app scheduling data
> structures to accomplish it. Please refer to the design doc in the parent
> JIRA for details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5283) Refactor container assignment into AbstractYarnScheduler#assignContainers
[
https://issues.apache.org/jira/browse/YARN-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15368720#comment-15368720
]
Subru Krishnan commented on YARN-5283:
--
Thanks [~rchiang] for driving this effort, overall I think it'll be very useful.
The changes I am making as part of YARN-4888 are in
{{reserve}}/{{getContainer}} in _Capacity/Fair/Fifo_ schedulers (even there
looks like quite similar code).
Regarding the patch - is there an opportunity to move common logic to
{{AbstractYarnScheduler::assignContainers}} like the
{{isReadyToAssignContainers}} check?
> Refactor container assignment into AbstractYarnScheduler#assignContainers
> -
>
> Key: YARN-5283
> URL: https://issues.apache.org/jira/browse/YARN-5283
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler, fairscheduler, resourcemanager,
> scheduler
>Reporter: Ray Chiang
>Assignee: Ray Chiang
> Attachments: YARN-5283.001.patch
>
>
> CapacityScheduler#allocateContainersToNode() and
> FairScheduler#attemptScheduling() have some common code that can be
> refactored into a common abstract method like
> AbstractYarnScheduler#assignContainers().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5327: - Description: YARN-5326 proposes adding native support for recurring reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes needed in ApplicationClientProtocol to accomplish it. Please refer to the design doc in the parent JIRA for details. (was: YARN-5326 proposes adding native support for recurring reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes in ApplicationClientProtocol to accomplish it. Please refer to the design doc in the parent JIRA for details.) > API changes required to support recurring reservations in the YARN > ReservationSystem > > > Key: YARN-5327 > URL: https://issues.apache.org/jira/browse/YARN-5327 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Sangeetha Abdu Jyothi > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes needed > in ApplicationClientProtocol to accomplish it. Please refer to the design doc > in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5331) Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem
Subru Krishnan created YARN-5331: Summary: Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem Key: YARN-5331 URL: https://issues.apache.org/jira/browse/YARN-5331 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Subru Krishnan Assignee: Sangeetha Abdu Jyothi YARN-5326 proposes adding native support for recurring reservations in the YARN ReservationSystem. This JIRA is a sub-task to add a PeriodicRLESparseResourceAllocation. Please refer to the design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5330) SharingPolicy enhancements required to support recurring reservations in the YARN ReservationSystem
Subru Krishnan created YARN-5330: Summary: SharingPolicy enhancements required to support recurring reservations in the YARN ReservationSystem Key: YARN-5330 URL: https://issues.apache.org/jira/browse/YARN-5330 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Subru Krishnan Assignee: Sangeetha Abdu Jyothi YARN-5326 proposes adding native support for recurring reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes required in SharingPolicy to accomplish it. Please refer to the design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5329) ReservationAgent enhancements required to support recurring reservations in the YARN ReservationSystem
Subru Krishnan created YARN-5329: Summary: ReservationAgent enhancements required to support recurring reservations in the YARN ReservationSystem Key: YARN-5329 URL: https://issues.apache.org/jira/browse/YARN-5329 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Subru Krishnan Assignee: Sangeetha Abdu Jyothi YARN-5326 proposes adding native support for recurring reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes required in ReservationAgent to accomplish it. Please refer to the design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5328) InMemoryPlan enhancements required to support recurring reservations in the YARN ReservationSystem
Subru Krishnan created YARN-5328: Summary: InMemoryPlan enhancements required to support recurring reservations in the YARN ReservationSystem Key: YARN-5328 URL: https://issues.apache.org/jira/browse/YARN-5328 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Subru Krishnan Assignee: Sangeetha Abdu Jyothi YARN-5326 proposes adding native support for recurring reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes required in InMemoryPlan to accomplish it. Please refer to the design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5327) API changes required to support recurring reservations in the YARN ReservationSystem
Subru Krishnan created YARN-5327: Summary: API changes required to support recurring reservations in the YARN ReservationSystem Key: YARN-5327 URL: https://issues.apache.org/jira/browse/YARN-5327 Project: Hadoop YARN Issue Type: Sub-task Components: resourcemanager Reporter: Subru Krishnan Assignee: Sangeetha Abdu Jyothi YARN-5326 proposes adding native support for recurring reservations in the YARN ReservationSystem. This JIRA is a sub-task to track the changes in ApplicationClientProtocol to accomplish it. Please refer to the design doc in the parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5326) Add support for recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5326: - Description: YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle time explicitly, i.e. users can now "reserve" capacity ahead of time which is predictably allocated to them. Most SLA jobs/workflows are recurring so they need the same resources periodically. With the current implementation, users will have to make individual reservations for each run. This is an umbrella JIRA to enhance the reservation system by adding native support for recurring reservations. (was: YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle time explicitly, i.e. users can now "reserve" capacity ahead of time which is predictably allocated to them. Most SLA jobs are recurring so they need the same resources periodically. With the current implementation, users will have to make individual reservations for each run. This is an umbrella JIRA to enhance the reservation system by adding native support for recurring reservations.) > Add support for recurring reservations in the YARN ReservationSystem > > > Key: YARN-5326 > URL: https://issues.apache.org/jira/browse/YARN-5326 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Subru Krishnan > Attachments: SupportRecurringReservationsInRayon.pdf > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. Most SLA jobs/workflows are recurring so they > need the same resources periodically. With the current implementation, users > will have to make individual reservations for each run. This is an umbrella > JIRA to enhance the reservation system by adding native support for recurring > reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5326) Add support for recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5326: - Attachment: SupportRecurringReservationsInRayon.pdf Attaching a design doc with our proposal to enhance ReservationSytem with native support for recurring jobs/workflows. > Add support for recurring reservations in the YARN ReservationSystem > > > Key: YARN-5326 > URL: https://issues.apache.org/jira/browse/YARN-5326 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Subru Krishnan > Attachments: SupportRecurringReservationsInRayon.pdf > > > YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle > time explicitly, i.e. users can now "reserve" capacity ahead of time which is > predictably allocated to them. Most SLA jobs are recurring so they need the > same resources periodically. With the current implementation, users will have > to make individual reservations for each run. This is an umbrella JIRA to > enhance the reservation system by adding native support for recurring > reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5326) Add support for recurring reservations in the YARN ReservationSystem
Subru Krishnan created YARN-5326: Summary: Add support for recurring reservations in the YARN ReservationSystem Key: YARN-5326 URL: https://issues.apache.org/jira/browse/YARN-5326 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Reporter: Subru Krishnan YARN-1051 introduced a ReservationSytem that enables the YARN RM to handle time explicitly, i.e. users can now "reserve" capacity ahead of time which is predictably allocated to them. Most SLA jobs are recurring so they need the same resources periodically. With the current implementation, users will have to make individual reservations for each run. This is an umbrella JIRA to enhance the reservation system by adding native support for recurring reservations. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15360676#comment-15360676 ] Subru Krishnan commented on YARN-3662: -- I have fixed all the checkstyle issues (ran test-patch locally apriori) and only reported ones are related to: * methods have public modifiers (this patch has API only, implementation will follow in YARN-3663...). * _FederationSubClusterInfo::newInstance_ having more than 7 params * one API signature that exceed 80 characters. I feel it is safe to ignore these. > Federation Membership State APIs > > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5307) Federation Application State APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5307: - Attachment: YARN-5307-YARN-2915-v1.patch Uploading v1 of the *FederationApplicationState* APIs. The patch is available for review but is not ready for CI as it depends on YARN-3662. I'll add the tests for the app state protocol records post commit of YARN-3662. > Federation Application State APIs > - > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5307-YARN-2915-v1.patch > > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Attachment: YARN-3662-YARN-2915-v2.patch Uploading v2 patch that contains only the *FederationMembershipState* APIs as mentioned above. > Federation Membership State APIs > > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch, YARN-3662-YARN-2915-v2.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3662) Federation Membership State APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359895#comment-15359895 ] Subru Krishnan commented on YARN-3662: -- Splitting the JIRA into 2 to allow for more manageable review/feedback cycle: * Current one which is for adding *FederationMembershipState*. * YARN-5307 for adding *FederationApplicationState*. > Federation Membership State APIs > > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Description: The Federation Application State encapsulates the information about the active RM of each sub-cluster that is participating in Federation. The information includes addresses for ClientRM, ApplicationMaster and Admin services along with the sub_cluster _capability_ which is currently defined by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for further details. was:The Federation State defines the additional state that needs to be maintained to loosely couple multiple individual sub-clusters into a single large federated cluster > Federation Membership State APIs > > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch > > > The Federation Application State encapsulates the information about the > active RM of each sub-cluster that is participating in Federation. The > information includes addresses for ClientRM, ApplicationMaster and Admin > services along with the sub_cluster _capability_ which is currently defined > by *ClusterMetricsInfo*. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation Membership State APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Summary: Federation Membership State APIs (was: Federation StateStore APIs) > Federation Membership State APIs > > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch > > > The Federation State defines the additional state that needs to be maintained > to loosely couple multiple individual sub-clusters into a single large > federated cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5307) Federation Application State APIs
[ https://issues.apache.org/jira/browse/YARN-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5307: - Description: The Federation Application State encapsulates the mapping between an application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is submitted to by the Router. Please refer to the design doc in parent JIRA for further details. (was: The Federation State defines the additional state that needs to be maintained to loosely couple multiple individual sub-clusters into a single large federated cluster) > Federation Application State APIs > - > > Key: YARN-5307 > URL: https://issues.apache.org/jira/browse/YARN-5307 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > > The Federation Application State encapsulates the mapping between an > application and it's _home_ sub-cluster, i.e. the sub-cluster to which it is > submitted to by the Router. Please refer to the design doc in parent JIRA for > further details. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5307) Federation Application State APIs
Subru Krishnan created YARN-5307: Summary: Federation Application State APIs Key: YARN-5307 URL: https://issues.apache.org/jira/browse/YARN-5307 Project: Hadoop YARN Issue Type: Sub-task Components: nodemanager, resourcemanager Reporter: Subru Krishnan Assignee: Subru Krishnan The Federation State defines the additional state that needs to be maintained to loosely couple multiple individual sub-clusters into a single large federated cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2664) Improve RM webapp to expose info about reservations.
[ https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15359633#comment-15359633 ] Subru Krishnan commented on YARN-2664: -- Assigning to [~elgoiri] as this has been idle for a while and [~elgoiri] is keen on taking this up for 2.8. Thanks! > Improve RM webapp to expose info about reservations. > > > Key: YARN-2664 > URL: https://issues.apache.org/jira/browse/YARN-2664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Inigo Goiri > Labels: BB2015-05-TBR > Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch, > YARN-2664.10.patch, YARN-2664.11.patch, YARN-2664.12.patch, > YARN-2664.2.patch, YARN-2664.3.patch, YARN-2664.4.patch, YARN-2664.5.patch, > YARN-2664.6.patch, YARN-2664.7.patch, YARN-2664.8.patch, YARN-2664.9.patch, > YARN-2664.patch, legal.patch, screenshot_reservation_UI.pdf > > > YARN-1051 provides a new functionality in the RM to ask for reservation on > resources. Exposing this through the webapp GUI is important. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2664) Improve RM webapp to expose info about reservations.
[ https://issues.apache.org/jira/browse/YARN-2664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-2664: - Assignee: Inigo Goiri (was: Matteo Mazzucchelli) > Improve RM webapp to expose info about reservations. > > > Key: YARN-2664 > URL: https://issues.apache.org/jira/browse/YARN-2664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Inigo Goiri > Labels: BB2015-05-TBR > Attachments: PlannerPage_screenshot.pdf, YARN-2664.1.patch, > YARN-2664.10.patch, YARN-2664.11.patch, YARN-2664.2.patch, YARN-2664.3.patch, > YARN-2664.4.patch, YARN-2664.5.patch, YARN-2664.6.patch, YARN-2664.7.patch, > YARN-2664.8.patch, YARN-2664.9.patch, YARN-2664.patch, legal.patch, > screenshot_reservation_UI.pdf > > > YARN-1051 provides a new functionality in the RM to ask for reservation on > resources. Exposing this through the webapp GUI is important. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation StateStore APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Attachment: YARN-3662-YARN-2915-v1.1.patch Attaching v1.1 after rebasing with YARN-5300 commit. > Federation StateStore APIs > -- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.1.patch, > YARN-3662-YARN-2915-v1.patch > > > The Federation State defines the additional state that needs to be maintained > to loosely couple multiple individual sub-clusters into a single large > federated cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3664) Federation PolicyStore APIs
[ https://issues.apache.org/jira/browse/YARN-3664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3664: - Attachment: YARN-3664-YARN-2915-v0.patch Attaching v1 of the Federation PolicyStore APIs. The APIs provide for a _generic_ mechanism to map the scheduler hierarchical queues across multiple YARN (sub)clusters. Please refer to the [design doc|https://issues.apache.org/jira/secure/attachment/12744104/FEDERATION_CAPACITY_ALLOCATION_JIRA.pdf] in the parent JIRA (YARN-2915) for further details. The patch is available for review but is not ready for CI as it depends on YARN-3662. > Federation PolicyStore APIs > --- > > Key: YARN-3664 > URL: https://issues.apache.org/jira/browse/YARN-3664 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3664-YARN-2915-v0.patch > > > The federation Policy Store contains information about the capacity > allocations made by users, their mapping to sub-clusters and the policies > that each of the components (Router, AMRMPRoxy, RMs) should enforce -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-3662) Federation StateStore APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352410#comment-15352410 ] Subru Krishnan edited comment on YARN-3662 at 6/28/16 6:00 AM: --- Attaching v1 of the Federation StateStore APIs. The APIs include: * _FederationMembershipState_ which encapsulates the YARN (sub)clusters that are participating in Federation. * _FederationApplicationState_ which is mapping between the Application and it's home (sub)cluster. Please refer to the [design doc|https://issues.apache.org/jira/secure/attachment/12733292/Yarn_federation_design_v1.pdf] in the parent JIRA (YARN-2915) for further details. was (Author: subru): Attaching v1 of the Federation StateStore APIs. > Federation StateStore APIs > -- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.patch > > > The Federation State defines the additional state that needs to be maintained > to loosely couple multiple individual sub-clusters into a single large > federated cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3662) Federation StateStore APIs
[ https://issues.apache.org/jira/browse/YARN-3662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-3662: - Attachment: YARN-3662-YARN-2915-v1.patch Attaching v1 of the Federation StateStore APIs. > Federation StateStore APIs > -- > > Key: YARN-3662 > URL: https://issues.apache.org/jira/browse/YARN-3662 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-3662-YARN-2915-v1.patch > > > The Federation State defines the additional state that needs to be maintained > to loosely couple multiple individual sub-clusters into a single large > federated cluster -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5300) Exclude generated federation protobuf sources from YARN Javadoc/findbugs build
[
https://issues.apache.org/jira/browse/YARN-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15352401#comment-15352401
]
Subru Krishnan commented on YARN-5300:
--
Thanks [~curino] for the review. The test case failure is unrelated as the
changes in the patch are confined to excluding generated federation protobuf
sources from YARN Javadoc/findbugs build.
I have committed to {{YARN-2915}} branch.
> Exclude generated federation protobuf sources from YARN Javadoc/findbugs build
> --
>
> Key: YARN-5300
> URL: https://issues.apache.org/jira/browse/YARN-5300
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager, resourcemanager
>Reporter: Subru Krishnan
>Assignee: Subru Krishnan
>Priority: Minor
> Attachments: YARN-5300-v1.patch, YARN-5300-v2.patch
>
>
> This JIRA is the equivalent of YARN-5132 for generated federation protobuf
> sources.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5300) Exclude generated federation protobuf sources from YARN Javadoc/findbugs build
[ https://issues.apache.org/jira/browse/YARN-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5300: - Summary: Exclude generated federation protobuf sources from YARN Javadoc/findbugs build (was: Exclude generated federation protobuf sources from YARN Javadoc build) > Exclude generated federation protobuf sources from YARN Javadoc/findbugs build > -- > > Key: YARN-5300 > URL: https://issues.apache.org/jira/browse/YARN-5300 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan >Priority: Minor > Attachments: YARN-5300-v1.patch, YARN-5300-v2.patch > > > This JIRA is the equivalent of YARN-5132 for generated federation protobuf > sources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5300) Exclude generated federation protobuf sources from YARN Javadoc build
[ https://issues.apache.org/jira/browse/YARN-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5300: - Attachment: YARN-5300-v2.patch Uploading v2 patch which fixes multiple package exclusion and adds findbug exclusion for generated federation source files > Exclude generated federation protobuf sources from YARN Javadoc build > - > > Key: YARN-5300 > URL: https://issues.apache.org/jira/browse/YARN-5300 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan >Priority: Minor > Attachments: YARN-5300-v1.patch, YARN-5300-v2.patch > > > This JIRA is the equivalent of YARN-5132 for generated federation protobuf > sources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5300) Exclude generated federation protobuf sources from YARN Javadoc build
[ https://issues.apache.org/jira/browse/YARN-5300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5300: - Attachment: YARN-5300-v1.patch Attaching a simple patch that excludes generated federation protobuf sources. This doesn't have any test cases as the modifications are limited to pom file. > Exclude generated federation protobuf sources from YARN Javadoc build > - > > Key: YARN-5300 > URL: https://issues.apache.org/jira/browse/YARN-5300 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan >Priority: Minor > Attachments: YARN-5300-v1.patch > > > This JIRA is the equivalent of YARN-5132 for generated federation protobuf > sources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-5300) Exclude generated federation protobuf sources from YARN Javadoc build
Subru Krishnan created YARN-5300: Summary: Exclude generated federation protobuf sources from YARN Javadoc build Key: YARN-5300 URL: https://issues.apache.org/jira/browse/YARN-5300 Project: Hadoop YARN Issue Type: Sub-task Reporter: Subru Krishnan Assignee: Subru Krishnan Priority: Minor This JIRA is the equivalent of YARN-5132 for generated federation protobuf sources. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5171) Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node
[
https://issues.apache.org/jira/browse/YARN-5171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15340875#comment-15340875
]
Subru Krishnan commented on YARN-5171:
--
I had an offline discussion with [~jianhe] and he agreed that option (2) is the
best option.
I would suggest keeping the method names as {{addRMContainer}} and
{{removeRMContainer}} as external container seems confusing with the
discussions going on in YARN-5215. You could open another JIRA to clean up code
to use the new methods instead of directly updating {{liveContainers}}.
> Extend DistributedSchedulerProtocol to notify RM of containers allocated by
> the Node
>
>
> Key: YARN-5171
> URL: https://issues.apache.org/jira/browse/YARN-5171
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: Inigo Goiri
> Attachments: YARN-5171.000.patch, YARN-5171.001.patch,
> YARN-5171.002.patch, YARN-5171.003.patch, YARN-5171.004.patch,
> YARN-5171.005.patch, YARN-5171.006.patch, YARN-5171.007.patch
>
>
> Currently, the RM does not know about Containers allocated by the
> OpportunisticContainerAllocator on the NM. This JIRA proposes to extend the
> Distributed Scheduler request interceptor and the protocol to notify the RM
> of new containers as and when they are allocated at the NM. The
> {{RMContainer}} should also be extended to expose the {{ExecutionType}} of
> the container.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
