[jira] [Comment Edited] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302521#comment-16302521 ] Arun Suresh edited comment on YARN-7613 at 12/23/17 5:56 PM: - Thanks for the patch [~pgaref] Couple of comments: * It looks like the temporary tags might be a common requirement for all / most algorithms. I propose we modify the place() method to pass in a temp Tags manager. In the placementDispacher, where we call the place method, we can create a temp tagsmanager and pass it to the algorithm - maybe call it a 'scratch' tags manager. Maybe, this can be a subclass of the TagsManager which just exposes an addTempTag and removeTempTag. I also noticed in the TagsManager, that we don't really need both the containerId and applicationId to be passed into the addContainer method. since the applicationId is derivable from the containerId. Also, the containerId is not exactly used in the method (we can fix this in another JIRA). * So I think we should keep the canAssign, private (or maybe have a util class that exposes a static canAssign that takes the containerManager, tagsManager, request, applicationId etc and returns boolean). The Algorithm, IMO, should expose only a single method in its interface. I am also thinking maybe we remove the init method also. and pass an AlgorithmContext in the place method, that encapsulates the TagsManager, ConstraintManager, the scratch TagsManager and the NodeCandidateSelector. Thoughts ? was (Author: asuresh): Thanks for the patch [~pgaref] Couple of comments: * It looks like the temporary tags for the might be a common requirement for all / most algorithms. I propose we modify the place() method to pass in a temp Tags manager. In the placementDispacher, where we call the place method, we can create a temp tagsmanager and pass it to the algorithm - maybe call it a 'scratch' tags manager. Maybe, this can be a subclass of the TagsManager which just exposes an addTempTag and removeTempTag. I also noticed in the TagsManager, that we don't really need both the containerId and applicationId to be passed into the addContainer method. since the applicationId is derivable from the containerId. Also, the containerId is not exactly used in the method (we can fix this in another JIRA). * So I think we should keep the canAssign, private (or maybe have a util class that exposes a static canAssign that takes the containerManager, tagsManager, request, applicationId etc and returns boolean). The Algorithm, IMO, should expose only a single method in its interface. I am also thinking maybe we remove the init method also. and pass an AlgorithmContext in the place method, that encapsulates the TagsManager, ConstraintManager, the scratch TagsManager and the NodeCandidateSelector. Thoughts ? > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302521#comment-16302521 ] Arun Suresh commented on YARN-7613: --- Thanks for the patch [~pgaref] Couple of comments: * It looks like the temporary tags for the might be a common requirement for all / most algorithms. I propose we modify the place() method to pass in a temp Tags manager. In the placementDispacher, where we call the place method, we can create a temp tagsmanager and pass it to the algorithm - maybe call it a 'scratch' tags manager. Maybe, this can be a subclass of the TagsManager which just exposes an addTempTag and removeTempTag. I also noticed in the TagsManager, that we don't really need both the containerId and applicationId to be passed into the addContainer method. since the applicationId is derivable from the containerId. Also, the containerId is not exactly used in the method (we can fix this in another JIRA). * So I think we should keep the canAssign, private (or maybe have a util class that exposes a static canAssign that takes the containerManager, tagsManager, request, applicationId etc and returns boolean). The Algorithm, IMO, should expose only a single method in its interface. I am also thinking maybe we remove the init method also. and pass an AlgorithmContext in the place method, that encapsulates the TagsManager, ConstraintManager, the scratch TagsManager and the NodeCandidateSelector. Thoughts ? > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302467#comment-16302467 ] Panagiotis Garefalakis edited comment on YARN-7613 at 12/23/17 2:40 PM: Attaching a proof of concept patch (applied cleanly on top of YARN-7612) that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class (debatable). Planning Algorithm should only add logic to the way we iterate through SchedulingRequests and cluster Nodes and not how we satisfy the constraints - this is the first step towards that goal was (Author: pgaref): Attaching a proof of concept patch (applied cleanly on top of YARN-7612) that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302467#comment-16302467 ] Panagiotis Garefalakis edited comment on YARN-7613 at 12/23/17 2:38 PM: Attaching a proof of concept patch (applied cleanly on top of YARN-7612) that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class was (Author: pgaref): Attaching a proof of concept patch (applied cleanly on top of YARN-7612) that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class but it should be a separate JIRA > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302467#comment-16302467 ] Panagiotis Garefalakis edited comment on YARN-7613 at 12/23/17 2:37 PM: Attaching a proof of concept patch that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class but it should be a separate JIRA was (Author: pgaref): Attaching a proof of concept patch that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302467#comment-16302467 ] Panagiotis Garefalakis edited comment on YARN-7613 at 12/23/17 2:37 PM: Attaching a proof of concept patch (applied cleanly on top of YARN-7612) that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class but it should be a separate JIRA was (Author: pgaref): Attaching a proof of concept patch that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class but it should be a separate JIRA > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7613) Implement Planning algorithms for rich placement
[ https://issues.apache.org/jira/browse/YARN-7613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Panagiotis Garefalakis updated YARN-7613: - Attachment: YARN-7613.wip.patch Attaching a proof of concept patch that decouples Algorithm from constraints introducing canAssign method which navigates thought placement constraints and just return if a placement in a Node is feasible or not. Currently as part of the algorithm class > Implement Planning algorithms for rich placement > > > Key: YARN-7613 > URL: https://issues.apache.org/jira/browse/YARN-7613 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Panagiotis Garefalakis > Attachments: YARN-7613.wip.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Improve handling of the Docker container life cycle
[
https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16302461#comment-16302461
]
genericqa commented on YARN-5366:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 53s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
4s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
7s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
19s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32
[jira] [Updated] (YARN-5366) Improve handling of the Docker container life cycle
[
https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf updated YARN-5366:
--
Attachment: YARN-5366.009.patch
> Improve handling of the Docker container life cycle
> ---
>
> Key: YARN-5366
> URL: https://issues.apache.org/jira/browse/YARN-5366
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Labels: oct16-medium
> Attachments: YARN-5366.001.patch, YARN-5366.002.patch,
> YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch,
> YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch,
> YARN-5366.009.patch
>
>
> There are several paths that need to be improved with regard to the Docker
> container lifecycle when running Docker containers on YARN.
> 1) Provide the ability to keep a container on the NodeManager for a set
> period of time for debugging purposes.
> 2) Support sending signals to the process in the container to allow for
> triggering stack traces, heap dumps, etc.
> 3) Support for Docker's live restore, which means moving away from the use of
> {{docker wait}}. (YARN-5818)
> 4) Improve the resiliency of liveliness checks (kill -0) by adding retries.
> 5) Improve the resiliency of container removal by adding retries.
> 6) Only attempt to stop, kill, and remove containers if the current container
> state allows for it.
> 7) Better handling of short lived containers when the container is stopped
> before the PID can be retrieved. (YARN-6305)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5366) Improve handling of the Docker container life cycle
[
https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shane Kumpf updated YARN-5366:
--
Attachment: (was: YARN-5366.009.patch)
> Improve handling of the Docker container life cycle
> ---
>
> Key: YARN-5366
> URL: https://issues.apache.org/jira/browse/YARN-5366
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Shane Kumpf
>Assignee: Shane Kumpf
> Labels: oct16-medium
> Attachments: YARN-5366.001.patch, YARN-5366.002.patch,
> YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch,
> YARN-5366.006.patch, YARN-5366.007.patch, YARN-5366.008.patch,
> YARN-5366.009.patch
>
>
> There are several paths that need to be improved with regard to the Docker
> container lifecycle when running Docker containers on YARN.
> 1) Provide the ability to keep a container on the NodeManager for a set
> period of time for debugging purposes.
> 2) Support sending signals to the process in the container to allow for
> triggering stack traces, heap dumps, etc.
> 3) Support for Docker's live restore, which means moving away from the use of
> {{docker wait}}. (YARN-5818)
> 4) Improve the resiliency of liveliness checks (kill -0) by adding retries.
> 5) Improve the resiliency of container removal by adding retries.
> 6) Only attempt to stop, kill, and remove containers if the current container
> state allows for it.
> 7) Better handling of short lived containers when the container is stopped
> before the PID can be retrieved. (YARN-6305)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
