[jira] [Comment Edited] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502377#comment-15502377 ] Jian He edited comment on YARN-5621 at 9/19/16 5:49 AM: Hi [~chris.douglas], I had tried to implement with your approach. However, this approach will not work in rollback scenario, as in that case no resources need to be localized - hence, no need to start the localizer processes. We only need to update the symlinks to old resources. Overall, do you think the latest patch is fine was (Author: jianhe): Hi [~chris.douglas], I had tried to implement with your approach. However, this approach will not work in rollback scenario, as in that case no resources need to be localized - hence, no need to start the localizer processes. We only need to update the symlinks to old resources. Overall, do you think the current approach is fine > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502377#comment-15502377 ] Jian He edited comment on YARN-5621 at 9/19/16 5:47 AM: Hi [~chris.douglas], I had tried to implement with your approach. However, this approach will not work in rollback scenario, as in that case no resources need to be localized - hence, no need to start the localizer processes. We only need to update the symlinks to old resources. Overall, do you think the current approach is fine was (Author: jianhe): Hi [~chris.douglas], I had tried to implement with your approach. However, this approach will not work in rollback scenario, as in that case no resources need to be localized - hence, no need to start the localizer processes. We only need to update the symlinks to old resources. Overall, do you think the current approach is fine > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5621) Support LinuxContainerExecutor to create symlinks for continuously localized resources
[ https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502377#comment-15502377 ] Jian He commented on YARN-5621: --- Hi [~chris.douglas], I had tried to implement with your approach. However, this approach will not work in rollback scenario, as in that case no resources need to be localized - hence, no need to start the localizer processes. We only need to update the symlinks to old resources. Overall, do you think the current approach is fine > Support LinuxContainerExecutor to create symlinks for continuously localized > resources > -- > > Key: YARN-5621 > URL: https://issues.apache.org/jira/browse/YARN-5621 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, > YARN-5621.4.patch, YARN-5621.5.patch > > > When new resources are localized, new symlink needs to be created for the > localized resource. This is the change for the LinuxContainerExecutor to > create the symlinks. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3692) Allow REST API to set a user generated message when killing an application
[
https://issues.apache.org/jira/browse/YARN-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S updated YARN-3692:
Attachment: 0006-YARN-3692.patch
Updated patch fixing first 2 comments. RMAdminCli and Web UI support can be
done in separate JIRA.
> Allow REST API to set a user generated message when killing an application
> --
>
> Key: YARN-3692
> URL: https://issues.apache.org/jira/browse/YARN-3692
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Rajat Jain
>Assignee: Rohith Sharma K S
> Attachments: 0001-YARN-3692.patch, 0002-YARN-3692.patch,
> 0003-YARN-3692.patch, 0004-YARN-3692.patch, 0005-YARN-3692.1.patch,
> 0005-YARN-3692.patch, 0006-YARN-3692.patch
>
>
> Currently YARN's REST API supports killing an application without setting a
> diagnostic message. It would be good to provide that support.
> *Use Case*
> Usually this helps in workflow management in a multi-tenant environment when
> the workflow scheduler (or the hadoop admin) wants to kill a job - and let
> the user know the reason why the job was killed. Killing the job by setting a
> diagnostic message is a very good solution for that. Ideally, we can set the
> diagnostic message on all such interface:
> yarn kill -applicationId ... -diagnosticMessage "some message added by
> admin/workflow"
> REST API { 'state': 'KILLED', 'diagnosticMessage': 'some message added by
> admin/workflow'}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3140) Improve locks in AbstractCSQueue/LeafQueue/ParentQueue
[ https://issues.apache.org/jira/browse/YARN-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502282#comment-15502282 ] Jian He commented on YARN-3140: --- [~leftnoteasy], the patch doesn't apply any more, could you rebase? > Improve locks in AbstractCSQueue/LeafQueue/ParentQueue > -- > > Key: YARN-3140 > URL: https://issues.apache.org/jira/browse/YARN-3140 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-3140.1.patch, YARN-3140.2.patch, YARN-3140.3.patch > > > Enhance locks in AbstractCSQueue/LeafQueue/ParentQueue, as mentioned in > YARN-3091, a possible solution is using read/write lock. Other fine-graind > locks for specific purposes / bugs should be addressed in separated tickets. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5609) Expose upgrade and restart API in ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502265#comment-15502265 ] Arun Suresh commented on YARN-5609: --- Yup... thats the way I plan to implement it... > Expose upgrade and restart API in ContainerManagementProtocol > - > > Key: YARN-5609 > URL: https://issues.apache.org/jira/browse/YARN-5609 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > > YARN-5620 and YARN-5637 allows an AM to explicitly *upgrade* a container with > a new launch context and subsequently *rollback* / *commit* the change on the > Container. This can also be used to simply *restart* the Container as well. > This JIRA proposes to extend the ContainerManagementProtocol with the > following API: > * *upgradeContainer* > * *rollbackLastUpgrade* > * *commitLastUpgrade* > * *restartContainer* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5609) Expose upgrade and restart API in ContainerManagementProtocol
[ https://issues.apache.org/jira/browse/YARN-5609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502259#comment-15502259 ] Jian He commented on YARN-5609: --- one question, what is the difference between restartContainer and upgradeContainer ? is restartContainer the same as upgradeContainer without new launchcontext. > Expose upgrade and restart API in ContainerManagementProtocol > - > > Key: YARN-5609 > URL: https://issues.apache.org/jira/browse/YARN-5609 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > > YARN-5620 and YARN-5637 allows an AM to explicitly *upgrade* a container with > a new launch context and subsequently *rollback* / *commit* the change on the > Container. This can also be used to simply *restart* the Container as well. > This JIRA proposes to extend the ContainerManagementProtocol with the > following API: > * *upgradeContainer* > * *rollbackLastUpgrade* > * *commitLastUpgrade* > * *restartContainer* -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5631) Missing refreshClusterMaxPriority usage in rmadmin help message
[
https://issues.apache.org/jira/browse/YARN-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15502150#comment-15502150
]
Rohith Sharma K S commented on YARN-5631:
-
Thanks [~lewuathe] for the patch.. patch looks good, one nit.
it seems patch has format issue in RMAdminCLI.java class. All the lines have
changed, but you have added only one line. Could it get fixed?
> Missing refreshClusterMaxPriority usage in rmadmin help message
> ---
>
> Key: YARN-5631
> URL: https://issues.apache.org/jira/browse/YARN-5631
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.0.0-alpha2
>Reporter: Kai Sasaki
>Assignee: Kai Sasaki
>Priority: Minor
> Attachments: YARN-5631-branch-2.8.01.patch,
> YARN-5631-branch-2.8.02.patch, YARN-5631-branch-2.8.03.patch,
> YARN-5631.01.patch, YARN-5631.02.patch
>
>
> {{rmadmin -help}} does not show {{-refreshClusterMaxPriority}} option in
> usage line.
> {code}
> $ bin/yarn rmadmin -help
> rmadmin is the command to execute YARN administrative commands.
> The full syntax is:
> yarn rmadmin [-refreshQueues] [-refreshNodes [-g|graceful [timeout in
> seconds] -client|server]] [-refreshNodesResources]
> [-refreshSuperUserGroupsConfiguration] [-refreshUserToGroupsMappings]
> [-refreshAdminAcls] [-refreshServiceAcl] [-getGroup [username]]
> [-addToClusterNodeLabels
> <"label1(exclusive=true),label2(exclusive=false),label3">]
> [-removeFromClusterNodeLabels ] [-replaceLabelsOnNode
> <"node1[:port]=label1,label2 node2[:port]=label1">]
> [-directlyAccessNodeLabelStore] [-updateNodeResource [NodeID] [MemSize]
> [vCores] ([OvercommitTimeout]) [-help [cmd]]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5587) Add support for resource profiles
[
https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501876#comment-15501876
]
Hadoop QA commented on YARN-5587:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 11 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
41s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 46s
{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 57s
{color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
28s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
25s {color} | {color:green} YARN-3926 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} YARN-3926 passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 6m 43s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 43s {color}
| {color:red} root generated 3 new + 714 unchanged - 0 fixed = 717 total (was
714) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 39s
{color} | {color:red} root: The patch generated 66 new + 1058 unchanged - 3
fixed = 1124 total (was 1061) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 4s
{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 18s
{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api generated
4 new + 156 unchanged - 0 fixed = 160 total (was 156) {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 27s {color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 23s {color}
| {color:red} hadoop-yarn-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 12m 56s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 55s {color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 2s {color} |
{color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 118m 15s
{color} | {color:green} hadoop-mapreduce-client-jobclient in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 32s
{color} | {color:red} The patch generated 5 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 232m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Null pointer dereference of ? in
[jira] [Updated] (YARN-4855) Should check if node exists when replace nodelabels
[ https://issues.apache.org/jira/browse/YARN-4855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tao Jie updated YARN-4855: -- Attachment: YARN-4855.010.patch > Should check if node exists when replace nodelabels > --- > > Key: YARN-4855 > URL: https://issues.apache.org/jira/browse/YARN-4855 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Tao Jie >Assignee: Tao Jie >Priority: Minor > Attachments: YARN-4855.001.patch, YARN-4855.002.patch, > YARN-4855.003.patch, YARN-4855.004.patch, YARN-4855.005.patch, > YARN-4855.006.patch, YARN-4855.007.patch, YARN-4855.008.patch, > YARN-4855.009.patch, YARN-4855.010.patch > > > Today when we add nodelabels to nodes, it would succeed even if nodes are not > existing NodeManger in cluster without any message. > It could be like this: > When we use *yarn rmadmin -replaceLabelsOnNode --fail-on-unkown-nodes > "node1=label1"* , it would be denied if node is unknown. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5587) Add support for resource profiles
[ https://issues.apache.org/jira/browse/YARN-5587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-5587: Attachment: YARN-5587-YARN-3926.002.patch > Add support for resource profiles > - > > Key: YARN-5587 > URL: https://issues.apache.org/jira/browse/YARN-5587 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev > Attachments: YARN-5587-YARN-3926.001.patch, > YARN-5587-YARN-3926.002.patch > > > Add support for resource profiles on the RM side to allow users to use > shorthands to specify resource requirements. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5637) Changes in NodeManager to support Container rollback and commit
[ https://issues.apache.org/jira/browse/YARN-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501408#comment-15501408 ] Hudson commented on YARN-5637: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10458 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10458/]) YARN-5637. Changes in NodeManager to support Container rollback and (arun suresh: rev 3552c2b99dff4f21489ff284f9dcba40e897a1e5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerReInitEvent.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerEventType.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManager.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestContainerManagerWithLCE.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/Container.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/MockContainer.java > Changes in NodeManager to support Container rollback and commit > --- > > Key: YARN-5637 > URL: https://issues.apache.org/jira/browse/YARN-5637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Fix For: 2.9.0, 3.0.0-alpha2 > > Attachments: YARN-5637.001.patch, YARN-5637.002.patch, > YARN-5637.003.patch, YARN-5637.004.patch, YARN-5637.005.patch, > YARN-5637.006.patch, YARN-5637.007.patch > > > YARN-5620 added support for re-initialization of Containers using a new > launch Context. > This JIRA proposes to use the above feature to support upgrade and subsequent > rollback or commit of the upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5636) Support reserving resources on certain nodes for certain applications
[ https://issues.apache.org/jira/browse/YARN-5636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501286#comment-15501286 ] Tao Jie commented on YARN-5636: --- We did some trial like this: # Add fields to ResourceRequest, ContainerRequest: #* String reservedLabel, to recognize those containers that resources reserved to (actually is slider app name) #* boolean useReserved, tell the RM that whether this container needs to use reserved resource (true when it is a restarted container) # When completing container on node, it will mark that resource as *reserved resource* if this container has a reservedLabel # When assigning resource to container on node, it will check available resource on node respect to reserved resource. # Also reserved resource would release when it expires. [~Naganarasimha], [~sunilg], [~leftnoteasy], Like to hear your thoughts. > Support reserving resources on certain nodes for certain applications > - > > Key: YARN-5636 > URL: https://issues.apache.org/jira/browse/YARN-5636 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler >Reporter: Tao Jie > > We have met such circumstance: > We are trying to run storm on yarn by Slider, and Storm writes > data to local disk on node. If some containers or the application fails, we > expect that those containers would restart on the same node as they run > before, otherwise data written on local would lost. > For slider, it will trying to ensure restarted container on same nodes as > before. However for yarn, resource may be assigned to other applications when > former long-running application is down. > As a result we'd better to have a mechanism that reserve some resource for > certain long-running applications on certain nodes for a period of time. Does > it make sense? -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5637) Changes in NodeManager to support Container rollback and commit
[ https://issues.apache.org/jira/browse/YARN-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501090#comment-15501090 ] Varun Vasudev commented on YARN-5637: - Makes sense. +1. > Changes in NodeManager to support Container rollback and commit > --- > > Key: YARN-5637 > URL: https://issues.apache.org/jira/browse/YARN-5637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5637.001.patch, YARN-5637.002.patch, > YARN-5637.003.patch, YARN-5637.004.patch, YARN-5637.005.patch, > YARN-5637.006.patch, YARN-5637.007.patch > > > YARN-5620 added support for re-initialization of Containers using a new > launch Context. > This JIRA proposes to use the above feature to support upgrade and subsequent > rollback or commit of the upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5637) Changes in NodeManager to support Container rollback and commit
[ https://issues.apache.org/jira/browse/YARN-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501051#comment-15501051 ] Arun Suresh edited comment on YARN-5637 at 9/18/16 2:22 PM: Thanks for the review [~vvasudev]. We don't really delete the new ResourceSet when we rollback. To clarify the scenario you mentioned, As part of the upgrade a rogue jar is symlinked into the Container's working directory and added to its CLASSPATH env variable. I see 2 cases: # The rogue jar's symlink has over-written the earlier version's symlink. In this case, on rollback, the symlink will be re over-written with the old jar and the relaunched old process will see only the old jar. # The rogue jar's is a new symlink in the container's working dir. In this case, on rollback, the symlink will still remain in the working dir and still point to the old jar. But since the CLASSPATH will also be reverted to the old CLASSPATH (since it's part of the launch context) which does not contain this new jar... I am guessing it should be fine. But yeah, we can maybe add more sophistication to actually delete / unlink the new new resources from the working directory on rollback. Let's handle that, as you suggested as part of another JIRA, and lets continue with the current approach as a first cut. Hope this made sense. If so, I will go ahead and commit this. was (Author: asuresh): Thanks for the review [~vvasudev]. We don't really delete the new ResourceSet when we rollback. To clarify the scenario you mentioned, As part of the upgrade a rogue jar is symlinked into the Container's working directory and added to its CLASSPATH env variable. I see 2 cases: # The rogue jar's symlink has over-written the earlier version's symlink. In this case, on rollback, the symlink will be re over-written with the old jar and the relaunched old process will see only the old jar. # The rogue jar's is a new symlink in the container's working dir. In this case, on rollback, the symlink will still remain in the working dir and still point to the old jar. But since the CLASSPATH will also be reverted to the old CLASSPATH which does not contain this new jar... I am guess it should be fine. But yeah, we can maybe add more sophistication to actually delete / unlink the new new resources from the working directory on rollback. Let's handle that, as you suggested as part of another JIRA, and lets continue with the current approach as a first cut. Hope this made sense. If so, I will go ahead and commit this. > Changes in NodeManager to support Container rollback and commit > --- > > Key: YARN-5637 > URL: https://issues.apache.org/jira/browse/YARN-5637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5637.001.patch, YARN-5637.002.patch, > YARN-5637.003.patch, YARN-5637.004.patch, YARN-5637.005.patch, > YARN-5637.006.patch, YARN-5637.007.patch > > > YARN-5620 added support for re-initialization of Containers using a new > launch Context. > This JIRA proposes to use the above feature to support upgrade and subsequent > rollback or commit of the upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5637) Changes in NodeManager to support Container rollback and commit
[ https://issues.apache.org/jira/browse/YARN-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15501051#comment-15501051 ] Arun Suresh commented on YARN-5637: --- Thanks for the review [~vvasudev]. We don't really delete the new ResourceSet when we rollback. To clarify the scenario you mentioned, As part of the upgrade a rogue jar is symlinked into the Container's working directory and added to its CLASSPATH env variable. I see 2 cases: # The rogue jar's symlink has over-written the earlier version's symlink. In this case, on rollback, the symlink will be re over-written with the old jar and the relaunched old process will see only the old jar. # The rogue jar's is a new symlink in the container's working dir. In this case, on rollback, the symlink will still remain in the working dir and still point to the old jar. But since the CLASSPATH will also be reverted to the old CLASSPATH which does not contain this new jar... I am guess it should be fine. But yeah, we can maybe add more sophistication to actually delete / unlink the new new resources from the working directory on rollback. Let's handle that, as you suggested as part of another JIRA, and lets continue with the current approach as a first cut. Hope this made sense. If so, I will go ahead and commit this. > Changes in NodeManager to support Container rollback and commit > --- > > Key: YARN-5637 > URL: https://issues.apache.org/jira/browse/YARN-5637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5637.001.patch, YARN-5637.002.patch, > YARN-5637.003.patch, YARN-5637.004.patch, YARN-5637.005.patch, > YARN-5637.006.patch, YARN-5637.007.patch > > > YARN-5620 added support for re-initialization of Containers using a new > launch Context. > This JIRA proposes to use the above feature to support upgrade and subsequent > rollback or commit of the upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5366) Add support for toggling the removal of completed and failed docker containers
[ https://issues.apache.org/jira/browse/YARN-5366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500955#comment-15500955 ] Varun Vasudev commented on YARN-5366: - The latest patch looks good to me. [~aw] can I commit the latest patch? It looks like [~shaneku...@gmail.com] has addressed your comments. > Add support for toggling the removal of completed and failed docker containers > -- > > Key: YARN-5366 > URL: https://issues.apache.org/jira/browse/YARN-5366 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Shane Kumpf >Assignee: Shane Kumpf > Attachments: YARN-5366.001.patch, YARN-5366.002.patch, > YARN-5366.003.patch, YARN-5366.004.patch, YARN-5366.005.patch, > YARN-5366.006.patch > > > Currently, completed and failed docker containers are removed by > container-executor. Add a job level environment variable to > DockerLinuxContainerRuntime to allow the user to toggle whether they want the > container deleted or not and remove the logic from container-executor. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5637) Changes in NodeManager to support Container rollback and commit
[ https://issues.apache.org/jira/browse/YARN-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500546#comment-15500546 ] Varun Vasudev commented on YARN-5637: - Thanks for the patch [~asuresh]! My apologies for the delay in responding. The patch looks mostly good to me. The only question I have to around deleting resources as part of the rollback. Let's say you download a new jar as part of the upgrade. When you do a rollback, this jar will be part of the classpath and this could result in the rollback failing(due to class/function not found issues). Is my understanding wrong? I wouldn't hold up the patch for fixing this - we should probably do it in a different JIRA. > Changes in NodeManager to support Container rollback and commit > --- > > Key: YARN-5637 > URL: https://issues.apache.org/jira/browse/YARN-5637 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-5637.001.patch, YARN-5637.002.patch, > YARN-5637.003.patch, YARN-5637.004.patch, YARN-5637.005.patch, > YARN-5637.006.patch, YARN-5637.007.patch > > > YARN-5620 added support for re-initialization of Containers using a new > launch Context. > This JIRA proposes to use the above feature to support upgrade and subsequent > rollback or commit of the upgrade. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
