[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523739#comment-14523739 ] Xuan Gong commented on YARN-1897: - [~mingma] we need to sync up with this. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Assignee: Ming Ma Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897-4.patch, YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14002875#comment-14002875 ] Gera Shegalov commented on YARN-1897: - I am confused, [~mingma]. I thought we agreed to do it as YARN-1515. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Assignee: Ming Ma Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897-4.patch, YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[
https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14003009#comment-14003009
]
Hadoop QA commented on YARN-1897:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12645735/YARN-1897-4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3771//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3771//console
This message is automatically generated.
Define SignalContainerRequest and SignalContainerResponse
-
Key: YARN-1897
URL: https://issues.apache.org/jira/browse/YARN-1897
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Ming Ma
Assignee: Ming Ma
Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897-4.patch,
YARN-1897.1.patch
We need to define SignalContainerRequest and SignalContainerResponse first as
they are needed by other sub tasks. SignalContainerRequest should use
OS-independent commands and provide a way to application to specify reason
for diagnosis. SignalContainerResponse might be empty.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14004166#comment-14004166 ] Ming Ma commented on YARN-1897: --- Chatted with Gera offline. The definition of SignalContainer* APIs is needed for other subtasks including YARN-1515. So we will resolve SignalContainer* APIs issues in this jira. After it is done, other subtasks can continue. Here are couple open issues. 1. Support for a list of containers. The latest patch in this jira just supports a flat list of signalContainerRequest, regardless if they are from the same containers or not. Gera's patch in YARN-1515 groups all commands under the same container together via signalContainerRequest.getSignals(). Either approach works. I don't have strong preference either way given the most common use case is for single container; although signalContainers is more consistent with startContainers. 2. Support for SIGTERM + delay + SIGKILL used in stopContainers. Latest YARN-1515 introduces Pause method so that containers can pause in between signals. We need something like that to support YARN-1515 scenario. Or we can provide some new SignalContainerCommand like sleep. Really appreciate any comments on this. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Assignee: Ming Ma Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897-4.patch, YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980273#comment-13980273 ] Xuan Gong commented on YARN-1897: - bq. 1. SignalContainerResponse has a flag to indicate the request was submitted successfully. If it fails, the application doesn't know why. Is that the diagnosis string for? Previous patch just throws exception. Yes. just throwing an exception should be fine. I just uploaded a new patch with a little modification. I removed @see ApplicationClientProtocol#signalContainer from both SignalContainerRequest and SignalContainerResponse. Right now, we do not have signalContainer in ApplicationClientProtocol. To make a clean build, I just remove them first. We can add them back later. The other javadoc warning is unrelated. Will create a spearate ticket for it. Also, I marked @unstable to SignalContainerResponse API instead of @stable. think this api might be changed later. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980343#comment-13980343 ] Ming Ma commented on YARN-1897: --- Looks good. Thanks, Xuan. SignalContainerResponse still has the flag defined. We can leave it like that and you can take it out later if necessary as part of https://issues.apache.org/jira/browse/YARN-1804. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13980383#comment-13980383 ] Xuan Gong commented on YARN-1897: - Cool. Let us wait for a couple of days. If there is no further comments, we can commit it. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Attachments: YARN-1897-2.patch, YARN-1897-3.patch, YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976268#comment-13976268 ] Xuan Gong commented on YARN-1897: - [~mingma] I uploaded an initial patch for this. Please take a look and feel free to do any editions, renaming, etc. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Attachments: YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976310#comment-13976310 ] Ming Ma commented on YARN-1897: --- Thanks, Xuan. I will merge this one with the version I have and provide an update shortly. BTW, why does SignalContainerResponse needs to provide diagnosis string, to explain why the request can't be processed? Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma Attachments: YARN-1897.1.patch We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[
https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976408#comment-13976408
]
Hadoop QA commented on YARN-1897:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12641175/YARN-1897-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
See
https://builds.apache.org/job/PreCommit-YARN-Build/3605//artifact/trunk/patchprocess/diffJavadocWarnings.txt
for details.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3605//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3605//console
This message is automatically generated.
Define SignalContainerRequest and SignalContainerResponse
-
Key: YARN-1897
URL: https://issues.apache.org/jira/browse/YARN-1897
Project: Hadoop YARN
Issue Type: Sub-task
Components: api
Reporter: Ming Ma
Attachments: YARN-1897-2.patch, YARN-1897.1.patch
We need to define SignalContainerRequest and SignalContainerResponse first as
they are needed by other sub tasks. SignalContainerRequest should use
OS-independent commands and provide a way to application to specify reason
for diagnosis. SignalContainerResponse might be empty.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13968521#comment-13968521 ] Xuan Gong commented on YARN-1897: - [~mingma] bq. For SignalContainerResponse, what is the semantics of isCMDCompleted? If we want to support synchronous signal container call and this flag indicates whether ContainerExecutor has signaled on the container successfully, that will require RM to wait for the response from NM after NM finishes the work; it implies ApplicationClientProtocol's signalContainer method will hold up a RPC handler for some period of time; we can have some time out or rate limiting on signalContainer call to make sure applications won't be able to consume all RM's RPC handlers. If isCMDCompleted means if the command has been submitted to RM successfully, then it is ok; or we can use exception to indicate failure of the request. OK. We should try the best to do it asynchronously. We will reply on node heartbeat to send the container command to related NM. After NM executes the commands, it can send response(whether the cmd is finished successfully) back to RM with the node heartbeat, too. But this will bring us another questions. Because we can not control how much the NM need to execute the cmds and send back to RM, we can not give a detail time on how long the client should wait for the response. Also, we need to consider the RM Restart, RM Failover, etc. To make progress, i think that right now, check whether command has submitted to RM successfully (check whether container is exist or not, whether the container has already been kill, etc), might be fine for now. So, keep isCMDCompleted in SignalContainerResponse ? What do you think ? Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13968544#comment-13968544 ] Ming Ma commented on YARN-1897: --- Sounds good. How about IsCMDSubmissionCompleted? Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962221#comment-13962221 ] Ming Ma commented on YARN-1897: --- +1 for SignalContainerRequest. For SignalContainerResponse, what is the semantics of isCMDCompleted? If we want to support synchronous signal container call and this flag indicates whether ContainerExecutor has signaled on the container successfully, that will require RM to wait for the response from NM after NM finishes the work; it implies ApplicationClientProtocol's signalContainer method will hold up a RPC handler for some period of time; we can have some time out or rate limiting on signalContainer call to make sure applications won't be able to consume all RM's RPC handlers. If isCMDCompleted means if the command has been submitted to RM successfully, then it is ok; or we can use exception to indicate failure of the request. Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1897) Define SignalContainerRequest and SignalContainerResponse
[ https://issues.apache.org/jira/browse/YARN-1897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13961193#comment-13961193 ] Xuan Gong commented on YARN-1897: - Any other suggestions ? Define SignalContainerRequest and SignalContainerResponse - Key: YARN-1897 URL: https://issues.apache.org/jira/browse/YARN-1897 Project: Hadoop YARN Issue Type: Sub-task Components: api Reporter: Ming Ma We need to define SignalContainerRequest and SignalContainerResponse first as they are needed by other sub tasks. SignalContainerRequest should use OS-independent commands and provide a way to application to specify reason for diagnosis. SignalContainerResponse might be empty. -- This message was sent by Atlassian JIRA (v6.2#6252)
