[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935274#comment-13935274 ] Zhijie Shen commented on YARN-1521: --- Just a reminder, ApplicationClientProtocol has four more methods, whose idempotency needs to be verified as well: 1. getApplicationAttemptReport 2. getApplicationAttempts 3. getContainerReport 4. getContainers > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935336#comment-13935336 ] Tsuyoshi OZAWA commented on YARN-1521: -- We should introduce RetryCache to avoid returning incorrect errors to client if non-idempotent RPC(ie. submitApplication) are executed. Opened YARN-1836 for this. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935821#comment-13935821 ] Xuan Gong commented on YARN-1521: - But first of all, we need to find which apis can be marked as Idempotent Here is the list of APIs that I think we can mark as Idempotent: * ResourceTracker ** registerNodeManager ** nodeHeartbeat * ResourceManagerAdministrationProtocol ** refreshQueues ** refreshNodes ** refreshSuperUserGroupsConfiguration ** refreshUserToGroupsMappings ** refreshAdminAcls ** refreshServiceAcls * ApplicationClientProtocol ** forceKillApplication ** getApplicationReport (already marked) ** getClusterMetrics ** getApplications ** getClusterNodes ** getQueueInfo ** getQueueUserAcls ** getApplicationAttemptReport ** getApplicationAttempts ** getContainerReport ** getContainers > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13935873#comment-13935873 ] Xuan Gong commented on YARN-1521: - APIs are not in previous list: * ApplicationMasterProtocol ** registerApplicationMaster ** finishApplicationMaster ** allocate * ResourceManagerAdministrationProtocol ** updateNodeResource * ApplicationClientProtocol ** getNewApplication ** submitApplication ** getDelegationToken ** renewDelegationToken ** cancelDelegationToken ** moveApplicationAcrossQueues > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942409#comment-13942409 ] Xuan Gong commented on YARN-1521: - We have three options for all the protocol APIs: 1. marked as Idempotent 2. marked as AtMostOnce. 3. do nothing The apis can be marked idempotent: * ResourceManagerAdministrationProtocol: 1. refreshQueues 2. refreshNodes 3. refreshSuperUserGroupsConfiguration 4. refreshUserToGroupsMappings 5. refreshAdminAcls 6. refreshServiceAcls * ResourceTracker: 1. registerNodeManager 2. nodeHeartbeat * ApplicationClientProtocol 1. getNewApplication 2. forceKillApplication 3. getApplicationReport 4. getClusterMetrics 5. getApplications 6. getClusterNodes 7. getQueueInfo 8. getQueueUserAcls 9. getApplicationAttemptReport 10. getApplicationAttempts 11. getContainerReport 12. getContainers 13. getDelegationToken These APIs can be marked as AtMostOnce: * ApplicationClientProtocol: 1. submitApplication These apis do not need to add any annotations * ResourceManagerAdministrationProtocol 1. updateNodeResource: Do not find any places using it. Maybe some other applications will use this API. Possibly can be marked idempotent. * ApplicationMasterProtocol 1. registerApplicationMaster 2. finishApplicationMaster 3. allocate * ApplicationClientProtocol: 1. moveApplicationAcrossQueues: Do not support yet There are two APIs which are not in this list: 1. renewDelegationToken 2. cancelDelegationToken We can add the proper annotations for them in https://issues.apache.org/jira/browse/YARN-1776 > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13942555#comment-13942555 ] Zhijie Shen commented on YARN-1521: --- bq. 2. cancelDelegationToken The situation of cancelDelegationToken is still different from that of renewDelegationToken. It is summarized in YARN-1860. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13943793#comment-13943793 ] Jian He commented on YARN-1521: --- - getNewApplicationId , getDelegationToken. since each call returns a new ID/Token, not sure this matches with idempotency. - For the registers protocols. For example, registerNodeManager : if previous call succeeds, RM didn't crash, registerNodeManager retry because of some network problem, the next call comes in, this node is deemed as a reconnected node instead of a new node. Probably AtMostOnce? > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944632#comment-13944632 ] Xuan Gong commented on YARN-1521: - updated: * mark renewDelegationToken, cancelDelegationToken, updateNodeResource and moveApplicationAcrossQueues as Idempotent * change submitApplication from AtMostOnce to Idempotetn * change nodeHeartbeat from Idempotent to AtMostOnce > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944633#comment-13944633 ] Xuan Gong commented on YARN-1521: - The patch includes: * Mark appropriate protocol methods with idempotent and atmostonce annotation based on the proposal. * Create testcases to test the annotation marked ** Limited scope: For all the testcases, we only test whether the method will be re-entry when failover happens. Does not cover the entire logic test. ** Test strategy: create a separate failover thread with a trigger flag, override all APIs that added trigger flag. When the apis are called, we will set trigger flag as true to kick off the failover. So We can make sure the failover happens during process of the method. If this API is marked as idempotent or atmostonce, the testcases will pass; otherwise, they will throw the exception. ** Did not add testcases for ResourceManagerAdministrationProtocol. All refresh* will be called during the process of transitionToActive that will break the test strategy I used here. But I did the manually testing. Simply add sleep thread into the refresh*, and verified that all refresh* apes can be re-entry when failover happens after we marked all refresh* as idempotent. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944675#comment-13944675
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636277/YARN-1521.0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3439//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3439//console
This message is automatically generated.
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13944723#comment-13944723 ] Karthik Kambatla commented on YARN-1521: I would like to take a closer look at the annotations before this gets committed. If not urgent, please wait for me until Tuesday. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945736#comment-13945736
]
Jian He commented on YARN-1521:
---
post initial review：
- this doesn’t seem correct, user is possible to get multiple delegation
tokens. Given this change, user can only get one token
{code}
// check whether the token exists or not.
// If this token existed, recover the token with
// DelegationTokenInformation
{code}
- Explicitly assert the App does not exist in RMStateStore or not.
{code}
// After submission, the applicationState will
// not be saved in RMStateStore
{code}
- Explicitly assert the app exists in RMContext after the 2nd RM comes Active.
{code}
// Submit Application
// After submission, the applicationState will be saved in RMStateStore.
{code}
- The bulk of test-specific hack in MiniYarnCluster can be moved to
TestHAProtocol, as MiniYarnCluster is commonly used by others.
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945905#comment-13945905 ] Xuan Gong commented on YARN-1521: - Thanks for the review. bq. The bulk of test-specific hack in MiniYarnCluster can be moved to TestHAProtocol, as MiniYarnCluster is commonly used by others. Yes, you are right. Remove those changes from MiniYarnCluster to TestHAProtocol bq. Explicitly assert the app exists in RMContext after the 2nd RM comes Active./ Explicitly assert the App does not exist in RMStateStore or not. DONE bq. this doesn’t seem correct, user is possible to get multiple delegation tokens. Given this change, user can only get one token Remove all changes from ClientRMService#getDelegationToken(). Just let the method re-entry if failover happens. [~jianhe] Please correct me if I am wrong. The getDelegationToken() is used to get a new Token, but is not saved in zookeeper yet. So, even if failover happens during getDelegationToken call, we can not get previous generated Token back, so just let the method re-generate a new token, and it should be fine. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch, YARN-1521.1.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945964#comment-13945964
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636467/YARN-1521.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3448//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3448//console
This message is automatically generated.
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13946246#comment-13946246
]
Zhijie Shen commented on YARN-1521:
---
Browsed through the newest, and have the following comments:
1. If the application is already in the RM cache, we shouldn't log success.
Otherwise, there may be multiple logs for one submission.
{code}
+ RMAuditLogger.logSuccess(user, AuditConstants.SUBMIT_APP_REQUEST,
+ "ClientRMService", applicationId);
{code}
2. Should it sleep a while before retry?
{code}
+ try {
+client.getApplications();
+return;
+ } catch (Exception e) {
+LOG.error(e.getMessage());
+ } finally {
+client.stop();
+ }
{code}
3. Again, sleep 1ms before next try, yielding to the thread of API methods'
invoking. And, have max retry when exception?
{code}
while (keepRunning) {
+ if (cluster.getStartFailoverFlag()) {
+try {
+ explicitFailover();
+ keepRunning = false;
+} catch (Exception e) {
+ // Do Nothing
+} finally {
+ keepRunning = false;
+}
+ }
+}
{code}
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947059#comment-13947059 ] Xuan Gong commented on YARN-1521: - bq. 1. If the application is already in the RM cache, we shouldn't log success. Otherwise, there may be multiple logs for one submission. Removed bq. Should it sleep a while before retry? DONE bq. Again, sleep 1ms before next try, yielding to the thread of API methods' invoking. And, have max retry when exception? DONE. Did not add max retry. The failover thread will be killed after every testcases > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947125#comment-13947125
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636766/YARN-1521.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3453//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3453//console
This message is automatically generated.
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947139#comment-13947139 ] Xuan Gong commented on YARN-1521: - [~kkambatl] Could you take a look ? > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947295#comment-13947295
]
Jian He commented on YARN-1521:
---
- we should at least log this event saying this is an earlier submitted
application, which is good for debugging.
- TestProtocolHA -> TestProtocolHABase.java
- I ran TestApplicationClientProtocolOnHA locally without core code changes,
the whole test eventually takes 15 mins to crash.
I observed that the test keeps doing failover even if the test is done. can you
investigate ?
{code}
14/03/25 15:02:08 WARN resourcemanager.RMAuditLogger: USER=jhe
OPERATION=transitionToStandby TARGET=RMHAProtocolService RESULT=FAILURE
DESCRIPTION=Exception transitioning to standby PERMISSIONS=All users are
allowed
14/03/25 15:02:09 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
14/03/25 15:02:11 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
14/03/25 15:02:14 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
14/03/25 15:02:23 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
14/03/25 15:02:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
14/03/25 15:02:57 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm2
14/03/25 15:03:30 INFO client.ConfiguredRMFailoverProxyProvider: Failing over
to rm1
{code}
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947302#comment-13947302 ] Vinod Kumar Vavilapalli commented on YARN-1521: --- bq. - TestProtocolHA -> TestProtocolHABase.java Or ProtocolHATestBase. BTW, why is it ProtocolHA? > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947311#comment-13947311 ] Jian He commented on YARN-1521: --- In TestApplicationClientProtocolOnHA , each client API call involves a new miniYarnCluster start and shutdown, how long does it take for the whole test to finish ? If that's too long, we can just reuse one miniYarnCluster. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947324#comment-13947324 ] Zhijie Shen commented on YARN-1521: --- bq. If that's too long, we can just reuse one miniYarnCluster. Some input: It may or may not run into the case I saw in YARN-1873. Before reusing a single yarn cluster, it's good to make sure the change on miniYarnCluster in the current test case will not disturb the following test cases. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947392#comment-13947392
]
Karthik Kambatla commented on YARN-1521:
Sorry for the delay in getting to this. Haven't look at the patch itself yet.
{quote}
ApplicationMasterProtocol
1. registerApplicationMaster
2. finishApplicationMaster
3. allocate
{quote}
Don't remember the source corresponding to these methods off the top of my
head, but would think we *should* mark all of them as well. I am okay with
doing this in a separate JIRA to unblock this.
# Register resets the sequence number - Idempotent. YARN-556 allows a running
AM to re-register, albeit on resync.
# Finish should be similar to kill - Idempotent. Should be similar to
killApplication?
# Allocate - Idempotent or AtMostOnce - as the AM heartbeats periodically.
Preferably Idempotent.
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947399#comment-13947399
]
Karthik Kambatla commented on YARN-1521:
Looked at the non-test source only. Looks good. One nit - it should be okay to
fix at commit time.
# Reword to "it checks whether the application already exists"?
{code}
* During the submission process, it checks whether the application
* has already exist. If the application exists, it will simply return
* SubmitApplicationResponse
{code}
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947414#comment-13947414 ] Xuan Gong commented on YARN-1521: - bq. In TestApplicationClientProtocolOnHA , each client API call involves a new miniYarnCluster start and shutdown, how long does it take for the whole test to finish ? If that's too long, we can just reuse one miniYarnCluster. It takes about 120s to finish all the test cases. bq. we should at least log this event saying this is an earlier submitted application, which is good for debugging. ADDED bq. TestProtocolHA -> TestProtocolHABase.java changed > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947466#comment-13947466
]
Hadoop QA commented on YARN-1521:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12636836/YARN-1521.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3459//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3459//console
This message is automatically generated.
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch,
> YARN-1521.3.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947511#comment-13947511 ] Jian He commented on YARN-1521: --- LGTM + 1. Regarding ApplicationMasterProtocol, since applications today are anyways killed after RM restarts, no point adding it now. We can add those in YARN-556. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947512#comment-13947512 ] Jian He commented on YARN-1521: --- checking this in. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947525#comment-13947525 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5404 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5404/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947813#comment-13947813 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #521 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/521/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947927#comment-13947927 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1713 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1713/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13947972#comment-13947972 ] Hudson commented on YARN-1521: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #1738 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1738/]) YARN-1521. Mark Idempotent/AtMostOnce annotations to the APIs in ApplicationClientProtcol, ResourceManagerAdministrationProtocol and ResourceTrackerProtocol so that they work in HA scenario. Contributed by Xuan Gong (jianhe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1581678) * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/ApplicationClientProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/server/api/ResourceManagerAdministrationProtocol.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/ProtocolHATestBase.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestApplicationClientProtocolOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/TestResourceTrackerOnHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/ResourceTracker.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestClientRMService.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestSubmitApplicationWithRMHA.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/MiniYARNCluster.java > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[
https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948184#comment-13948184
]
Bikas Saha commented on YARN-1521:
--
How are the following idempotent? Looks like submitApplication was made
idempotent in this patch itself? What about moveApplication? If the app has
been moved from A->B then asking another move from A would be an error. So a
retry after the operation should also return an error unless there is specific
handling for the retry.
{code}
+ @Idempotent
public SubmitApplicationResponse submitApplication(
SubmitApplicationRequest request)
+ @Idempotent
public MoveApplicationAcrossQueuesResponse moveApplicationAcrossQueues(
{code}
IMO, how can we simply create an empty response and return it? Currently this
is safe because that response object has no fields. But if a field is added
tomorrow then this code would be wrong since users would get a response object
from which they cannot get response information. We should be using RetryCache
to populate the response from the actual response that was returned for the
original request that got retried.
{code}- throw RPCUtil.getRemoteException(message);
+ LOG.info("This is an earlier submitted application: " + applicationId);
+ return SubmitApplicationResponse.newInstance();{code}
> Mark appropriate protocol methods with the idempotent annotation or
> AtMostOnce annotation
> -
>
> Key: YARN-1521
> URL: https://issues.apache.org/jira/browse/YARN-1521
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Xuan Gong
>Assignee: Xuan Gong
>Priority: Blocker
> Fix For: 2.4.0
>
> Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch,
> YARN-1521.3.patch
>
>
> After YARN-1028, we add the automatically failover into RMProxy. This JIRA is
> to identify whether we need to add idempotent annotation and which methods
> can be marked as idempotent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948263#comment-13948263 ] Xuan Gong commented on YARN-1521: - Right now, moveApplication is only implemented for FairScheduler. We should make sure it will be implemented as Idempotent for CapacityScheduler and verify it works idempotent for FairScheduler when failover happens. Created https://issues.apache.org/jira/browse/YARN-1882 to track this > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948457#comment-13948457 ] Bikas Saha commented on YARN-1521: -- OK. So we are requiring the moveAppAcrossQueues API to be idempotent? How can that be done without ignoring an illegal move of an app from a queue in which the app does not exist? Is that correct behavior? Any response to the second comment above about returning a new SubmitApplicationResponse instead of the original response for the original request. What if there are member values in SubmitApplicationResponse (there aren't any right now). > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948498#comment-13948498 ] Xuan Gong commented on YARN-1521: - bq. OK. So we are requiring the moveAppAcrossQueues API to be idempotent? How can that be done without ignoring an illegal move of an app from a queue in which the app does not exist? Is that correct behavior? I think we can do: * Check if the app exists in target queue. If yes, return * Check if the app exists in original queue. If yes, start to move. Otherwise, throw Exception. So, even the failover happens during the process of moving application, as long as we can recover the application as well as the queue information(we may need to find out which information we should save), I think we are fine. bq. Any response to the second comment above about returning a new SubmitApplicationResponse instead of the original response for the original request. What if there are member values in SubmitApplicationResponse (there aren't any right now). I think that it will based on the information we want. If they are related to the application, we should saved them in the applicationState. In that case, as long as we will recover the applicationState, we should be easier to get them. If they are dynamic information, such as something related to current RM, NM, resource, etc, even we are using RetryCache to return the previous response. The information we get is still not correct because we submit the application to different RM, we need to get the updated values. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-1521) Mark appropriate protocol methods with the idempotent annotation or AtMostOnce annotation
[ https://issues.apache.org/jira/browse/YARN-1521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13948528#comment-13948528 ] Xuan Gong commented on YARN-1521: - bq. We should be using RetryCache to populate the response from the actual response that was returned for the original request that got retried. Also, about the RetryCache. In HDFS, it saves each operation in the editLog, and then recovers the RetryCache based on the editLog. For Yarn, I think that we already have our own RetryCache. The RMStateStore can do the same thing. We save the applicationState, etc. to RMStateStore just like HDFS saves all operations into editLog. We read all states back to RM just like we recover the RetryCache from the editLog. How HDFS decides whether gives the previous responses is based on the operation status in the RetryCache. For Yarn, how we decide whether submitApplication is successful or not when failover happens, we check whether the app exist in RMContext (which is recovered from RMStateStore). We have similar behaviors. The only difference, I think, is that RetryCache has status for all the operations (delete, create, etc). For us, we save them into different places, such as RMContext, RMDelegationTokenSecretManager. > Mark appropriate protocol methods with the idempotent annotation or > AtMostOnce annotation > - > > Key: YARN-1521 > URL: https://issues.apache.org/jira/browse/YARN-1521 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Xuan Gong >Assignee: Xuan Gong >Priority: Blocker > Fix For: 2.4.0 > > Attachments: YARN-1521.0.patch, YARN-1521.1.patch, YARN-1521.2.patch, > YARN-1521.3.patch > > > After YARN-1028, we add the automatically failover into RMProxy. This JIRA is > to identify whether we need to add idempotent annotation and which methods > can be marked as idempotent. -- This message was sent by Atlassian JIRA (v6.2#6252)
