[jira] [Commented] (YARN-422) Add AM-NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646448#comment-13646448
]
Hadoop QA commented on YARN-422:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581346/YARN-422.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:
org.apache.hadoop.yarn.client.TestNMClientAsync
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/852//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/852//console
This message is automatically generated.
Add AM-NM client library
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch
Create a simple wrapper over the AM-NM container protocol to provide hide the
details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[
https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13646473#comment-13646473
]
Hadoop QA commented on YARN-422:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12581348/YARN-422.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/853//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/853//console
This message is automatically generated.
Add AM-NM client library
Key: YARN-422
URL: https://issues.apache.org/jira/browse/YARN-422
Project: Hadoop YARN
Issue Type: Sub-task
Reporter: Bikas Saha
Assignee: Zhijie Shen
Attachments: AMNMClient_Defination.txt,
AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf,
YARN-422.1.patch, YARN-422.2.patch
Create a simple wrapper over the AM-NM container protocol to provide hide the
details of the protocol implementation.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645307#comment-13645307 ] Vinod Kumar Vavilapalli commented on YARN-422: -- bq. 1. Semantically, it is a bit strange RM use AMNMClient. Agreed. May be we should just call it NMClient? bq. 2. Technically, hadoop-yarn-client has dependency on hadoop-yarn-server-resourcemanager in test scope. If we want to use AMNMClient in AMLauncher, hadoop-yarn-server-resourcemanager needs to add the dependency on hadoop-yarn-client, forming a circular dependency. The dependencies are per scope, so there is not circular dependency either in test scope or non-test scope. Is this patch ready for review? Or just a definition file? Doesn't seem so. In any case, I think we need to have either - separate call-backs for failures on startContainer() and failure on stopContainer() - or may be just one call-back with the original event-type? Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, AMNMClient_Definition_Updated_With_Tests.txt, proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642038#comment-13642038 ] Bikas Saha commented on YARN-422: - Overall the approach looks fine. The patch looks like work in progress and so I am not reviewing it thoroughly. A lot of the complexity and verbosity is coming from the fact that we need to create a new proxy per container rather than per NM. We should file a jira to fix that in the protocol. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642119#comment-13642119 ] Vinod Kumar Vavilapalli commented on YARN-422: -- startContainer/stopContainer can fail, we need to handle the exceptions correctly and report them back to the callers. The CallBackHandler also needs APIs to report back such failures. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: AMNMClient_Defination.txt, proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640660#comment-13640660 ] Zhijie Shen commented on YARN-422: -- Yes, non-blocking API is a must, but shall we define AMNMClientImpl the simple wrapper around the RPC (but to all NMs), such that it contain blocking APIs? After that, we define another class, called AMNMClientAsync, to further wrap AMNMClientImpl and provide non-blocking APIs. Similar to what we did in AMRMClientImpl and AMRMClientAsync. How do you think? Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640717#comment-13640717 ] Vinod Kumar Vavilapalli commented on YARN-422: -- bq. Yes, non-blocking API is a must, but shall we define AMNMClientImpl the simple wrapper around the RPC (but to all NMs), such that it contain blocking APIs? Like I mentioned, I don't see any value in it. Let's skip it for now. I do feel we'll never need one, but we can pursue that later anyways. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639639#comment-13639639 ] Vinod Kumar Vavilapalli commented on YARN-422: -- Comments on your proposal - AMNMClient is good. - We should have only one instance per AM which talks to *all* NMs. Just like ContainerLauncherImpl in MR App. Clearly, the interface you proposed is already accommodating for that. - All the APIs will be blocking? It isn't clear. Today's MR App's ContainerLauncher is non-blocking and it has call-backs/signals as to what happened with a container launch/stop - succeeded/failed. I like making it non-blocking as AMs more commonly should want to kick off the launch and go away. A blocking API could be implemented on top of the non-blocking API if need be. - I think we should change AMLauncher also to use this, but we can scope it into a separate ticket depending on this patch's size What do others think? Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639660#comment-13639660 ] Bikas Saha commented on YARN-422: - I was hoping that the final AMNMClient library would be a one stop library for AM's to communicate with all the NM's in the cluster. The document outlines a wrapper around the NMProtocol for talking to 1 NM. This wrapper is useful by itself because it hides the PBImpl etc implementation details of the protocol itself, similar to how YarnClient started off as. AMRMClient does a little bit more than simply wrapping the Java protocol. So as a first step the above design for wrapping the boiler plate code of the NM Protocol into a utility library wrapper sounds good. I think the method calls are all blocking since the RPC itself is blocking and this is a simple wrapper around the RPC. Maybe we can call be NMClient. Once this is done we can look at creating an AMNMClient library that will provide a single object that an AM can use to manage its containers. This may have its own internal threading etc which will allow the AM to make non-blocking calls. Lets figure that out in a subsequent jira and use this one to just write the wrapper NMCLient that you have proposed. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-422) Add AM-NM client library
[ https://issues.apache.org/jira/browse/YARN-422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639834#comment-13639834 ] Bikas Saha commented on YARN-422: - Sounds good to me to skip the first wrapper step if there are no consumers for it. This would still be an NMClient library since potentially the RM and AM can both use it. Add AM-NM client library Key: YARN-422 URL: https://issues.apache.org/jira/browse/YARN-422 Project: Hadoop YARN Issue Type: Sub-task Reporter: Bikas Saha Assignee: Zhijie Shen Attachments: proposal_v1.pdf Create a simple wrapper over the AM-NM container protocol to provide hide the details of the protocol implementation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
