[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655384#comment-13655384 ] Bikas Saha commented on YARN-660: - bq. Coming back to current issues at hand. Correct me if I am wrong, but it looks like these StoredContainerRequests are only useful if I have a Task in my job which needs a container and the returned container will only be used by that task? I am afraid that is wrong. Every scheduler has a bunch of tasks that it needs to allocate. It makes requests for containers. When containers are returned then it chooses which tasks to assign to those containers. That is the generic use case for schedulers. StoredContaineRequest is unrelated to it. AMRMClientImpl/AMRMClientAsync are already built for the use case specified. A bunch of requests can be made and when containers are allocated then the user can get them and do its second pass of scheduling. bq.The current test-case is clearly not doing enough to explain how it is usable - we can implement a simple job which has tasks and needs containers inside this test-case and then illustrate the library usage. Please read the code in TestAMRMClient.testAMRMClient() and the code in the DistributedShell example app. Both create an RM, register an application, request containers, make allocations and deallocations and then unregister the application. I believe the code clearly shows how all the api's can be used for the full lifecycle of an application. The new tests only test for newly added storage feature and bug fixes. I could add more test code to go through the same lifecycle operations when using StoreContainerRequest. It might be useful if you can go through all the comments in YARN-103, YARN-277 and YARN-417. It may help in understanding why the code/apis are the way they are. 7 contributors were involved in those jiras in writing and reviewing the code and I am confident in the technical and stylistic quality of the code as it stands. That being said, there is always room for improvement if you find issues. Let me try to explain the contribution made by this jira so that we all can be on the same page. When schedulers receive container allocations then they need to figure out which of their requests matched those containers so that they can remove those requests from the pending requests. Containers can match requests at node/rack/* level and hence the library itself cannot determine which pending request to remove. It needs to be told by the scheduler which request to remove. All of this already works if the scheduler can find matching requests for a container on its own or when requests were made at only at * priority (for locality other than * the RM needs client help to do book-keeping so that it does not end up allocating containers at unwanted locality). This jira takes up the problem of helping schedulers find matching requests for allocated containers. Since the library is the entity that translates request to the RM, it would be best to let the library provide matching containers. If the scheduler maintains its own separate mapping information then its logic may diverge from the library's. So the main contribution of the jira is to store requests made via addContainerRequest() and retrieve matching requests via getMatchingRequests(). Stored containers must be removed or else the library will end up holding onto a lot of memory. If ContainerRequest is not constrained to a container count of 1 then doing this add and removal book-keeping becomes difficult to implement in a water-tight manner. That is why StoredContainerRequest was created which only constrains the container count to 1. Now the API becomes elegant and intuitive. addContainerRequest() to add requests. getMatchingRequests() to get all matching requests. Pick a container from matching requests and call removeContainerRequest() with it to remove it. That is all that there is to it. There are some other minor fixes to AMRMClientAsync. We are on the same page wrt this being a needed functionality. The cookie added to StoredContainerRequest is very useful IMO but it strictly does not have to be there. Users can extend StoreContainerRequest to do the same thing and most users will IMO and so I added it there. StoredContainerRequest or any extensions of ContainerRequest can be passed into the library. Doing it without using the help of generics allows for the possibility of writing non-type checked code as well as mixing different types of ContainerRequests. Besides, it will require users to make ugly unsafe casts in their code for which the compiler will generate warnings. While it may be fine for internal YARN code to have 100's of suppressed unchecked cast warnings, we should not force external user code to have to live with that. They may have their own coding guidelines. This makes the use of gener
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655134#comment-13655134 ] Vinod Kumar Vavilapalli commented on YARN-660: -- Okay, I took this as an opportunity to learn more about AMRMClient. Have seen other API issues in AMRMCient, will file tickets. Clearly we need a validation of the APIs like you are doing in TEZ. We should definitely try to change MR AppMaster to optionally use AMRMClient. Coming back to current issues at hand. Correct me if I am wrong, but it looks like these StoredContainerRequests are only useful if I have a Task in my job which needs a container and the returned container will only be used by that task? If yes - That is a very narrow use-cases. May be the simplest use-case which we can address by building on top of AMRMClient, like for e.g., a SimpleAMRMClient which only deals with a container at a time. - AMRMClient should be built for the more common use-cases - I give you these requests, you tell me when the containers are allocated, I'll do the second schedule pass according to my own logic. Again, I think I should see a more concrete usage. The current test-case is clearly not doing enough to explain how it is usable - we can implement a simple job which has tasks and needs containers inside this test-case and then illustrate the library usage. > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655049#comment-13655049 ] Bikas Saha commented on YARN-660: - Vinod, I have iterated through different combinations of generics including the one you suggest. I found that the current version is the least ugly wrt user code. Both the test code for the AMRMClient as well as my use of it in TEZ gave me that impression. If you write user code that has to always cast/convert back from ContainerRequest to user-defined types you will see that it looks ugly and I dont want users to keep having to cast these value when they use specific types. The only user overhead when they use ContainerRequest is to add a when declaring the member variable and during new. Thereafter everything works. And ContainerRequest is something that they anyways have to include in the code. Look at the diff of TestAMRMClient to see the minimal changes needed to make the existing code compile. Ideally if there was a typedef in Java then even that could be hidden but there isnt. IMO, ContainerRequest is useful mostly in cases when the user wants a bunch of containers at *. For most of the other cases where requests are made with more specificity StoredContainerRequest is more useful. Its provided by the library because it will be commonly needed and also to support returning matching requests from within the AMRMClient for reasons outlined earlier. Storing and retrieving matching requests in a meaningful manner cannot be done until we limit the number of containers in an individual request to 1. StoredContainerRequest provides a clear type inside the AMRMClient to say what to store and also to limit the container count per request to 1. e.g. if AMRMClient will save simple ContainerRequest then lets say it saves the container request in addContainerRequest(ContainerRequest(P1,R1,Count=4)) and then if user calls removeContainerRequest(ContainerRequest(P1,R1,Count=3)) its hard for AMRMClient to tell if the stored container request should be removed or not. Cookies are not required but very useful in reducing user burden. If we think about using AMRMClient inside the MR app master (or see the impl of TEZ) then we will find that for every request we need to save a cookie somewhere (eg. Scheduler*LaunchEvent) that will be used when the request is matched to a container. Either the client can write a map to maintain and store this relationship or the library provides a helper cookie to keep the info in one place. Let me know if you have any other comments. > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch, > YARN-660.4.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655043#comment-13655043
]
Hadoop QA commented on YARN-660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582739/YARN-660.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/914//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/914//console
This message is automatically generated.
> Improve AMRMClient with cookies and matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.0.5-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.0.5-beta
>
> Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch,
> YARN-660.4.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13655012#comment-13655012 ] Vinod Kumar Vavilapalli commented on YARN-660: -- Quick comment: h4.Cookies: Seems like the cookies aren't used by the library itself. Is that true? If so, why does the library need to provide StoredContainerRequest, can't the users implement one and pass it on? h4. Generics/typing: Adding in this type information for every AMRMClient is an unnecessary burden for most cases IMO. Users can skip explicit typing, but the compiler will warn unnecessarily. Can't this be done without the explicit typing? I can see that the new API getMatchingRequests is useful, +1 for that. What if we just return a Collection all the time, and get clients to do explicit conversion if they care about adding in more information. Similarly, add and remove APIs already take in a ContainerRequest, you can just continue to pass sub-types of ContainerRequest when you want to. I quickly did this on your latest patch, removing all typing in the library, it works. Have to dig deeper into the client libraries, haven't looked at them at all. > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654973#comment-13654973
]
Alejandro Abdelnur commented on YARN-660:
-
On the {{ResourceRequestInfo}}'s {{HashSet containerRequests}}, wouldn't be
more intuitive for AMRMClient users if they get the matching requests in the
other they were submitted? This could easily be done using a {{LinkedHashSet}}.
> Improve AMRMClient with cookies and matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.0.5-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.0.5-beta
>
> Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654952#comment-13654952 ] Vinod Kumar Vavilapalli commented on YARN-660: -- I'd like to have a quick look before commit. Looking right now. > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654944#comment-13654944 ] Hitesh Shah commented on YARN-660: -- +1. Looks good to me. > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654923#comment-13654923
]
Hadoop QA commented on YARN-660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582705/YARN-660.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/912//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/912//console
This message is automatically generated.
> Improve AMRMClient with cookies and matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.0.5-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.0.5-beta
>
> Attachments: YARN-660.1.patch, YARN-660.2.patch, YARN-660.3.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654713#comment-13654713
]
Hadoop QA commented on YARN-660:
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582656/YARN-660.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 2 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/910//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/910//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/910//console
This message is automatically generated.
> Improve AMRMClient with cookies and matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.0.5-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.0.5-beta
>
> Attachments: YARN-660.1.patch, YARN-660.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654683#comment-13654683 ] Bikas Saha commented on YARN-660: - As of now there has been no need to override the default hashcode and equals impls. For stored containers I think I prefer an object comparison because I would like the same objects to be passed around for efficiency. Making the other changes. > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13654677#comment-13654677 ] Hitesh Shah commented on YARN-660: -- Comments: - DistributedShell#ApplicationMaster#onError should be handled. - Cookie should be a final var and part of StoredContainerRequest's constructor - Don't ContainerRequest and StoredContainerReq require hashCode() and equals() implementations? > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.0.5-beta >Reporter: Bikas Saha >Assignee: Bikas Saha > Fix For: 2.0.5-beta > > Attachments: YARN-660.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[
https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653217#comment-13653217
]
Hadoop QA commented on YARN-660:
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12582512/YARN-660.1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/903//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/903//console
This message is automatically generated.
> Improve AMRMClient with cookies and matching requests
> -
>
> Key: YARN-660
> URL: https://issues.apache.org/jira/browse/YARN-660
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.0.5-beta
>Reporter: Bikas Saha
>Assignee: Bikas Saha
> Fix For: 2.0.5-beta
>
> Attachments: YARN-660.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-660) Improve AMRMClient with cookies and matching requests
[ https://issues.apache.org/jira/browse/YARN-660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13653177#comment-13653177 ] Bikas Saha commented on YARN-660: - As outlined in the comments on YARN-103, there were plans to add cookies in ContainerRequest to enable schedulers to save information about the request when they make a container request. This reduces developer burden. Common example is to save information about which task this request was made for so that when a received container matches the request then the task info can be readily obtained without having to maintain duplicate maps at the user level. Also, it would be helpful if the AMRMClient allowed users to look up its own requests that match a set of criteria. This can be used to find out which request to assign a new container to. The user could again maintain their own maps but they may diverge in logic from what the AMRMClient actually translates them to. So if the AMRMClient had a getMatchingRequests method then it could be used as a single source of truth. The patch provides these changes as well as fixes some bugs/minor improvements in error handling to AMRMClientAsync. The changes are backed by real use case scenarios used in the development of the scheduler for the [TEZ|https://issues.apache.org/jira/browse/TEZ] application master. > Improve AMRMClient with cookies and matching requests > - > > Key: YARN-660 > URL: https://issues.apache.org/jira/browse/YARN-660 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > Attachments: YARN-660.1.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
