[jira] [Commented] (YARN-226) Log aggregation should not assume an AppMaster will have containerId 1
[ https://issues.apache.org/jira/browse/YARN-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13603527#comment-13603527 ] Eli Reisman commented on YARN-226: -- Hmm. It would not be impossible to ask for one container, launch it, then the rest of the graph tasks so that first one can be set up to know its first. I think the whole "task 0" thing is old in Giraph and could be handled another way by now. I will look into it. It didn't make a lot of sense when that approach worked to start with, but it worked right away so I went with it ;) > Log aggregation should not assume an AppMaster will have containerId 1 > -- > > Key: YARN-226 > URL: https://issues.apache.org/jira/browse/YARN-226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth > > In case of reservcations, etc - AppMasters may not get container id 1. We > likely need additional info in the CLC / tokens indicating whether a > container is an AM or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-226) Log aggregation should not assume an AppMaster will have containerId 1
[ https://issues.apache.org/jira/browse/YARN-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602667#comment-13602667 ] Hitesh Shah commented on YARN-226: -- @Sid, nevermind - reservations could effectively increment the id and never assign 1 to anything. @Eli, this will occur on clusters when the AM resource ask is greater than a single slot and it requires multiple scheduling cycles before a large free slot is available to launch the AM. > Log aggregation should not assume an AppMaster will have containerId 1 > -- > > Key: YARN-226 > URL: https://issues.apache.org/jira/browse/YARN-226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth > > In case of reservcations, etc - AppMasters may not get container id 1. We > likely need additional info in the CLC / tokens indicating whether a > container is an AM or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-226) Log aggregation should not assume an AppMaster will have containerId 1
[ https://issues.apache.org/jira/browse/YARN-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602663#comment-13602663 ] Hitesh Shah commented on YARN-226: -- @Sid, unless the AM is launched, there would be no asks for any container from that application attempt. Are we talking about a case where there is a possibility of transferring a container from a previous attempt to the current attempt? Could you clarify a bit more on how container id 2 could get assigned to an AM. @Eli, relying on container ids is not a good idea in general. The AM should be acting as a controller so it may be good to pull the bootstrap into it. We should discuss offline as to what you are trying to do with giraph task ids and why they need translating as compared to just passing them in via the command line layer of the ContainerLaunchContext when the giraph task is launched. > Log aggregation should not assume an AppMaster will have containerId 1 > -- > > Key: YARN-226 > URL: https://issues.apache.org/jira/browse/YARN-226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth > > In case of reservcations, etc - AppMasters may not get container id 1. We > likely need additional info in the CLC / tokens indicating whether a > container is an AM or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-226) Log aggregation should not assume an AppMaster will have containerId 1
[ https://issues.apache.org/jira/browse/YARN-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602661#comment-13602661 ] Robert Joseph Evans commented on YARN-226: -- Big means amount of memory/CPU relative to the minimum allocation size. For example you ask for a 4 GB container with a min allocation size of 500MB. > Log aggregation should not assume an AppMaster will have containerId 1 > -- > > Key: YARN-226 > URL: https://issues.apache.org/jira/browse/YARN-226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth > > In case of reservcations, etc - AppMasters may not get container id 1. We > likely need additional info in the CLC / tokens indicating whether a > container is an AM or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-226) Log aggregation should not assume an AppMaster will have containerId 1
[ https://issues.apache.org/jira/browse/YARN-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13602614#comment-13602614 ] Eli Reisman commented on YARN-226: -- Hmm, that stirs up some trouble. Giraph tasks may or may not need to be contiguous ID's but will need a "task 0" for at least one of the reserved containers (so container 2 right now is the one) in order to bootstrap our master election process. I am using container Id's to translate into giraph task Id's rigth now by just subtracting two from container Id! It works in all my tests, but the reservation thing could kick in on big asks (1000 container ask etc.) is that what you're saying? How big can the ask be? Perhaps I can move this bootstrap stuff from Giraph into my app master if this is a big problem. Good to know, thanks! > Log aggregation should not assume an AppMaster will have containerId 1 > -- > > Key: YARN-226 > URL: https://issues.apache.org/jira/browse/YARN-226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth > > In case of reservcations, etc - AppMasters may not get container id 1. We > likely need additional info in the CLC / tokens indicating whether a > container is an AM or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-226) Log aggregation should not assume an AppMaster will have containerId 1
[ https://issues.apache.org/jira/browse/YARN-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13601952#comment-13601952 ] Siddharth Seth commented on YARN-226: - bq. The Giraph to YARN port assumes giraph tasks start at 2 and up as far as container #'s go. Is this unsafe for the future? One scenario in which the AM does not get container id 1 is when it requires more resources than the minimum allocation - in which case reservations come in to play. Depending on whether the reservation is the final allocation or whether it happens elsewhere - the container id may not be one. Similarly, assuming the container IDs are contiguous is not valid. IDs can be skipped. > Log aggregation should not assume an AppMaster will have containerId 1 > -- > > Key: YARN-226 > URL: https://issues.apache.org/jira/browse/YARN-226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth > > In case of reservcations, etc - AppMasters may not get container id 1. We > likely need additional info in the CLC / tokens indicating whether a > container is an AM or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-226) Log aggregation should not assume an AppMaster will have containerId 1
[ https://issues.apache.org/jira/browse/YARN-226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13601921#comment-13601921 ] Eli Reisman commented on YARN-226: -- Under which conditions will AM not be task 1? Does a client ask have to fail and be re-requested for this to occur? Can this ever happen when the first container asked for in client is successful? Or, right to the point: If your application is set to fail if that first container isn't available from the first client ask, will the AM ever get anything (on successful run) except task 1? The Giraph to YARN port assumes giraph tasks start at 2 and up as far as container #'s go. Is this unsafe for the future? > Log aggregation should not assume an AppMaster will have containerId 1 > -- > > Key: YARN-226 > URL: https://issues.apache.org/jira/browse/YARN-226 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Siddharth Seth > > In case of reservcations, etc - AppMasters may not get container id 1. We > likely need additional info in the CLC / tokens indicating whether a > container is an AM or not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
