[jira] [Commented] (YARN-513) Verify all clients will wait for RM to restart
[ https://issues.apache.org/jira/browse/YARN-513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13630971#comment-13630971 ] Xuan Gong commented on YARN-513: Oh, yes, you are right. We need to handle more cases. For Client: getNewApplication(),submitApplication(),killApplication(),getApplicationReport(),getApplicationList(),getYarnClusterMetrics(),getNodeReports(),getRMDelegationToken(),getQueueInfo(),getQueueAclsInfo(),getAllQueues(),getRootQueueInfos() and getChildQueueInfos() For ApplicationMaster: registerApplicationMaster(),allocate() and unregisterApplicationMaster() > Verify all clients will wait for RM to restart > -- > > Key: YARN-513 > URL: https://issues.apache.org/jira/browse/YARN-513 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Bikas Saha >Assignee: Xuan Gong > > When the RM is restarting, the NM, AM and Clients should wait for some time > for the RM to come back up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631006#comment-13631006 ] Hudson commented on YARN-412: - Integrated in Hadoop-Yarn-trunk #182 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/182/]) YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470) YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than its host:port during scheduling which caused incorrect locality for containers. Contributed by Roger Hoover. (Revision 1467244) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.5-beta > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631027#comment-13631027 ] Hudson commented on YARN-412: - Integrated in Hadoop-Hdfs-trunk #1371 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1371/]) YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470) YARN-412. Fixed FifoScheduler to check hostname of a NodeManager rather than its host:port during scheduling which caused incorrect locality for containers. Contributed by Roger Hoover. (Revision 1467244) Result = FAILURE acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467244 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.5-beta > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-412) FifoScheduler incorrectly checking for node locality
[ https://issues.apache.org/jira/browse/YARN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631053#comment-13631053 ] Hudson commented on YARN-412: - Integrated in Hadoop-Mapreduce-trunk #1398 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1398/]) YARN-412. Pushing to 2.0.5-beta only. (Revision 1467470) Result = SUCCESS acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1467470 Files : * /hadoop/common/trunk/hadoop-yarn-project/CHANGES.txt > FifoScheduler incorrectly checking for node locality > > > Key: YARN-412 > URL: https://issues.apache.org/jira/browse/YARN-412 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Reporter: Roger Hoover >Assignee: Roger Hoover >Priority: Minor > Labels: patch > Fix For: 2.0.5-beta > > Attachments: YARN-412.patch > > > In the FifoScheduler, the assignNodeLocalContainers method is checking if the > data is local to a node by searching for the nodeAddress of the node in the > set of outstanding requests for the app. This seems to be incorrect as it > should be checking hostname instead. The offending line of code is 455: > application.getResourceRequest(priority, node.getRMNode().getNodeAddress()); > Requests are formated by hostname (e.g. host1.foo.com) whereas node addresses > are a concatenation of hostname and command port (e.g. host1.foo.com:1234) > In the CapacityScheduler, it's done using hostname. See > LeafQueue.assignNodeLocalContainers, line 1129 > application.getResourceRequest(priority, node.getHostName()); > Note that this bug does not affect the actual scheduling decisions made by > the FifoScheduler because even though it incorrect determines that a request > is not local to the node, it will still schedule the request immediately > because it's rack-local. However, this bug may be adversely affecting the > reporting of job status by underreporting the number of tasks that were node > local. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-573) Shared data structures in Public Localizer and Private Localizer are not Thread safe.
[ https://issues.apache.org/jira/browse/YARN-573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631268#comment-13631268 ] Chris Douglas commented on YARN-573: Pardon? > Shared data structures in Public Localizer and Private Localizer are not > Thread safe. > - > > Key: YARN-573 > URL: https://issues.apache.org/jira/browse/YARN-573 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Omkar Vinit Joshi >Assignee: Omkar Vinit Joshi > > PublicLocalizer > 1) pending accessed by addResource (part of event handling) and run method > (as a part of PublicLocalizer.run() ). > PrivateLocalizer > 1) pending accessed by addResource (part of event handling) and > findNextResource (i.remove()). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
