[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365070#comment-15365070 ]
Jason Lowe commented on YARN-5215: ---------------------------------- Maybe I'm missing something, but any of the proposed approaches has YARN assuming it can leverage the unused resources on the node. That's sort of the whole point, we want YARN to use those unused resources rather than just hard-partitioning the node between YARN and the other system. Some of the approaches start with the assumption that the whole node belongs to YARN and YARN will scale back usage of the node based on utilization feedback, while other approaches start with YARN assuming it has a smaller portion of the node and can reach beyond it when utilization is low. It's the same scenario from two perspectives. IIUC any of these approaches can react relatively quickly to the other workload's demands by having the nodemanager take action directly (by preempting containers) when the periodically monitored node utilization goes above some configured limit. The original proposal in this JIRA doesn't do that, which means it won't be super-responsive to the other subsystem. The RM won't allocate any additional containers when the utilization gets high, but some of the containers would have to exit on their own before YARN's existing utilization would decrease. It sounds like the version Inigo has deployed in production does do some sort of preemption, but it sounded like it was coming from the RM rather than the NM which would be slightly slower response time than if the NM did it directly. If the latency demands of the other workload are so severe that it's impossible for YARN to react quickly enough then I don't see how YARN can leverage those resources when they are unused. We'd have to resort to some kind of hard-partitioning (either giving the nodemanager less resources than the node actually has or using proxy containers in YARN on behalf of the other workload to reserve the resources) and live with the underutilization of those resources when the other workload is idle. > Scheduling containers based on external load in the servers > ----------------------------------------------------------- > > Key: YARN-5215 > URL: https://issues.apache.org/jira/browse/YARN-5215 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Inigo Goiri > Attachments: YARN-5215.000.patch, YARN-5215.001.patch > > > Currently YARN runs containers in the servers assuming that they own all the > resources. The proposal is to use the utilization information in the node and > the containers to estimate how much is consumed by external processes and > schedule based on this estimation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org