[ https://issues.apache.org/jira/browse/YARN-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16128826#comment-16128826 ]
Jason Lowe commented on YARN-7019: ---------------------------------- I'm totally OK if we want to generalize this to a preemption score or whatever as long as applications have the ability to inform YARN in some way. bq. In YARN-3784, we were trying to send preemption timeout to AMs, so could AM take an immediate action about checkpointing OR even send a feedback to RM with alternative container to preempt instead of selected one? The problem with relying on the AM is that feedback loop can be too long for some use cases. For example, YARN-1011 is proposing to have the NM preempt containers on its own in order to preserve the health of the node when too many containers end up on the node and resource utilization is at critical levels. In those cases the NM doesn't have time to wait for the NM to tell the RM about it, have the RM wait for the AM to heartbeat in so it can tell the AM about it, wait for the AM to respond with a preemption preference, then wait for the NM to heartbeat in again so the RM can relay the priority. It would be nice if the NM could be proactively told during container execution when the cost of preemption changes so it can make better decisions on its own when pressed for time. > Ability for applications to notify YARN about container reuse > ------------------------------------------------------------- > > Key: YARN-7019 > URL: https://issues.apache.org/jira/browse/YARN-7019 > Project: Hadoop YARN > Issue Type: New Feature > Reporter: Jason Lowe > > During preemption calculations YARN can try to reduce the amount of work lost > by considering how long a container has been running. However when an > application framework like Tez reuses a container across multiple tasks it > changes the work lost calculation since the container has essentially > checkpointed between task assignments. It would be nice if applications > could inform YARN when a container has been reused/checkpointed and therefore > is a better candidate for preemption wrt. lost work than other, younger > containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org