[ https://issues.apache.org/jira/browse/YARN-7019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139397#comment-16139397 ]
Joep Rottinghuis commented on YARN-7019: ---------------------------------------- Agreed that we should generalize this to a score. The same could be used for asking for additional resources. For example, if thousands mappers ran, and several reducers are done but tons more are running then a map output cannot be fetched (and a mapper has to be re-run), getting one more resource to quickly get this done and release a bunch or resources has one score. Earlier in the job execution when waves of mappers run, killing one would have a lower score. Similarly a score could be used to indicate to please not kill an app-master, or perhaps if one container gets killed, all the work is lost (in case of gang-scheduling needs). A generalized score of give me x containers and I can make Y progress, vs. cost of killing these containers is Y would be useful. I think the challenge will be to multiply an app or framework reported score to scale that to a fair share or priority for the application / user / queue so that apps cannot lie about these scores and rig scheduling decisions. > Ability for applications to notify YARN about container reuse > ------------------------------------------------------------- > > Key: YARN-7019 > URL: https://issues.apache.org/jira/browse/YARN-7019 > Project: Hadoop YARN > Issue Type: New Feature > Reporter: Jason Lowe > > During preemption calculations YARN can try to reduce the amount of work lost > by considering how long a container has been running. However when an > application framework like Tez reuses a container across multiple tasks it > changes the work lost calculation since the container has essentially > checkpointed between task assignments. It would be nice if applications > could inform YARN when a container has been reused/checkpointed and therefore > is a better candidate for preemption wrt. lost work than other, younger > containers. -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org