[ https://issues.apache.org/jira/browse/SPARK-3456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14131993#comment-14131993 ]
Apache Spark commented on SPARK-3456: ------------------------------------- User 'tgravescs' has created a pull request for this issue: https://github.com/apache/spark/pull/2373 > YarnAllocator can lose container requests to RM > ----------------------------------------------- > > Key: SPARK-3456 > URL: https://issues.apache.org/jira/browse/SPARK-3456 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 1.2.0 > Reporter: Thomas Graves > Priority: Critical > > I haven't actually tested this yet, but I believe that spark on yarn can lose > container requests to the RM. The reason is that we ask for the total number > upfront (say x) but then we don't ask for anymore unless some are missing and > if we do then we could erase the original request. > For example > - ask for 3 containers > - 1 is allocated > - ask for 0 containers since asked for 3 originally (2 left) > - the 1 allocated dies > - We now ask for 1 since its missing, this will override whatever is on the > yarn side (in this case 2). > Then we lose the 2 more we need. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org