[ https://issues.apache.org/jira/browse/SPARK-9396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14647258#comment-14647258 ]
prakhar jauhari commented on SPARK-9396: ---------------------------------------- Can you please assign this issue to me, I am adding a PR. > Spark yarn allocator does not call "removeContainerRequest" for allocated > Container requests, resulting in bloated ask[] toYarn RM. > ----------------------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-9396 > URL: https://issues.apache.org/jira/browse/SPARK-9396 > Project: Spark > Issue Type: Bug > Components: YARN > Affects Versions: 1.2.1 > Environment: Spark-1.2.1 on hadoop-yarn-2.4.0 cluster. All servers in > cluster running Linux version 2.6.32. > Reporter: prakhar jauhari > > Note : Attached logs contain logs that i added (spark yarn allocator side and > Yarn client side) for debugging purpose. > !!!!! My spark job is configured for 2 executors, on killing 1 executor the > ask is of 3 !!!!!!! > On killing a executor - resource request logs : > *************Killed container: ask for 3 containers, instead for 1*********** > 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: Will allocate 1 executor > containers, each with 2432 MB memory including 384 MB overhead > 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: numExecutors: 1 > 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: host preferences is empty > 15/07/15 10:49:01 INFO yarn.YarnAllocationHandler: Container request (host: > Any, priority: 1, capability: <memory:2432, vCores:4> > 15/07/15 10:49:01 INFO impl.AMRMClientImpl: prakhar : AMRMClientImpl : > allocate: this.ask = [{Priority: 1, Capability: <memory:2432, vCores:4>, # > Containers: 3, Location: *, Relax Locality: true}] > 15/07/15 10:49:01 INFO impl.AMRMClientImpl: prakhar : AMRMClientImpl : > allocate: allocateRequest = ask { priority{ priority: 1 } resource_name: "*" > capability { memory: 2432 virtual_cores: 4 } num_containers: 3 > relax_locality: true } blacklist_request { } response_id: 354 progress: 0.1 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org