Re: Spark (1.2) yarn allocator does not remove container request for allocated container, resulting in a bloated ask[] of containers and inefficient resource utilization of cluster resources.

hey all, Thanks in advance. I am facing this issue in production, where due to increased container request RM is reserving memory and hampering cluster utilization. Thus the fix needs to be patched on spark 1.2. Has any one looked in the removeContainerRequest part for allocated containers in

Spark (1.2) yarn allocator does not remove container request for allocated container, resulting in a bloated ask[] of containers and inefficient resource utilization of cluster resources.

This is because Yarn's AM client does not remove fulfilled container request from its MAP until the application's AM specifically calls removeContainerRequest for fulfilled container requests. Spark-1.2 : Spark's yarn AM does not call removeContainerRequest for fulfilled container request.