himanshu-mishra opened a new pull request, #374: URL: https://github.com/apache/tez/pull/374
When container reuse is enabled, preemption of lower priority containers that are not yet assigned to task, takes long time as they are released one at a time, and not the number of containers based when tez.am.preemption.percentage is high added in https://issues.apache.org/jira/browse/TEZ-1742. Further investigation lead to following conclusion: 1. [Warn log / Assertion error](https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L1335) thrown because in [preemptIfNeeded()](https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L1314), when releasing new containers, the loop counter is being decremented with each `releaseUnassignedContainers`, leading to looping only half number of times. By using another counter, assertion passes because of condition method returns with check `if (numPendingRequestsToService < 1) {`. 2. In [releaseContainer()](https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L1566), the container is not getting removed from `delayedContainers` queue and only from `heldContainers` map, hence same container is being picked up for release in every iteration till next cycle of `DelayedContainerManager` finds out that the container is not in `heldContainers` and skips it [with log ](https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/YarnTaskSchedulerService.java#L2095)`Skipping delayed container as container is no longer running, containerId=...` This change adds a method in `DelayedContainerManager` to allow removal of delayed container and invokes it in releaseContainer method, which so far only removed it from `heldContainers` map. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
