[
https://issues.apache.org/jira/browse/TEZ-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13945434#comment-13945434
]
Siddharth Seth commented on TEZ-708:
------------------------------------
[~jeagles], thanks for posting the patch without the TaskScheduler rename,
makes it much easier to look at.
Comments
getAvailableResources - should this be using numContainers instead of JVM
memory ? Eventually, it's used by Vertex Plugins and InputInitializers to
figure out number of splits - so this may just be fine.
getTotalResources - should this use .totalMemory or .maxMemory ?
The taskRequesutQueue could be a priority queue - with priority being the sort
field. That doesn't solve the problem of a task failing which may require a
rerun - untill some form of pre-emption is in place, but increases the chances
of the correct task being scheduled.
allocateRequest only adds to the taskAllocations if number of entries in that
is less than maxTasks. It looks like the allocate request will be lost if
multiple tasks (> maxTasks) are running - since the allocateRequests is removed
from the queue and never put back.
Also, if using the same queue to process alloacteRequests and
deallocateRequests - I think putting de-allocates as highest priority would be
useful.
new ContainerPBImpl - please use Container.newInstance instead (or the
BuilderUtils). Containers are private to YARN - so this isn't safe in terms of
compatibility. However, there's already multiple tests which do the same; we
may need to define our own Container construct - but that's a completely
different jira.
> Avoid asking for new containers in uber-local mode
> --------------------------------------------------
>
> Key: TEZ-708
> URL: https://issues.apache.org/jira/browse/TEZ-708
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Chen He
> Assignee: Jonathan Eagles
> Attachments: TEZ-708.patch
>
>
> Once we finish this task, the TaskSchedulerEventHandler should not ask for
> new container in the Uber-mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
