[
https://issues.apache.org/jira/browse/TEZ-708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13919108#comment-13919108
]
Bikas Saha edited comment on TEZ-708 at 3/4/14 7:58 AM:
--------------------------------------------------------
I dont think the TaskScheduler needs to get involved here. If the DAG chooses
to run a container within the AM then there is really no scheduling involved
where the TaskScheduler can be useful. What we need is something in the middle
that re-directs to the TaskScheduler if the task needs to run externally or the
LocalTaskScheduler if the task needs to run inside. The
TaskSchedulerEventHandler sits in-between the TaskScheduler and the rest of the
DAG and could be a place to host this logic. The TaskAttempt sends a "I need a
container" event to the TaskSchedulerEventHandler which then redirects to
TaskScheduler for remote containers or LocalTaskScheduler for local containers.
Depending on our local mode implementation, LocalTaskScheduler could be as
simple as immediately forwarding the request to our LocalContainerLauncher,
where the request is serviced or waits for local execution threads to become
available. This way both execution modes remain cleanly separated until we
figure out when/how to go mixed.
The TaskScheduler is already very complicated and in need of some structural
changes. My suggestion would be to avoid touching it for local mode. It will
only make life hard for you without any obvious/immediate benefit. Lets get
full local mode up and running. Later, once we figure out how to run mixed
mode, we can revisit potential for merging logic.
was (Author: bikassaha):
I dont quite understand why the TaskScheduler needs to get involved here. If
the DAG chooses to run a container within the AM then there is really no
scheduling involved where the TaskScheduler can be useful. What we need is
something in the middle that re-directs to the TaskScheduler if the task needs
to run outside or the local launcher if the task needs to run inside. The
TaskSchedulerEventHandler sits in-between the TaskScheduler and the rest of the
DAG and could be a place to host this logic. The TaskAttempt sends a "I need a
container" event to the TaskSchedulerEventHandler which then redirects to
TaskScheduler for remote containers or LocalTaskScheduler for local containers.
Depending on our local mode implementation, LocalTaskScheduler could be as
simple as immediately forwarding the request to our LocalContainerLauncher,
where the request is serviced or waits for local execution threads to become
available. This way both execution modes remain cleanly separated until we
figure out when/how to go mixed.
The TaskScheduler is already very complicated and in need of some structural
changes. My suggestion would be to avoid touching it for local mode. It will
only make life hard for you without any obvious/immediate benefit. Lets get
full local mode up and running. Later, once we figure out how to run mixed
mode, we can revisit potential for merging logic.
> Avoid asking for new containers in uber-local mode
> --------------------------------------------------
>
> Key: TEZ-708
> URL: https://issues.apache.org/jira/browse/TEZ-708
> Project: Apache Tez
> Issue Type: Sub-task
> Reporter: Chen He
> Assignee: Jonathan Eagles
>
> Once we finish this task, the TaskSchedulerEventHandler should not ask for
> new container in the Uber-mode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
