GitHub user sebastienrainville opened a pull request:
https://github.com/apache/spark/pull/1140
Launch tasks on the proper executors in mesos fine-grained mode
The scheduler for Mesos in fine-grained mode launches tasks on random
executors. MesosSchedulerBackend.resourceOffers(SchedulerDriver, List[Offer])
is assuming that TaskSchedulerImpl.resourceOffers(Seq[WorkerOffer]) is
returning task lists in the same order as the offers it was passed, but in the
current implementation TaskSchedulerImpl.resourceOffers shuffles the offers to
avoid assigning the tasks always to the same executors. The result is that the
tasks are launched on random executors.
The solution is to not assume that the order in which the tasks are
returned is the same as the offers. What I am not sure about is that I
considered slaveId and executorId to be the same, which is true at least in my
setup, but I don't know if that is always true.
I tested this and it seems to work fine on our cluster.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sebastienrainville/spark
fine-grained-mode-fix-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1140.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1140
----
commit d6ffe540e2824e2e518a3638c43b32ff57edd716
Author: Sebastien Rainville <sebast...@hopper.com>
Date: 2014-06-19T17:46:40Z
Launch tasks on the proper executors in mesos fine-grained mode
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
