GitHub user susanxhuynh opened a pull request:
https://github.com/apache/spark/pull/18674
[SPARK-21456][MESOS] Make the driver failover_timeout configurable
## What changes were proposed in this pull request?
Current behavior: in Mesos cluster mode, the driver failover_timeout is set
to zero. If the driver temporarily loses connectivity with the Mesos master,
the framework will be torn down and all executors killed.
Proposed change: make the failover_timeout configurable via a new option,
spark.mesos.driver.failoverTimeout. The default value is still zero.
Note: with non-zero failover_timeout, an explicit teardown is needed in
some cases. This is captured in
https://issues.apache.org/jira/browse/SPARK-21458
## How was this patch tested?
Added a unit test to make sure the config option is set while creating the
scheduler driver.
Ran an integration test with mesosphere/spark showing that with a non-zero
failover_timeout the Spark job finishes after a driver is disconnected from the
master.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mesosphere/spark sh-mesos-failover-timeout
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18674.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18674
----
commit ae3e5bd0e0ab139f9d871d667c7fad7f2682285d
Author: Susan X. Huynh <[email protected]>
Date: 2017-07-18T18:07:53Z
Made the driver failover_timeout configurable as
spark.mesos.driver.failoverTimeout. Added a unit test.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
