Lars Francke created SPARK-24049:
------------------------------------
Summary: Add a feature to not start speculative tasks before an
absolute minimum amount of time has passed
Key: SPARK-24049
URL: https://issues.apache.org/jira/browse/SPARK-24049
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.3.0
Reporter: Lars Francke
Speculation currently has four different configuration options according to the
docs:
* spark.speculation to turn it on or off
* .interval - how often it'll check
* .multiplier - how many times slower than average a task must be
* .quantile - fraction of tasks that have to be completed before speculation
starts
What I'd love to see is a feature that allows me to set a minimum average
duration. We are using a Jobserver to submit varying queries to Spark. Some of
those have stages with tasks of an average length of ~20ms (yes, not idea but
it happens). So with a multiplier of 2 a speculative task starts after 40ms.
This happens quite a lot and in our use-case it doesn't make sense. We don't
consider 40ms tasks "stragglers".
So it would make sense to add a parameter to only start speculation when the
average task time is greater than X seconds/minutes/millis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
