[ https://issues.apache.org/jira/browse/SPARK-24049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-24049. ---------------------------------- Resolution: Incomplete > Add a feature to not start speculative tasks when average task duration is > less than a configurable absolute number > ------------------------------------------------------------------------------------------------------------------- > > Key: SPARK-24049 > URL: https://issues.apache.org/jira/browse/SPARK-24049 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 2.3.0 > Reporter: Lars Francke > Priority: Minor > Labels: bulk-closed > > Speculation currently has four different configuration options according to > the docs: > * spark.speculation to turn it on or off > * .interval - how often it'll check > * .multiplier - how many times slower than average a task must be > * .quantile - fraction of tasks that have to be completed before speculation > starts > What I'd love to see is a feature that allows me to set a minimum average > duration. We are using a Jobserver to submit varying queries to Spark. Some > of those have stages with tasks of an average length of ~20ms (yes, not idea > but it happens). So with a multiplier of 2 a speculative task starts after > 40ms. This happens quite a lot and in our use-case it doesn't make sense. We > don't consider 40ms tasks "stragglers". > So it would make sense to add a parameter to only start speculation when the > average task time is greater than X seconds/minutes/millis. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org