[ https://issues.apache.org/jira/browse/SPARK-32170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17151288#comment-17151288 ]
Apache Spark commented on SPARK-32170: -------------------------------------- User 'weixiuli' has created a pull request for this issue: https://github.com/apache/spark/pull/28994 > Improve the speculation for the inefficient tasks by the task metrics. > ----------------------------------------------------------------------- > > Key: SPARK-32170 > URL: https://issues.apache.org/jira/browse/SPARK-32170 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Spark Core > Affects Versions: 3.0.0 > Reporter: weixiuli > Priority: Major > Fix For: 3.0.0 > > > 1) Tasks will be speculated when meet certain conditions no matter they are > inefficient or not,this would be a huge waste of cluster resources. > 2) In production,the speculation task comes from an efficient one will be > killed finally,which is unnecessary and will waste of cluster resources. > 3) So, we should evaluate whether the task is inefficient by success tasks > metrics firstly, and then decide to speculate it or not. The inefficient > task will be speculated and efficient one will not, it is better for the > cluster resources. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org