[jira] [Commented] (SPARK-21171) Speculate task scheduling block dirve handle normal task when a job task number more than one hundred thousand
[ https://issues.apache.org/jira/browse/SPARK-21171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16060348#comment-16060348 ] wangminfeng commented on SPARK-21171: - We have modified some code for this feature, i will add a benchmark soon. It is the first time i contribute to community, give me some time for learn the rule. Thank you. > Speculate task scheduling block dirve handle normal task when a job task > number more than one hundred thousand > -- > > Key: SPARK-21171 > URL: https://issues.apache.org/jira/browse/SPARK-21171 > Project: Spark > Issue Type: Improvement > Components: Block Manager >Affects Versions: 2.0.0 > Environment: We have more than two hundred high-performance machine > to handle more than 2T data by one query >Reporter: wangminfeng > > If a job have more then one hundred thousand tasks and spark.speculation is > true, when speculable tasks start, choosing a speculable will waste lots of > time and block other tasks. We do a ad-hoc query for data analyse, we can't > tolerate one job wasting time even it is a large job -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21171) Speculate task scheduling block dirve handle normal task when a job task number more than one hundred thousand
[ https://issues.apache.org/jira/browse/SPARK-21171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16059018#comment-16059018 ] Sean Owen commented on SPARK-21171: --- There's no real detail here. I'd have to close this. This should start as a discussion on a mailing list, or at least include specific benchmarks and specific proposed changes. > Speculate task scheduling block dirve handle normal task when a job task > number more than one hundred thousand > -- > > Key: SPARK-21171 > URL: https://issues.apache.org/jira/browse/SPARK-21171 > Project: Spark > Issue Type: Improvement > Components: Block Manager >Affects Versions: 2.0.0 > Environment: We have more than two hundred high-performance machine > to handle more than 2T data by one query >Reporter: wangminfeng > > If a job have more then one hundred thousand tasks and spark.speculation is > true, when speculable tasks start, choosing a speculable will waste lots of > time and block other tasks. We do a ad-hoc query for data analyse, we can't > tolerate one job wasting time even it is a large job -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
