[jira] [Commented] (TEZ-2218) Turn on speculation by default
[ https://issues.apache.org/jira/browse/TEZ-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14372450#comment-14372450 ] Siddharth Seth commented on TEZ-2218: - Are there any pre-conditions for this, testing required etc ? We may be better leaving it off by default - especially for single node instances, and allow this to be enabled per site. Progress reporting from tasks being one of the concerns. > Turn on speculation by default > -- > > Key: TEZ-2218 > URL: https://issues.apache.org/jira/browse/TEZ-2218 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (TEZ-2218) Turn on speculation by default
[ https://issues.apache.org/jira/browse/TEZ-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16491428#comment-16491428 ] Sergey Shelukhin commented on TEZ-2218: --- Hmm... > Turn on speculation by default > -- > > Key: TEZ-2218 > URL: https://issues.apache.org/jira/browse/TEZ-2218 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2218) Turn on speculation by default
[ https://issues.apache.org/jira/browse/TEZ-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503381#comment-16503381 ] Jonathan Eagles commented on TEZ-2218: -- [~sershe], there are some issues with enabling speculative execution. Despite no visible work in this JIRA for a while, there has been much recent progress. Namely, TEZ-3938, TEZ-3941, TEZ-3934, TEZ-3917, TEZ-3952 come up as recently filed JIRAs in various states. Apart from those bug fixes, there are more difficult issues to address. In order for this to be turned on by default, set progress api needs to be implemented by the consuming applications (Pig, Hive) and their supported version (otherwise we will greatly over-speculate). This will be a good candidate for 0.10.0 as that will reduce the versions of Pig and Hive to ones that support Hadoop 3+ which will have implemented the progress api in those versions. > Turn on speculation by default > -- > > Key: TEZ-2218 > URL: https://issues.apache.org/jira/browse/TEZ-2218 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2218) Turn on speculation by default
[ https://issues.apache.org/jira/browse/TEZ-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503385#comment-16503385 ] Jonathan Eagles commented on TEZ-2218: -- The other consideration, is that for Pig and Hive, they are converting the mapreduce.map.speculative mapreduce.reduce.speculative settings which are defaulted to true in the hadoop distribution. From that perspective there may not be any noticeable change from those application (Pig and Hive). > Turn on speculation by default > -- > > Key: TEZ-2218 > URL: https://issues.apache.org/jira/browse/TEZ-2218 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-2218) Turn on speculation by default
[ https://issues.apache.org/jira/browse/TEZ-2218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16503803#comment-16503803 ] Sergey Shelukhin commented on TEZ-2218: --- Thanks for the info... as far as I know Hive doesn't turn on speculative execution on Tez. But it would potentially be smth useful :) > Turn on speculation by default > -- > > Key: TEZ-2218 > URL: https://issues.apache.org/jira/browse/TEZ-2218 > Project: Apache Tez > Issue Type: Sub-task >Reporter: Bikas Saha >Assignee: Bikas Saha >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
