[jira] [Commented] (HIVE-9424) SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285956#comment-14285956
]
Xuefu Zhang commented on HIVE-9424:
---
[~csun], to clarify, w/o this patch, there is no problem. If this is case, I
don't think this is a blocker (for 0.15). On the contrary, I'm a little
concerned to include this in 0.15. Please share your thought.
SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark
Branch]
---
Key: HIVE-9424
URL: https://issues.apache.org/jira/browse/HIVE-9424
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chao
Assignee: Chao
Priority: Blocker
Attachments: HIVE-9424.1-spark.patch
Sometimes, in SparkMapJoinResolver, after a new SparkTask is generated and
added to context, it may be processed by the {{dispatch}} method in this
class. This could introduce wrong plan since {{generateLocalWork}} will
overwrite existing local work in the SparkTask.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9424) SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286137#comment-14286137
]
Xuefu Zhang commented on HIVE-9424:
---
I see. Please feel free to include this in your prototyping. We will visit this
when we review that work. Thanks.
SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark
Branch]
---
Key: HIVE-9424
URL: https://issues.apache.org/jira/browse/HIVE-9424
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9424.1-spark.patch
Sometimes, in SparkMapJoinResolver, after a new SparkTask is generated and
added to context, it may be processed by the {{dispatch}} method in this
class. This could introduce wrong plan since {{generateLocalWork}} will
overwrite existing local work in the SparkTask.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9424) SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14286124#comment-14286124
]
Chao commented on HIVE-9424:
[~xuefuz] I think the issue will not happen with the current impl - it could
only occur when the main SparkTask is not in the root task set, due to the
post order traversing of the TaskGraphWalker. In my dynamic partition
implementation this case could happen, since the main task could be child of
pruning task.
Yes, I think it's OK not to include it in 0.15. It will not affect anything
that we have right now.
SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark
Branch]
---
Key: HIVE-9424
URL: https://issues.apache.org/jira/browse/HIVE-9424
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chao
Assignee: Chao
Priority: Blocker
Attachments: HIVE-9424.1-spark.patch
Sometimes, in SparkMapJoinResolver, after a new SparkTask is generated and
added to context, it may be processed by the {{dispatch}} method in this
class. This could introduce wrong plan since {{generateLocalWork}} will
overwrite existing local work in the SparkTask.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9424) SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285849#comment-14285849
]
Chao commented on HIVE-9424:
[~brocknoland] I'm fine making it a blocker - at least it blocks dynamic
partition pruning.
Also, I don't think the test failures are related.
SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark
Branch]
---
Key: HIVE-9424
URL: https://issues.apache.org/jira/browse/HIVE-9424
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chao
Assignee: Chao
Priority: Blocker
Attachments: HIVE-9424.1-spark.patch
Sometimes, in SparkMapJoinResolver, after a new SparkTask is generated and
added to context, it may be processed by the {{dispatch}} method in this
class. This could introduce wrong plan since {{generateLocalWork}} will
overwrite existing local work in the SparkTask.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9424) SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285180#comment-14285180
]
Hive QA commented on HIVE-9424:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12693491/HIVE-9424.1-spark.patch
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 7312 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_memcheck
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hive.hcatalog.streaming.TestStreaming.testEndpointConnection
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/656/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/656/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-656/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12693491 - PreCommit-HIVE-SPARK-Build
SparkMapJoinResolver shouldn't process newly generated SparkTask [Spark
Branch]
---
Key: HIVE-9424
URL: https://issues.apache.org/jira/browse/HIVE-9424
Project: Hive
Issue Type: Bug
Affects Versions: spark-branch
Reporter: Chao
Assignee: Chao
Attachments: HIVE-9424.1-spark.patch
Sometimes, in SparkMapJoinResolver, after a new SparkTask is generated and
added to context, it may be processed by the {{dispatch}} method in this
class. This could introduce wrong plan since {{generateLocalWork}} will
overwrite existing local work in the SparkTask.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
