[
https://issues.apache.org/jira/browse/HIVE-9007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14265309#comment-14265309
]
Hive QA commented on HIVE-9007:
-------------------------------
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12690135/HIVE-9007.2-spark.patch
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 7282 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_groupby3_map_skew
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/602/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/602/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-602/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12690135 - PreCommit-HIVE-SPARK-Build
> Hive may generate wrong plan for map join queries due to
> IdentityProjectRemover [Spark Branch]
> ----------------------------------------------------------------------------------------------
>
> Key: HIVE-9007
> URL: https://issues.apache.org/jira/browse/HIVE-9007
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
> Affects Versions: spark-branch
> Reporter: Chao
> Assignee: Szehon Ho
> Attachments: HIVE-9007-spark.patch, HIVE-9007.2-spark.patch
>
>
> HIVE-8435 introduces a new logical optimizer called IdentityProjectRemover,
> which may cause map join in spark branch to generate wrong plan.
> Currently, the map join conversion in spark branch first goes through a
> method {{convertJoinMapJoin}}, which replaces a join op with a mapjoin op,
> removes RS associated with big table, and keep RSs for all small tables.
> Afterwards, in {{SparkReduceSinkMapJoinProc}} it replaces all parent RSs of
> the mapjoin op with HTS (note it doesn't check whether the RS belongs to
> small table or big table.)
> The issue arises, when IdentityProjectRemover comes into play, which may
> result into a situation that a operator tree has two consecutive RSs. Imaging
> the following example:
> {noformat}
> Join MapJoin
> / \ / \
> RS RS ---> RS RS
> / \ / \
> TS RS TS TS (big table)
> \ (small table)
> TS
> {noformat}
> In this case, all parents of the mapjoin op will be RS, even the branch for
> big table! In {{SparkReduceSinkMapJoinProc}}, they will be replaced with HTS,
> which is obviously incorrect.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
