[
https://issues.apache.org/jira/browse/HIVE-24308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Panagiotis Garefalakis updated HIVE-24308:
------------------------------------------
Description:
Found a weird scenario when looking at the ConvertJoinMapJoin logic:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1198]
When the distinct keys cannot fit in memory AND the DPHJ ShuffleSize is lower
than expected the code returns a MJ because of the condition above!
In general, I believe the ShuffleSize check:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1624]
should be part of the shuffleJoin DPHJ conversion.
And the preferred conversion would be: MJ > DPHJ > SMB
was:
Found a weird scenario when looking at the ConvertJoinMapJoin logic:
[https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1198]
When the distinct keys cannot fit in memory AND the DPHJ ShuffleSize is lower
than expected the code returns a MJ because of the condition above!
In general, I believe the ShuffleSize check:
[https://github.com/apache/hive/blob/052c9da958f5cf3998091a7eb4b24192a5bb61e9/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1624]
should be part of the shuffleJoin DPHJ conversion.
And the preferred conversion would be: MJ > DPHJ > SMB
> FIX conditions used for DPHJ conversion
> -----------------------------------------
>
> Key: HIVE-24308
> URL: https://issues.apache.org/jira/browse/HIVE-24308
> Project: Hive
> Issue Type: Bug
> Reporter: Panagiotis Garefalakis
> Assignee: Panagiotis Garefalakis
> Priority: Major
>
> Found a weird scenario when looking at the ConvertJoinMapJoin logic:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1198]
> When the distinct keys cannot fit in memory AND the DPHJ ShuffleSize is
> lower than expected the code returns a MJ because of the condition above!
> In general, I believe the ShuffleSize check:
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1624]
> should be part of the shuffleJoin DPHJ conversion.
> And the preferred conversion would be: MJ > DPHJ > SMB
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
