[
https://issues.apache.org/jira/browse/HIVE-28583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17891270#comment-17891270
]
yongzhi.shao commented on HIVE-28583:
-------------------------------------
[~zabetak]
Hello.
Have you encountered this problem?
> In the case of subqueries, HIVE often incorrectly uses MAP-JOIN for large
> tables.
> ---------------------------------------------------------------------------------
>
> Key: HIVE-28583
> URL: https://issues.apache.org/jira/browse/HIVE-28583
> Project: Hive
> Issue Type: Bug
> Security Level: Public(Viewable by anyone)
> Reporter: yongzhi.shao
> Priority: Major
>
> Hello. Team.
>
> Currently we found that in version 4.0.1, HIVE many many occasions will be
> wrongly estimated table size, and thus incorrectly use MapJoin to optimise,
> ultimately leading to a series of problems such as
> OOM/MapJoinMemoryExhaustionError.
>
> We found a typical scenario as follows:
> {code:java}
> ---dataset size
> select
> c1,c2,c3
> from big_table_2; 50GB
>
> big_table_1 1TB；
>
>
> ----- use map join. and cause oom/MapJoinMemoryExhaustionError
> select
> *
> from
> big_table_1 t1
> join
> (
> select
> c1,c2,c3
> from big_table_2
> ) t2 on xxxxx;
>
>
>
> ----- use smj. no map join. job success
> create table t2 as
> select
> c1,c2,c3
> from big_table_2;
> select
> *
> from
> big_table_1 t1
> join
> t2 on xxxxx;
> {code}
> The above SQL can be executed normally in HIVE3.
> Can anyone guide me on how to deal with this kind of problem?
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
