yongzhi.shao created HIVE-28583:
-----------------------------------
Summary: In the case of subqueries, HIVE often incorrectly uses
MAP-JOIN for large tables.
Key: HIVE-28583
URL: https://issues.apache.org/jira/browse/HIVE-28583
Project: Hive
Issue Type: Bug
Security Level: Public (Viewable by anyone)
Reporter: yongzhi.shao
Hello. Team.
Currently we found that in version 4.0, HIVE many many occasions will be
wrongly estimated table size, and thus incorrectly use MapJoin to optimise,
ultimately leading to a series of problems such as
OOM/MapJoinMemoryExhaustionError.
We found a typical scenario as follows:
{code:java}
---dataset size
select
c1,c2,c3
from big_table_2; 50GB
big_table_1 1TB；
----- use map join. and cause oom/MapJoinMemoryExhaustionError
select
*
from
big_table_1 t1
join
(
select
c1,c2,c3
from big_table_2
) t2 on xxxxx;
----- use smj. no map join. job success
create table t2 as
select
c1,c2,c3
from big_table_2;
select
*
from
big_table_1 t1
join
t2 on xxxxx;
{code}
The above SQL can be executed normally in HIVE3.
Can anyone guide me on how to deal with this kind of problem?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
