[jira] [Created] (HIVE-25335) Unreasonable setting reduce number, when join big size table(but small row count) and small size table

Thu, 15 Jul 2021 02:18:27 -0700 

zhengchenyu created HIVE-25335:
----------------------------------

             Summary: Unreasonable setting reduce number, when join big size 
table(but small row count) and small size table
                 Key: HIVE-25335
                 URL: https://issues.apache.org/jira/browse/HIVE-25335
             Project: Hive
          Issue Type: Improvement
            Reporter: zhengchenyu


I found an application which is slow in our cluster, because the proccess bytes 
of one reduce is very huge, but only two reduce. 
when I debug, I found the reason. Because in this sql, one big size table 
(about 30G) with few row count(about 3.5M), another small size table (about 
100M) have more row count (about 3.6M). So JoinStatsRule.process only use 100M 
to estimate reducer's number. But we need to  process 30G byte in fact.  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to