Is the small-side table large, does it have a lot of rows for the same keys, or does it have a lot of skew? Are there lots of misses (where there’d be no value in the small table for the large table value)?
If you have enough memory you can try increasing initial size and decreasing load factor. Although without low-level debugging it’s hard to tell if the issue is not obvious (I.e the above). If there’s no obvious problem you might consider not using map join. From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Reply-To: user <[email protected]<mailto:[email protected]>> Date: Thursday, August 27, 2015 at 18:51 To: user <[email protected]<mailto:[email protected]>> Subject: Re: Re: sql mapjoin very slow I use MR. My mapjoin config as showed in follow picture: [cid:_Foxmail.1@7f3eed6a-4406-fa48-f0a1-ec347b3ed46e] [cid:_Foxmail.1@b40e2cf8-de17-015c-7e26-f2a86647137d] ________________________________ [email protected]<mailto:[email protected]> From: Sergey Shelukhin<mailto:[email protected]> Date: 2015-08-28 09:21 To: user<mailto:[email protected]> Subject: Re: sql mapjoin very slow Are you using MR and Tez? You could try optimized hash table in case of Tez, although it’s supposed to improve memory, not necessarily perf. Can you also share characteristics of the query and data? It is surprising to see so much time for HashMap.get. From: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Reply-To: user <[email protected]<mailto:[email protected]>> Date: Thursday, August 27, 2015 at 18:03 To: user <[email protected]<mailto:[email protected]>> Subject: sql mapjoin very slow When I enable mapjoin ,I see Mapjoin task run very slow. My envrioment is hadoop 2.3.0 hive 1.1.0. My attach is one map hive log and this map's xprof log. In map xprof log ,I see Compiled + native Method 92.3% 643527 + 0 java.util.HashMap.get 2.8% 19856 + 0 java.util.HashMap.put 1.2% 8623 + 0 org.apache.hadoop.hive.ql.exec.persistence.HashMapWrapper$GetAdaptor.setFromRow 0.1% 953 + 0 org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate 0.1% 576 + 0 org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject ________________________________ [email protected]<mailto:[email protected]>
