> On Jan. 27, 2014, 9:52 p.m., Vikram Dixit Kumaraswamy wrote: > > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java, line 90 > > <https://reviews.apache.org/r/16728/diff/4/?file=431458#file431458line90> > > > > This would break tez tests.
Included by mistake (described in HIVE-6241). I'll remove this. > On Jan. 27, 2014, 9:52 p.m., Vikram Dixit Kumaraswamy wrote: > > itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java, line 369 > > <https://reviews.apache.org/r/16728/diff/4/?file=431458#file431458line369> > > > > This would eliminate tez unit tests. Was this intentional? Same with above one > On Jan. 27, 2014, 9:52 p.m., Vikram Dixit Kumaraswamy wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java, > > line 142 > > <https://reviews.apache.org/r/16728/diff/4/?file=431471#file431471line142> > > > > Could you raise a jira for this. Ok. sure. > On Jan. 27, 2014, 9:52 p.m., Vikram Dixit Kumaraswamy wrote: > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java, > > line 165 > > <https://reviews.apache.org/r/16728/diff/4/?file=431471#file431471line165> > > > > can it be only these 2 operators? Maybe common join operator can be > > used? Logically, it might possible but in real, it seemed not possible to have CommonJoinOperator as a parent which would be in reducer task. Tez could exploit that but I don't know well on tez. - Navis ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16728/#review32885 ----------------------------------------------------------- On Jan. 28, 2014, 1:37 a.m., Navis Ryu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/16728/ > ----------------------------------------------------------- > > (Updated Jan. 28, 2014, 1:37 a.m.) > > > Review request for hive. > > > Bugs: HIVE-6144 > https://issues.apache.org/jira/browse/HIVE-6144 > > > Repository: hive-git > > > Description > ------- > > For map join, all data in small aliases are hashed and stored into temporary > file in MapRedLocalTask. But for some aliases without filter or projection, > it seemed not necessary to do that. For example. > > {noformat} > select a.* from src a join src b on a.key=b.key; > {noformat} > > makes plan like this. > {noformat} > STAGE PLANS: > Stage: Stage-4 > Map Reduce Local Work > Alias -> Map Local Tables: > a > Fetch Operator > limit: -1 > Alias -> Map Local Operator Tree: > a > TableScan > alias: a > HashTable Sink Operator > condition expressions: > 0 {key} {value} > 1 > handleSkewJoin: false > keys: > 0 [Column[key]] > 1 [Column[key]] > Position of Big Table: 1 > > Stage: Stage-3 > Map Reduce > Alias -> Map Operator Tree: > b > TableScan > alias: b > Map Join Operator > condition map: > Inner Join 0 to 1 > condition expressions: > 0 {key} {value} > 1 > handleSkewJoin: false > keys: > 0 [Column[key]] > 1 [Column[key]] > outputColumnNames: _col0, _col1 > Position of Big Table: 1 > Select Operator > File Output Operator > Local Work: > Map Reduce Local Work > Stage: Stage-0 > Fetch Operator > {noformat} > > table src(a) is fetched and stored as-is in MRLocalTask. With this patch, > plan can be like below. > {noformat} > Stage: Stage-3 > Map Reduce > Alias -> Map Operator Tree: > b > TableScan > alias: b > Map Join Operator > condition map: > Inner Join 0 to 1 > condition expressions: > 0 {key} {value} > 1 > handleSkewJoin: false > keys: > 0 [Column[key]] > 1 [Column[key]] > outputColumnNames: _col0, _col1 > Position of Big Table: 1 > Select Operator > File Output Operator > Local Work: > Map Reduce Local Work > Alias -> Map Local Tables: > a > Fetch Operator > limit: -1 > Alias -> Map Local Operator Tree: > a > TableScan > alias: a > Has Any Stage Alias: false > Stage: Stage-0 > Fetch Operator > {noformat} > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84ee78f > conf/hive-default.xml.template 66d22f9 > > itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/MapJoinCounterHook.java > 1b0d57e > ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java > d8f4eb4 > ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java a080fcc > ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java > fc08b28 > ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 1e0314d > ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java bdc85b9 > ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 56676df > ql/src/java/org/apache/hadoop/hive/ql/exec/TemporaryHashSinkOperator.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 22e5777 > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java 58484af > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 2d2508d > ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 2df8ab9 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java > 5a53e15 > > ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java > 83b8d6e > > ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java > ebccb14 > ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java c30da56 > ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 709c50e > ql/src/test/queries/clientpositive/auto_join_without_localtask.q > PRE-CREATION > ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 0595cd6 > ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out > fa261b3 > ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out > bca069a > ql/src/test/results/clientpositive/auto_join1.q.out b93c10f > ql/src/test/results/clientpositive/auto_join15.q.out 6bfcfc7 > ql/src/test/results/clientpositive/auto_join17.q.out 698270e > ql/src/test/results/clientpositive/auto_join19.q.out 247b081 > ql/src/test/results/clientpositive/auto_join2.q.out 756cfa1 > ql/src/test/results/clientpositive/auto_join21.q.out 053475f > ql/src/test/results/clientpositive/auto_join22.q.out 6e5ae4c > ql/src/test/results/clientpositive/auto_join24.q.out dbecb2c > ql/src/test/results/clientpositive/auto_join25.q.out d6d0c90 > ql/src/test/results/clientpositive/auto_join26.q.out 7ca6bb9 > ql/src/test/results/clientpositive/auto_join28.q.out 488e15c > ql/src/test/results/clientpositive/auto_join29.q.out 7798a0e > ql/src/test/results/clientpositive/auto_join3.q.out 4279e8c > ql/src/test/results/clientpositive/auto_join30.q.out a46ea2a > ql/src/test/results/clientpositive/auto_join31.q.out 4504426 > ql/src/test/results/clientpositive/auto_join32.q.out da4ef55 > ql/src/test/results/clientpositive/auto_join9.q.out d2589ed > ql/src/test/results/clientpositive/auto_join_without_localtask.q.out > PRE-CREATION > ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 8e11616 > ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 95e86e1 > ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out fd8e9cc > ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out 3c20e0f > ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out 6189df6 > ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c00b519 > ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 42bab30 > ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out ec33acd > ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out f8c164a > ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 6ee49a7 > ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 3f2672e > ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 4cfd022 > ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 7b9f811 > ql/src/test/results/clientpositive/bucket_map_join_1.q.out f052025 > ql/src/test/results/clientpositive/bucket_map_join_2.q.out eb51e23 > ql/src/test/results/clientpositive/bucketcontext_1.q.out 9bdc1ee > ql/src/test/results/clientpositive/bucketcontext_2.q.out d5d67dc > ql/src/test/results/clientpositive/bucketcontext_3.q.out b226246 > ql/src/test/results/clientpositive/bucketcontext_4.q.out c27d927 > ql/src/test/results/clientpositive/bucketcontext_5.q.out 819362f > ql/src/test/results/clientpositive/bucketcontext_6.q.out d7b1ad7 > ql/src/test/results/clientpositive/bucketcontext_7.q.out 80749f0 > ql/src/test/results/clientpositive/bucketcontext_8.q.out b8a4d4f > ql/src/test/results/clientpositive/bucketmapjoin1.q.out 8d24d6f > ql/src/test/results/clientpositive/bucketmapjoin10.q.out 371af7d > ql/src/test/results/clientpositive/bucketmapjoin11.q.out fcfa41f > ql/src/test/results/clientpositive/bucketmapjoin12.q.out aa50cfe > ql/src/test/results/clientpositive/bucketmapjoin13.q.out eceeade > ql/src/test/results/clientpositive/bucketmapjoin2.q.out 1dfe3e7 > ql/src/test/results/clientpositive/bucketmapjoin3.q.out ba1962d > ql/src/test/results/clientpositive/bucketmapjoin4.q.out a9c4bce > ql/src/test/results/clientpositive/bucketmapjoin5.q.out 4812a21 > ql/src/test/results/clientpositive/bucketmapjoin7.q.out f8ea2e2 > ql/src/test/results/clientpositive/bucketmapjoin8.q.out 8e2be73 > ql/src/test/results/clientpositive/bucketmapjoin9.q.out 38b9c87 > ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 0139546 > ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out bf00114 > ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out 2cc1a0f > ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out > 0014891 > ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out > 14d0b6a > ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out > e83ef59 > ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out > b5828f0 > ql/src/test/results/clientpositive/correlationoptimizer1.q.out 3ef5cf2 > ql/src/test/results/clientpositive/correlationoptimizer3.q.out e633287 > ql/src/test/results/clientpositive/correlationoptimizer4.q.out c037d3c > ql/src/test/results/clientpositive/correlationoptimizer5.q.out 02ba37a > ql/src/test/results/clientpositive/correlationoptimizer6.q.out 557ee9e > ql/src/test/results/clientpositive/correlationoptimizer7.q.out b1fe60a > ql/src/test/results/clientpositive/explain_rearrange.q.out 47b1afb > ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out > effa6c8 > ql/src/test/results/clientpositive/join25.q.out 630d862 > ql/src/test/results/clientpositive/join26.q.out 9fca2c3 > ql/src/test/results/clientpositive/join27.q.out 0ad8bff > ql/src/test/results/clientpositive/join28.q.out f50072c > ql/src/test/results/clientpositive/join29.q.out 0e0bbb4 > ql/src/test/results/clientpositive/join30.q.out 924579c > ql/src/test/results/clientpositive/join31.q.out 85b0196 > ql/src/test/results/clientpositive/join32.q.out 9e84d84 > ql/src/test/results/clientpositive/join32_lessSize.q.out 0f6f8ec > ql/src/test/results/clientpositive/join33.q.out 9e84d84 > ql/src/test/results/clientpositive/join34.q.out 1e2aa91 > ql/src/test/results/clientpositive/join35.q.out 83f5ba2 > ql/src/test/results/clientpositive/join36.q.out 922ddef > ql/src/test/results/clientpositive/join37.q.out db905bd > ql/src/test/results/clientpositive/join_map_ppr.q.out 131e65c > ql/src/test/results/clientpositive/join_reorder4.q.out b14ea08 > ql/src/test/results/clientpositive/join_star.q.out 9b1adbf > ql/src/test/results/clientpositive/mapjoin1.q.out e8b4fa8 > ql/src/test/results/clientpositive/mapjoin_distinct.q.out c4dc9d2 > ql/src/test/results/clientpositive/mapjoin_hook.q.out 8da5168 > ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 2c9f7f6 > ql/src/test/results/clientpositive/mapjoin_subquery.q.out e90a50a > ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 220bf322 > ql/src/test/results/clientpositive/mapjoin_test_outer.q.out d66e079 > ql/src/test/results/clientpositive/multiMapJoin1.q.out 624b47f > ql/src/test/results/clientpositive/multiMapJoin2.q.out 6d0c7d2 > ql/src/test/results/clientpositive/multi_join_union.q.out be6e172 > ql/src/test/results/clientpositive/reduce_deduplicate_exclude_join.q.out > fdde8df > ql/src/test/results/clientpositive/select_transform_hint.q.out d9ca1fc > ql/src/test/results/clientpositive/skewjoin.q.out 2e03b7e > ql/src/test/results/clientpositive/skewjoin_noskew.q.out fe8da1e > ql/src/test/results/clientpositive/smb_mapjoin_13.q.out b7dc7f2 > ql/src/test/results/clientpositive/smb_mapjoin_15.q.out 9adb0d0 > ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out 2011bc3 > ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out 3f2f6ab > ql/src/test/results/clientpositive/stats11.q.out 44ddc03 > ql/src/test/results/clientpositive/subq_where_serialization.q.out 94840c4 > ql/src/test/results/clientpositive/subquery_in_having.q.out b293ed8 > > Diff: https://reviews.apache.org/r/16728/diff/ > > > Testing > ------- > > > Thanks, > > Navis Ryu > >
