----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/16728/ -----------------------------------------------------------
(Updated Jan. 28, 2014, 1:37 a.m.) Review request for hive. Bugs: HIVE-6144 https://issues.apache.org/jira/browse/HIVE-6144 Repository: hive-git Description ------- For map join, all data in small aliases are hashed and stored into temporary file in MapRedLocalTask. But for some aliases without filter or projection, it seemed not necessary to do that. For example. {noformat} select a.* from src a join src b on a.key=b.key; {noformat} makes plan like this. {noformat} STAGE PLANS: Stage: Stage-4 Map Reduce Local Work Alias -> Map Local Tables: a Fetch Operator limit: -1 Alias -> Map Local Operator Tree: a TableScan alias: a HashTable Sink Operator condition expressions: 0 {key} {value} 1 handleSkewJoin: false keys: 0 [Column[key]] 1 [Column[key]] Position of Big Table: 1 Stage: Stage-3 Map Reduce Alias -> Map Operator Tree: b TableScan alias: b Map Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 {key} {value} 1 handleSkewJoin: false keys: 0 [Column[key]] 1 [Column[key]] outputColumnNames: _col0, _col1 Position of Big Table: 1 Select Operator File Output Operator Local Work: Map Reduce Local Work Stage: Stage-0 Fetch Operator {noformat} table src(a) is fetched and stored as-is in MRLocalTask. With this patch, plan can be like below. {noformat} Stage: Stage-3 Map Reduce Alias -> Map Operator Tree: b TableScan alias: b Map Join Operator condition map: Inner Join 0 to 1 condition expressions: 0 {key} {value} 1 handleSkewJoin: false keys: 0 [Column[key]] 1 [Column[key]] outputColumnNames: _col0, _col1 Position of Big Table: 1 Select Operator File Output Operator Local Work: Map Reduce Local Work Alias -> Map Local Tables: a Fetch Operator limit: -1 Alias -> Map Local Operator Tree: a TableScan alias: a Has Any Stage Alias: false Stage: Stage-0 Fetch Operator {noformat} Diffs (updated) ----- common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84ee78f conf/hive-default.xml.template 66d22f9 itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestUtil.java 9ad5986 itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/MapJoinCounterHook.java 1b0d57e ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java d8f4eb4 ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java a080fcc ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java fc08b28 ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 1e0314d ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java bdc85b9 ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 56676df ql/src/java/org/apache/hadoop/hive/ql/exec/TemporaryHashSinkOperator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 22e5777 ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java 58484af ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 2d2508d ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 2df8ab9 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java 5a53e15 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java 83b8d6e ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java ebccb14 ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java c30da56 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 709c50e ql/src/test/queries/clientpositive/auto_join_without_localtask.q PRE-CREATION ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 0595cd6 ql/src/test/results/clientnegative/deletejar.q.out b873e34 ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out fa261b3 ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out bca069a ql/src/test/results/clientpositive/auto_join1.q.out b93c10f ql/src/test/results/clientpositive/auto_join15.q.out 6bfcfc7 ql/src/test/results/clientpositive/auto_join17.q.out 698270e ql/src/test/results/clientpositive/auto_join19.q.out 247b081 ql/src/test/results/clientpositive/auto_join2.q.out 756cfa1 ql/src/test/results/clientpositive/auto_join21.q.out 053475f ql/src/test/results/clientpositive/auto_join22.q.out 6e5ae4c ql/src/test/results/clientpositive/auto_join24.q.out dbecb2c ql/src/test/results/clientpositive/auto_join25.q.out d6d0c90 ql/src/test/results/clientpositive/auto_join26.q.out 7ca6bb9 ql/src/test/results/clientpositive/auto_join28.q.out 488e15c ql/src/test/results/clientpositive/auto_join29.q.out 7798a0e ql/src/test/results/clientpositive/auto_join3.q.out 4279e8c ql/src/test/results/clientpositive/auto_join30.q.out a46ea2a ql/src/test/results/clientpositive/auto_join31.q.out 4504426 ql/src/test/results/clientpositive/auto_join32.q.out da4ef55 ql/src/test/results/clientpositive/auto_join9.q.out d2589ed ql/src/test/results/clientpositive/auto_join_without_localtask.q.out PRE-CREATION ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 8e11616 ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 95e86e1 ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out fd8e9cc ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out 3c20e0f ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out 6189df6 ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c00b519 ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 42bab30 ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out ec33acd ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out f8c164a ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 6ee49a7 ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 3f2672e ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 4cfd022 ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 7b9f811 ql/src/test/results/clientpositive/bucket_map_join_1.q.out f052025 ql/src/test/results/clientpositive/bucket_map_join_2.q.out eb51e23 ql/src/test/results/clientpositive/bucketcontext_1.q.out 9bdc1ee ql/src/test/results/clientpositive/bucketcontext_2.q.out d5d67dc ql/src/test/results/clientpositive/bucketcontext_3.q.out b226246 ql/src/test/results/clientpositive/bucketcontext_4.q.out c27d927 ql/src/test/results/clientpositive/bucketcontext_5.q.out 819362f ql/src/test/results/clientpositive/bucketcontext_6.q.out d7b1ad7 ql/src/test/results/clientpositive/bucketcontext_7.q.out 80749f0 ql/src/test/results/clientpositive/bucketcontext_8.q.out b8a4d4f ql/src/test/results/clientpositive/bucketmapjoin1.q.out 8d24d6f ql/src/test/results/clientpositive/bucketmapjoin10.q.out 371af7d ql/src/test/results/clientpositive/bucketmapjoin11.q.out fcfa41f ql/src/test/results/clientpositive/bucketmapjoin12.q.out aa50cfe ql/src/test/results/clientpositive/bucketmapjoin13.q.out eceeade ql/src/test/results/clientpositive/bucketmapjoin2.q.out 1dfe3e7 ql/src/test/results/clientpositive/bucketmapjoin3.q.out ba1962d ql/src/test/results/clientpositive/bucketmapjoin4.q.out a9c4bce ql/src/test/results/clientpositive/bucketmapjoin5.q.out 4812a21 ql/src/test/results/clientpositive/bucketmapjoin7.q.out f8ea2e2 ql/src/test/results/clientpositive/bucketmapjoin8.q.out 8e2be73 ql/src/test/results/clientpositive/bucketmapjoin9.q.out 38b9c87 ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 0139546 ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out bf00114 ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out 2cc1a0f ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out 0014891 ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out 14d0b6a ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out e83ef59 ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out b5828f0 ql/src/test/results/clientpositive/correlationoptimizer1.q.out 3ef5cf2 ql/src/test/results/clientpositive/correlationoptimizer3.q.out e633287 ql/src/test/results/clientpositive/correlationoptimizer4.q.out c037d3c ql/src/test/results/clientpositive/correlationoptimizer5.q.out 02ba37a ql/src/test/results/clientpositive/correlationoptimizer6.q.out 557ee9e ql/src/test/results/clientpositive/correlationoptimizer7.q.out b1fe60a ql/src/test/results/clientpositive/explain_rearrange.q.out 47b1afb ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out effa6c8 ql/src/test/results/clientpositive/infer_bucket_sort_dyn_part.q.out 5a5c788 ql/src/test/results/clientpositive/join25.q.out 630d862 ql/src/test/results/clientpositive/join26.q.out 9fca2c3 ql/src/test/results/clientpositive/join27.q.out 0ad8bff ql/src/test/results/clientpositive/join28.q.out f50072c ql/src/test/results/clientpositive/join29.q.out 0e0bbb4 ql/src/test/results/clientpositive/join30.q.out 924579c ql/src/test/results/clientpositive/join31.q.out 85b0196 ql/src/test/results/clientpositive/join32.q.out 9e84d84 ql/src/test/results/clientpositive/join32_lessSize.q.out 0f6f8ec ql/src/test/results/clientpositive/join33.q.out 9e84d84 ql/src/test/results/clientpositive/join34.q.out 1e2aa91 ql/src/test/results/clientpositive/join35.q.out 83f5ba2 ql/src/test/results/clientpositive/join36.q.out 922ddef ql/src/test/results/clientpositive/join37.q.out db905bd ql/src/test/results/clientpositive/join_map_ppr.q.out 131e65c ql/src/test/results/clientpositive/join_reorder4.q.out b14ea08 ql/src/test/results/clientpositive/join_star.q.out 9b1adbf ql/src/test/results/clientpositive/mapjoin1.q.out e8b4fa8 ql/src/test/results/clientpositive/mapjoin_distinct.q.out c4dc9d2 ql/src/test/results/clientpositive/mapjoin_hook.q.out 8da5168 ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 2c9f7f6 ql/src/test/results/clientpositive/mapjoin_subquery.q.out e90a50a ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 220bf322 ql/src/test/results/clientpositive/mapjoin_test_outer.q.out d66e079 ql/src/test/results/clientpositive/multiMapJoin1.q.out 624b47f ql/src/test/results/clientpositive/multiMapJoin2.q.out 6d0c7d2 ql/src/test/results/clientpositive/multi_join_union.q.out be6e172 ql/src/test/results/clientpositive/reduce_deduplicate_exclude_join.q.out fdde8df ql/src/test/results/clientpositive/select_transform_hint.q.out d9ca1fc ql/src/test/results/clientpositive/skewjoin.q.out 2e03b7e ql/src/test/results/clientpositive/skewjoin_noskew.q.out fe8da1e ql/src/test/results/clientpositive/smb_mapjoin_13.q.out b7dc7f2 ql/src/test/results/clientpositive/smb_mapjoin_15.q.out 9adb0d0 ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out 2011bc3 ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out 3f2f6ab ql/src/test/results/clientpositive/stats11.q.out 44ddc03 ql/src/test/results/clientpositive/subq_where_serialization.q.out 94840c4 ql/src/test/results/clientpositive/subquery_in_having.q.out b293ed8 Diff: https://reviews.apache.org/r/16728/diff/ Testing ------- Thanks, Navis Ryu