-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/16728/
-----------------------------------------------------------
(Updated Jan. 28, 2014, 1:38 a.m.)
Review request for hive.
Bugs: HIVE-6144
https://issues.apache.org/jira/browse/HIVE-6144
Repository: hive-git
Description
-------
For map join, all data in small aliases are hashed and stored into temporary
file in MapRedLocalTask. But for some aliases without filter or projection, it
seemed not necessary to do that. For example.
{noformat}
select a.* from src a join src b on a.key=b.key;
{noformat}
makes plan like this.
{noformat}
STAGE PLANS:
Stage: Stage-4
Map Reduce Local Work
Alias -> Map Local Tables:
a
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
a
TableScan
alias: a
HashTable Sink Operator
condition expressions:
0 {key} {value}
1
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
Position of Big Table: 1
Stage: Stage-3
Map Reduce
Alias -> Map Operator Tree:
b
TableScan
alias: b
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {key} {value}
1
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
outputColumnNames: _col0, _col1
Position of Big Table: 1
Select Operator
File Output Operator
Local Work:
Map Reduce Local Work
Stage: Stage-0
Fetch Operator
{noformat}
table src(a) is fetched and stored as-is in MRLocalTask. With this patch, plan
can be like below.
{noformat}
Stage: Stage-3
Map Reduce
Alias -> Map Operator Tree:
b
TableScan
alias: b
Map Join Operator
condition map:
Inner Join 0 to 1
condition expressions:
0 {key} {value}
1
handleSkewJoin: false
keys:
0 [Column[key]]
1 [Column[key]]
outputColumnNames: _col0, _col1
Position of Big Table: 1
Select Operator
File Output Operator
Local Work:
Map Reduce Local Work
Alias -> Map Local Tables:
a
Fetch Operator
limit: -1
Alias -> Map Local Operator Tree:
a
TableScan
alias: a
Has Any Stage Alias: false
Stage: Stage-0
Fetch Operator
{noformat}
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 84ee78f
conf/hive-default.xml.template 66d22f9
itests/util/src/main/java/org/apache/hadoop/hive/ql/hooks/MapJoinCounterHook.java
1b0d57e
ql/src/java/org/apache/hadoop/hive/ql/exec/AbstractMapJoinOperator.java
d8f4eb4
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableLoader.java a080fcc
ql/src/java/org/apache/hadoop/hive/ql/exec/HashTableSinkOperator.java fc08b28
ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java 1e0314d
ql/src/java/org/apache/hadoop/hive/ql/exec/MapJoinOperator.java bdc85b9
ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 56676df
ql/src/java/org/apache/hadoop/hive/ql/exec/TemporaryHashSinkOperator.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java 22e5777
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/HashTableLoader.java 58484af
ql/src/java/org/apache/hadoop/hive/ql/exec/mr/MapredLocalTask.java 2d2508d
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/HashTableLoader.java 2df8ab9
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/LocalMapJoinProcFactory.java
5a53e15
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java
83b8d6e
ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverCommonJoin.java
ebccb14
ql/src/java/org/apache/hadoop/hive/ql/plan/HashTableSinkDesc.java c30da56
ql/src/java/org/apache/hadoop/hive/ql/plan/MapredLocalWork.java 709c50e
ql/src/test/queries/clientpositive/auto_join_without_localtask.q PRE-CREATION
ql/src/test/results/clientnegative/bucket_mapjoin_mismatch1.q.out 0595cd6
ql/src/test/results/clientnegative/file_with_header_footer_negative.q.out
fa261b3
ql/src/test/results/clientnegative/sortmerge_mapjoin_mismatch_1.q.out bca069a
ql/src/test/results/clientpositive/auto_join1.q.out b93c10f
ql/src/test/results/clientpositive/auto_join15.q.out 6bfcfc7
ql/src/test/results/clientpositive/auto_join17.q.out 698270e
ql/src/test/results/clientpositive/auto_join19.q.out 247b081
ql/src/test/results/clientpositive/auto_join2.q.out 756cfa1
ql/src/test/results/clientpositive/auto_join21.q.out 053475f
ql/src/test/results/clientpositive/auto_join22.q.out 6e5ae4c
ql/src/test/results/clientpositive/auto_join24.q.out dbecb2c
ql/src/test/results/clientpositive/auto_join25.q.out d6d0c90
ql/src/test/results/clientpositive/auto_join26.q.out 7ca6bb9
ql/src/test/results/clientpositive/auto_join28.q.out 488e15c
ql/src/test/results/clientpositive/auto_join29.q.out 7798a0e
ql/src/test/results/clientpositive/auto_join3.q.out 4279e8c
ql/src/test/results/clientpositive/auto_join30.q.out a46ea2a
ql/src/test/results/clientpositive/auto_join31.q.out 4504426
ql/src/test/results/clientpositive/auto_join32.q.out da4ef55
ql/src/test/results/clientpositive/auto_join9.q.out d2589ed
ql/src/test/results/clientpositive/auto_join_without_localtask.q.out
PRE-CREATION
ql/src/test/results/clientpositive/auto_sortmerge_join_1.q.out 8e11616
ql/src/test/results/clientpositive/auto_sortmerge_join_11.q.out 95e86e1
ql/src/test/results/clientpositive/auto_sortmerge_join_12.q.out fd8e9cc
ql/src/test/results/clientpositive/auto_sortmerge_join_14.q.out 3c20e0f
ql/src/test/results/clientpositive/auto_sortmerge_join_15.q.out 6189df6
ql/src/test/results/clientpositive/auto_sortmerge_join_2.q.out c00b519
ql/src/test/results/clientpositive/auto_sortmerge_join_3.q.out 42bab30
ql/src/test/results/clientpositive/auto_sortmerge_join_4.q.out ec33acd
ql/src/test/results/clientpositive/auto_sortmerge_join_5.q.out f8c164a
ql/src/test/results/clientpositive/auto_sortmerge_join_6.q.out 6ee49a7
ql/src/test/results/clientpositive/auto_sortmerge_join_7.q.out 3f2672e
ql/src/test/results/clientpositive/auto_sortmerge_join_8.q.out 4cfd022
ql/src/test/results/clientpositive/auto_sortmerge_join_9.q.out 7b9f811
ql/src/test/results/clientpositive/bucket_map_join_1.q.out f052025
ql/src/test/results/clientpositive/bucket_map_join_2.q.out eb51e23
ql/src/test/results/clientpositive/bucketcontext_1.q.out 9bdc1ee
ql/src/test/results/clientpositive/bucketcontext_2.q.out d5d67dc
ql/src/test/results/clientpositive/bucketcontext_3.q.out b226246
ql/src/test/results/clientpositive/bucketcontext_4.q.out c27d927
ql/src/test/results/clientpositive/bucketcontext_5.q.out 819362f
ql/src/test/results/clientpositive/bucketcontext_6.q.out d7b1ad7
ql/src/test/results/clientpositive/bucketcontext_7.q.out 80749f0
ql/src/test/results/clientpositive/bucketcontext_8.q.out b8a4d4f
ql/src/test/results/clientpositive/bucketmapjoin1.q.out 8d24d6f
ql/src/test/results/clientpositive/bucketmapjoin10.q.out 371af7d
ql/src/test/results/clientpositive/bucketmapjoin11.q.out fcfa41f
ql/src/test/results/clientpositive/bucketmapjoin12.q.out aa50cfe
ql/src/test/results/clientpositive/bucketmapjoin13.q.out eceeade
ql/src/test/results/clientpositive/bucketmapjoin2.q.out 1dfe3e7
ql/src/test/results/clientpositive/bucketmapjoin3.q.out ba1962d
ql/src/test/results/clientpositive/bucketmapjoin4.q.out a9c4bce
ql/src/test/results/clientpositive/bucketmapjoin5.q.out 4812a21
ql/src/test/results/clientpositive/bucketmapjoin7.q.out f8ea2e2
ql/src/test/results/clientpositive/bucketmapjoin8.q.out 8e2be73
ql/src/test/results/clientpositive/bucketmapjoin9.q.out 38b9c87
ql/src/test/results/clientpositive/bucketmapjoin_negative.q.out 0139546
ql/src/test/results/clientpositive/bucketmapjoin_negative2.q.out bf00114
ql/src/test/results/clientpositive/bucketmapjoin_negative3.q.out 2cc1a0f
ql/src/test/results/clientpositive/bucketsortoptimize_insert_2.q.out 0014891
ql/src/test/results/clientpositive/bucketsortoptimize_insert_4.q.out 14d0b6a
ql/src/test/results/clientpositive/bucketsortoptimize_insert_5.q.out e83ef59
ql/src/test/results/clientpositive/bucketsortoptimize_insert_6.q.out b5828f0
ql/src/test/results/clientpositive/correlationoptimizer1.q.out 3ef5cf2
ql/src/test/results/clientpositive/correlationoptimizer3.q.out e633287
ql/src/test/results/clientpositive/correlationoptimizer4.q.out c037d3c
ql/src/test/results/clientpositive/correlationoptimizer5.q.out 02ba37a
ql/src/test/results/clientpositive/correlationoptimizer6.q.out 557ee9e
ql/src/test/results/clientpositive/correlationoptimizer7.q.out b1fe60a
ql/src/test/results/clientpositive/explain_rearrange.q.out 47b1afb
ql/src/test/results/clientpositive/infer_bucket_sort_convert_join.q.out
effa6c8
ql/src/test/results/clientpositive/join25.q.out 630d862
ql/src/test/results/clientpositive/join26.q.out 9fca2c3
ql/src/test/results/clientpositive/join27.q.out 0ad8bff
ql/src/test/results/clientpositive/join28.q.out f50072c
ql/src/test/results/clientpositive/join29.q.out 0e0bbb4
ql/src/test/results/clientpositive/join30.q.out 924579c
ql/src/test/results/clientpositive/join31.q.out 85b0196
ql/src/test/results/clientpositive/join32.q.out 9e84d84
ql/src/test/results/clientpositive/join32_lessSize.q.out 0f6f8ec
ql/src/test/results/clientpositive/join33.q.out 9e84d84
ql/src/test/results/clientpositive/join34.q.out 1e2aa91
ql/src/test/results/clientpositive/join35.q.out 83f5ba2
ql/src/test/results/clientpositive/join36.q.out 922ddef
ql/src/test/results/clientpositive/join37.q.out db905bd
ql/src/test/results/clientpositive/join_map_ppr.q.out 131e65c
ql/src/test/results/clientpositive/join_reorder4.q.out b14ea08
ql/src/test/results/clientpositive/join_star.q.out 9b1adbf
ql/src/test/results/clientpositive/mapjoin1.q.out e8b4fa8
ql/src/test/results/clientpositive/mapjoin_distinct.q.out c4dc9d2
ql/src/test/results/clientpositive/mapjoin_hook.q.out 8da5168
ql/src/test/results/clientpositive/mapjoin_mapjoin.q.out 2c9f7f6
ql/src/test/results/clientpositive/mapjoin_subquery.q.out e90a50a
ql/src/test/results/clientpositive/mapjoin_subquery2.q.out 220bf322
ql/src/test/results/clientpositive/mapjoin_test_outer.q.out d66e079
ql/src/test/results/clientpositive/multiMapJoin1.q.out 624b47f
ql/src/test/results/clientpositive/multiMapJoin2.q.out 6d0c7d2
ql/src/test/results/clientpositive/multi_join_union.q.out be6e172
ql/src/test/results/clientpositive/reduce_deduplicate_exclude_join.q.out
fdde8df
ql/src/test/results/clientpositive/select_transform_hint.q.out d9ca1fc
ql/src/test/results/clientpositive/skewjoin.q.out 2e03b7e
ql/src/test/results/clientpositive/skewjoin_noskew.q.out fe8da1e
ql/src/test/results/clientpositive/smb_mapjoin_13.q.out b7dc7f2
ql/src/test/results/clientpositive/smb_mapjoin_15.q.out 9adb0d0
ql/src/test/results/clientpositive/sort_merge_join_desc_6.q.out 2011bc3
ql/src/test/results/clientpositive/sort_merge_join_desc_7.q.out 3f2f6ab
ql/src/test/results/clientpositive/stats11.q.out 44ddc03
ql/src/test/results/clientpositive/subq_where_serialization.q.out 94840c4
ql/src/test/results/clientpositive/subquery_in_having.q.out b293ed8
Diff: https://reviews.apache.org/r/16728/diff/
Testing
-------
Thanks,
Navis Ryu
