-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30388/
-----------------------------------------------------------
(Updated Jan. 29, 2015, 6:51 p.m.)
Review request for hive and Xuefu Zhang.
Changes
-------
Regenerated golden files (mostly plan change due to the backup task), and added
auto_join25.q. Also addressed initial feedback from review board.
Bugs: HIVE-9103
https://issues.apache.org/jira/browse/HIVE-9103
Repository: hive-git
Description
-------
This patch adds backup task to map join task. The backup task, which uses
common join, will be triggered
in case the mapjoin task failed.
Note that, no matter how many map joins there are in the SparkTask, we will
only generate one backup task.
This means that if the original task failed at the very last map join, the
whole task will be re-executed.
The handling of backup task is a little bit different from what MR does, mostly
because we convert JOIN to
MAPJOIN during the operator plan optimization phase, at which time no task/work
exist yet. In the patch, we
cloned the whole operator tree before the JOIN operator is converted. The
operator tree will be processed
and generate a separate work tree for a separate backup SparkTask.
Diffs (updated)
-----
itests/src/test/resources/testconfiguration.properties f583aaf
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
69004dc
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/StageIDsRearranger.java
79c3e02
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkJoinOptimizer.java
d57ceff
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
9ff47c7
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java
6e0ac38
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java b838bff
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java
773cfbd
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/OptimizeSparkProcContext.java
f7586a4
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 3a7477a
ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 0e85990
ql/src/test/results/clientpositive/spark/auto_join0.q.out 7f8eb63
ql/src/test/results/clientpositive/spark/auto_join1.q.out b640b9d
ql/src/test/results/clientpositive/spark/auto_join10.q.out f01765c
ql/src/test/results/clientpositive/spark/auto_join11.q.out 69c10e6
ql/src/test/results/clientpositive/spark/auto_join12.q.out bc763ed
ql/src/test/results/clientpositive/spark/auto_join13.q.out 935ebf5
ql/src/test/results/clientpositive/spark/auto_join14.q.out 830314e
ql/src/test/results/clientpositive/spark/auto_join15.q.out 780540b
ql/src/test/results/clientpositive/spark/auto_join16.q.out f705339
ql/src/test/results/clientpositive/spark/auto_join17.q.out 3144db6
ql/src/test/results/clientpositive/spark/auto_join19.q.out f2b0140
ql/src/test/results/clientpositive/spark/auto_join2.q.out 2424cca
ql/src/test/results/clientpositive/spark/auto_join20.q.out 9258f3b
ql/src/test/results/clientpositive/spark/auto_join21.q.out aa8f6dd
ql/src/test/results/clientpositive/spark/auto_join22.q.out d49dda9
ql/src/test/results/clientpositive/spark/auto_join23.q.out a179d87
ql/src/test/results/clientpositive/spark/auto_join24.q.out cfb076e
ql/src/test/results/clientpositive/spark/auto_join25.q.out ab01b8a
ql/src/test/results/clientpositive/spark/auto_join26.q.out 58821e9
ql/src/test/results/clientpositive/spark/auto_join28.q.out d30133b
ql/src/test/results/clientpositive/spark/auto_join29.q.out 780c6cb
ql/src/test/results/clientpositive/spark/auto_join3.q.out 54e24f3
ql/src/test/results/clientpositive/spark/auto_join30.q.out 4c832e2
ql/src/test/results/clientpositive/spark/auto_join31.q.out 5980814
ql/src/test/results/clientpositive/spark/auto_join32.q.out 9629f53
ql/src/test/results/clientpositive/spark/auto_join4.q.out 3366f75
ql/src/test/results/clientpositive/spark/auto_join5.q.out b6d8798
ql/src/test/results/clientpositive/spark/auto_join8.q.out 5b6cc80
ql/src/test/results/clientpositive/spark/auto_join9.q.out 6daf348
ql/src/test/results/clientpositive/spark/auto_join_filters.q.out 8934433
ql/src/test/results/clientpositive/spark/auto_join_nulls.q.out 1f37c75
ql/src/test/results/clientpositive/spark/auto_join_stats.q.out 1fa1a74
ql/src/test/results/clientpositive/spark/auto_join_stats2.q.out c6473d3
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 3d465db
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_10.q.out fe7b96d
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out f4e889a
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out c358721
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out 7eadcd0
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_14.q.out 984db20
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_15.q.out 2acc323
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out f05b0cc
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out c7d23f8
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out f5dc2f7
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 26e7957
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out a5c0562
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out ef13a40
ql/src/test/results/clientpositive/spark/auto_sortmerge_join_9.q.out a2b98fc
ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out 6230bef
ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out 1a33625
ql/src/test/results/clientpositive/spark/bucket_map_join_spark3.q.out fed923c
ql/src/test/results/clientpositive/spark/bucket_map_join_spark4.q.out 8b5e8d4
ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 1c81d1b
ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out 04a934f
ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_2.q.out
365306e
ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_4.q.out
3846de7
ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_6.q.out
5b559c4
ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_7.q.out
cefc6aa
ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_8.q.out
ca44d7c
ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out dda6c38
ql/src/test/results/clientpositive/spark/identity_project_remove_skip.q.out
7238009
ql/src/test/results/clientpositive/spark/infer_bucket_sort_convert_join.q.out
3d4eb18
ql/src/test/results/clientpositive/spark/join28.q.out f23f662
ql/src/test/results/clientpositive/spark/join29.q.out 0b4284c
ql/src/test/results/clientpositive/spark/join31.q.out a52a8b6
ql/src/test/results/clientpositive/spark/join32.q.out a9d50b4
ql/src/test/results/clientpositive/spark/join32_lessSize.q.out dac9610
ql/src/test/results/clientpositive/spark/join33.q.out a9d50b4
ql/src/test/results/clientpositive/spark/join_reorder4.q.out 5cc30f7
ql/src/test/results/clientpositive/spark/join_star.q.out 69c2fd7
ql/src/test/results/clientpositive/spark/mapjoin_decimal.q.out b681e5f
ql/src/test/results/clientpositive/spark/mapjoin_filter_on_outerjoin.q.out
0271f97
ql/src/test/results/clientpositive/spark/mapjoin_hook.q.out 7aa8ce9
ql/src/test/results/clientpositive/spark/mapjoin_mapjoin.q.out 65a7d06
ql/src/test/results/clientpositive/spark/mapjoin_memcheck.q.out 14f316c
ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out 2d1e7a7
ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out a757d0b
ql/src/test/results/clientpositive/spark/mapjoin_test_outer.q.out 7143348
ql/src/test/results/clientpositive/spark/multi_join_union.q.out bda569d
ql/src/test/results/clientpositive/spark/parquet_join.q.out 390aeb1
ql/src/test/results/clientpositive/spark/reduce_deduplicate_exclude_join.q.out
19ab4c8
ql/src/test/results/clientpositive/spark/smb_mapjoin_17.q.out bd3a6a1
ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out cb811ed
ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.java1.7.out
92a8595
ql/src/test/results/clientpositive/spark/vector_decimal_mapjoin.q.out 5ec95c2
ql/src/test/results/clientpositive/spark/vector_left_outer_join.q.out ca8918a
ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 02c1fc6
ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out 237df98
ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out
f8e8ba7
Diff: https://reviews.apache.org/r/30388/diff/
Testing
-------
auto_join25.q
Thanks,
Chao Sun
