-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30388/
-----------------------------------------------------------

(Updated Jan. 29, 2015, 6:51 p.m.)


Review request for hive and Xuefu Zhang.


Changes
-------

Regenerated golden files (mostly plan change due to the backup task), and added 
auto_join25.q. Also addressed initial feedback from review board.


Bugs: HIVE-9103
    https://issues.apache.org/jira/browse/HIVE-9103


Repository: hive-git


Description
-------

This patch adds backup task to map join task. The backup task, which uses 
common join, will be triggered
in case the mapjoin task failed.

Note that, no matter how many map joins there are in the SparkTask, we will 
only generate one backup task.
This means that if the original task failed at the very last map join, the 
whole task will be re-executed.

The handling of backup task is a little bit different from what MR does, mostly 
because we convert JOIN to
MAPJOIN during the operator plan optimization phase, at which time no task/work 
exist yet. In the patch, we
cloned the whole operator tree before the JOIN operator is converted. The 
operator tree will be processed
and generate a separate work tree for a separate backup SparkTask.


Diffs (updated)
-----

  itests/src/test/resources/testconfiguration.properties f583aaf 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SparkMapJoinResolver.java
 69004dc 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/StageIDsRearranger.java
 79c3e02 
  ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkJoinOptimizer.java 
d57ceff 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkMapJoinOptimizer.java
 9ff47c7 
  
ql/src/java/org/apache/hadoop/hive/ql/optimizer/spark/SparkSortMergeJoinFactory.java
 6e0ac38 
  ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java b838bff 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java 
773cfbd 
  
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/OptimizeSparkProcContext.java 
f7586a4 
  ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 3a7477a 
  ql/src/java/org/apache/hadoop/hive/ql/plan/TableScanDesc.java 0e85990 
  ql/src/test/results/clientpositive/spark/auto_join0.q.out 7f8eb63 
  ql/src/test/results/clientpositive/spark/auto_join1.q.out b640b9d 
  ql/src/test/results/clientpositive/spark/auto_join10.q.out f01765c 
  ql/src/test/results/clientpositive/spark/auto_join11.q.out 69c10e6 
  ql/src/test/results/clientpositive/spark/auto_join12.q.out bc763ed 
  ql/src/test/results/clientpositive/spark/auto_join13.q.out 935ebf5 
  ql/src/test/results/clientpositive/spark/auto_join14.q.out 830314e 
  ql/src/test/results/clientpositive/spark/auto_join15.q.out 780540b 
  ql/src/test/results/clientpositive/spark/auto_join16.q.out f705339 
  ql/src/test/results/clientpositive/spark/auto_join17.q.out 3144db6 
  ql/src/test/results/clientpositive/spark/auto_join19.q.out f2b0140 
  ql/src/test/results/clientpositive/spark/auto_join2.q.out 2424cca 
  ql/src/test/results/clientpositive/spark/auto_join20.q.out 9258f3b 
  ql/src/test/results/clientpositive/spark/auto_join21.q.out aa8f6dd 
  ql/src/test/results/clientpositive/spark/auto_join22.q.out d49dda9 
  ql/src/test/results/clientpositive/spark/auto_join23.q.out a179d87 
  ql/src/test/results/clientpositive/spark/auto_join24.q.out cfb076e 
  ql/src/test/results/clientpositive/spark/auto_join25.q.out ab01b8a 
  ql/src/test/results/clientpositive/spark/auto_join26.q.out 58821e9 
  ql/src/test/results/clientpositive/spark/auto_join28.q.out d30133b 
  ql/src/test/results/clientpositive/spark/auto_join29.q.out 780c6cb 
  ql/src/test/results/clientpositive/spark/auto_join3.q.out 54e24f3 
  ql/src/test/results/clientpositive/spark/auto_join30.q.out 4c832e2 
  ql/src/test/results/clientpositive/spark/auto_join31.q.out 5980814 
  ql/src/test/results/clientpositive/spark/auto_join32.q.out 9629f53 
  ql/src/test/results/clientpositive/spark/auto_join4.q.out 3366f75 
  ql/src/test/results/clientpositive/spark/auto_join5.q.out b6d8798 
  ql/src/test/results/clientpositive/spark/auto_join8.q.out 5b6cc80 
  ql/src/test/results/clientpositive/spark/auto_join9.q.out 6daf348 
  ql/src/test/results/clientpositive/spark/auto_join_filters.q.out 8934433 
  ql/src/test/results/clientpositive/spark/auto_join_nulls.q.out 1f37c75 
  ql/src/test/results/clientpositive/spark/auto_join_stats.q.out 1fa1a74 
  ql/src/test/results/clientpositive/spark/auto_join_stats2.q.out c6473d3 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_1.q.out 3d465db 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_10.q.out fe7b96d 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_11.q.out f4e889a 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_12.q.out c358721 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_13.q.out 7eadcd0 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_14.q.out 984db20 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_15.q.out 2acc323 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_2.q.out f05b0cc 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_3.q.out c7d23f8 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_4.q.out f5dc2f7 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_5.q.out 26e7957 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_7.q.out a5c0562 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_8.q.out ef13a40 
  ql/src/test/results/clientpositive/spark/auto_sortmerge_join_9.q.out a2b98fc 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark1.q.out 6230bef 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark2.q.out 1a33625 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark3.q.out fed923c 
  ql/src/test/results/clientpositive/spark/bucket_map_join_spark4.q.out 8b5e8d4 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez1.q.out 1c81d1b 
  ql/src/test/results/clientpositive/spark/bucket_map_join_tez2.q.out 04a934f 
  ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_2.q.out 
365306e 
  ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_4.q.out 
3846de7 
  ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_6.q.out 
5b559c4 
  ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_7.q.out 
cefc6aa 
  ql/src/test/results/clientpositive/spark/bucketsortoptimize_insert_8.q.out 
ca44d7c 
  ql/src/test/results/clientpositive/spark/cross_product_check_2.q.out dda6c38 
  ql/src/test/results/clientpositive/spark/identity_project_remove_skip.q.out 
7238009 
  ql/src/test/results/clientpositive/spark/infer_bucket_sort_convert_join.q.out 
3d4eb18 
  ql/src/test/results/clientpositive/spark/join28.q.out f23f662 
  ql/src/test/results/clientpositive/spark/join29.q.out 0b4284c 
  ql/src/test/results/clientpositive/spark/join31.q.out a52a8b6 
  ql/src/test/results/clientpositive/spark/join32.q.out a9d50b4 
  ql/src/test/results/clientpositive/spark/join32_lessSize.q.out dac9610 
  ql/src/test/results/clientpositive/spark/join33.q.out a9d50b4 
  ql/src/test/results/clientpositive/spark/join_reorder4.q.out 5cc30f7 
  ql/src/test/results/clientpositive/spark/join_star.q.out 69c2fd7 
  ql/src/test/results/clientpositive/spark/mapjoin_decimal.q.out b681e5f 
  ql/src/test/results/clientpositive/spark/mapjoin_filter_on_outerjoin.q.out 
0271f97 
  ql/src/test/results/clientpositive/spark/mapjoin_hook.q.out 7aa8ce9 
  ql/src/test/results/clientpositive/spark/mapjoin_mapjoin.q.out 65a7d06 
  ql/src/test/results/clientpositive/spark/mapjoin_memcheck.q.out 14f316c 
  ql/src/test/results/clientpositive/spark/mapjoin_subquery.q.out 2d1e7a7 
  ql/src/test/results/clientpositive/spark/mapjoin_subquery2.q.out a757d0b 
  ql/src/test/results/clientpositive/spark/mapjoin_test_outer.q.out 7143348 
  ql/src/test/results/clientpositive/spark/multi_join_union.q.out bda569d 
  ql/src/test/results/clientpositive/spark/parquet_join.q.out 390aeb1 
  
ql/src/test/results/clientpositive/spark/reduce_deduplicate_exclude_join.q.out 
19ab4c8 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_17.q.out bd3a6a1 
  ql/src/test/results/clientpositive/spark/smb_mapjoin_25.q.out cb811ed 
  ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.java1.7.out 
92a8595 
  ql/src/test/results/clientpositive/spark/vector_decimal_mapjoin.q.out 5ec95c2 
  ql/src/test/results/clientpositive/spark/vector_left_outer_join.q.out ca8918a 
  ql/src/test/results/clientpositive/spark/vector_mapjoin_reduce.q.out 02c1fc6 
  ql/src/test/results/clientpositive/spark/vectorized_mapjoin.q.out 237df98 
  ql/src/test/results/clientpositive/spark/vectorized_nested_mapjoin.q.out 
f8e8ba7 

Diff: https://reviews.apache.org/r/30388/diff/


Testing
-------

auto_join25.q


Thanks,

Chao Sun

Reply via email to