-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26706/
-----------------------------------------------------------
(Updated Oct. 17, 2014, 6:04 p.m.)
Review request for hive and Xuefu Zhang.
Changes
-------
Added a test to check that splitting work doesn't create duplicate FSs.
Bugs: HIVE-8436
https://issues.apache.org/jira/browse/HIVE-8436
Repository: hive-git
Description
-------
Based on the design doc, we need to split the operator tree of a work in
SparkWork if the work is connected to multiple child works. The way splitting
the operator tree is performed by cloning the original work and removing
unwanted branches in the operator tree. Please refer to the design doc for
details.
This process should be done right before we generate SparkPlan. We should have
a utility method that takes the orignal SparkWork and return a modified
SparkWork.
This process should also keep the information about the original work and its
clones. Such information will be needed during SparkPlan generation (HIVE-8437).
Diffs (updated)
-----
ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 7d9feac
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/HiveReduceFunction.java
5153885
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 3fd37a0
ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java
126cb9f
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkProcContext.java
d7744e9
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkUtils.java 280edde
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/GenSparkWork.java ac94ea0
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkCompiler.java 644c681
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMergeTaskProcessor.java
1d01040
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkMultiInsertionProcessor.java
93940bc
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkProcessAnalyzeTable.java
20eb344
ql/src/java/org/apache/hadoop/hive/ql/parse/spark/SparkTableScanProcessor.java
a62643a
ql/src/java/org/apache/hadoop/hive/ql/plan/BaseWork.java 05be1f1
ql/src/test/queries/clientpositive/spark_multi_insert_split_work.q
PRE-CREATION
ql/src/test/results/clientpositive/spark/groupby7_map.q.out 2d99a81
ql/src/test/results/clientpositive/spark/groupby7_map_skew.q.out ca73985
ql/src/test/results/clientpositive/spark/groupby7_noskew.q.out 2d2c55b
ql/src/test/results/clientpositive/spark/groupby_cube1.q.out 942cdaa
ql/src/test/results/clientpositive/spark/groupby_multi_single_reducer.q.out
399fe41
ql/src/test/results/clientpositive/spark/groupby_position.q.out 5e68807
ql/src/test/results/clientpositive/spark/groupby_rollup1.q.out 4259412
ql/src/test/results/clientpositive/spark/groupby_sort_1_23.q.out e0e882e
ql/src/test/results/clientpositive/spark/groupby_sort_skew_1_23.q.out a43921e
ql/src/test/results/clientpositive/spark/input12.q.out 4b0cf44
ql/src/test/results/clientpositive/spark/input13.q.out 260a65a
ql/src/test/results/clientpositive/spark/input1_limit.q.out 1f3b484
ql/src/test/results/clientpositive/spark/input_part2.q.out f2f3a2d
ql/src/test/results/clientpositive/spark/insert1.q.out 65032cb
ql/src/test/results/clientpositive/spark/insert_into3.q.out 5318a8b
ql/src/test/results/clientpositive/spark/load_dyn_part1.q.out 3b669fc
ql/src/test/results/clientpositive/spark/load_dyn_part8.q.out 50c052d
ql/src/test/results/clientpositive/spark/multi_insert.q.out bae325f
ql/src/test/results/clientpositive/spark/multi_insert_gby3.q.out 280a893
ql/src/test/results/clientpositive/spark/multi_insert_lateral_view.q.out
b07c582
ql/src/test/results/clientpositive/spark/multi_insert_move_tasks_share_dependencies.q.out
fd477ca
ql/src/test/results/clientpositive/spark/multigroupby_singlemr.q.out 44991e3
ql/src/test/results/clientpositive/spark/ppd_multi_insert.q.out 96f2c06
ql/src/test/results/clientpositive/spark/ppd_transform.q.out 7ec5d8d
ql/src/test/results/clientpositive/spark/spark_multi_insert_split_work.q.out
PRE-CREATION
ql/src/test/results/clientpositive/spark/subquery_multiinsert.q.out 2b4a331
ql/src/test/results/clientpositive/spark/union18.q.out f94fa0b
ql/src/test/results/clientpositive/spark/union19.q.out 8dcb543
ql/src/test/results/clientpositive/spark/union_remove_6.q.out 6730010
ql/src/test/results/clientpositive/spark/vectorized_ptf.q.out 909378b
Diff: https://reviews.apache.org/r/26706/diff/
Testing
-------
Thanks,
Chao Sun
