[
https://issues.apache.org/jira/browse/HIVE-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13487852#comment-13487852
]
Hudson commented on HIVE-3276:
------------------------------
Integrated in Hive-trunk-h0.21 #1767 (See
[https://builds.apache.org/job/Hive-trunk-h0.21/1767/])
HIVE-3276. optimize union sub-queries. (njain via kevinwilfong) (Revision
1403928)
Result = FAILURE
kevinwilfong :
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1403928
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ErrorMsg.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcContext.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcessor.java
*
/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/FileSinkDesc.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/SelectDesc.java
* /hive/trunk/ql/src/test/queries/clientpositive/skewjoin_union_remove_1.q
* /hive/trunk/ql/src/test/queries/clientpositive/skewjoin_union_remove_2.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_1.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_10.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_11.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_12.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_13.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_14.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_15.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_16.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_17.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_18.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_19.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_2.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_20.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_21.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_22.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_23.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_24.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_3.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_4.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_5.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_6.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_7.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_8.q
* /hive/trunk/ql/src/test/queries/clientpositive/union_remove_9.q
* /hive/trunk/ql/src/test/results/clientpositive/skewjoin_union_remove_1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/skewjoin_union_remove_2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_1.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_10.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_11.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_12.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_13.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_14.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_15.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_16.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_17.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_18.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_19.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_2.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_20.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_21.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_22.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_23.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_24.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_4.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_5.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_6.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_7.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_8.q.out
* /hive/trunk/ql/src/test/results/clientpositive/union_remove_9.q.out
> optimize union sub-queries
> --------------------------
>
> Key: HIVE-3276
> URL: https://issues.apache.org/jira/browse/HIVE-3276
> Project: Hive
> Issue Type: Bug
> Affects Versions: 0.10.0
> Reporter: Namit Jain
> Assignee: Namit Jain
> Fix For: 0.10.0
>
> Attachments: hive.3276.10.patch, hive.3276.11.patch,
> hive.3276.12.patch, hive.3276.13.patch, hive.3276.14.patch,
> HIVE-3276.1.patch, hive.3276.2.patch, hive.3276.3.patch, hive.3276.4.patch,
> hive.3276.5.patch, hive.3276.6.patch, hive.3276.7.patch, hive.3276.8.patch,
> hive.3276.9.patch
>
>
> It might be a good idea to optimize simple union queries containing
> map-reduce jobs in at least one of the sub-qeuries.
> For eg:
> a query like:
> insert overwrite table T1 partition P1
> select * from
> (
> subq1
> union all
> subq2
> ) u;
> today creates 3 map-reduce jobs, one for subq1, another for subq2 and
> the final one for the union.
> It might be a good idea to optimize this. Instead of creating the union
> task, it might be simpler to create a move task (or something like a move
> task), where the outputs of the two sub-queries will be moved to the final
> directory. This can easily extend to more than 2 sub-queries in the union.
> This is very useful if there is a select * followed by filesink after the
> union. This can be independently useful, and also be used to optimize the
> skewed joins https://cwiki.apache.org/Hive/skewed-join-optimization.html.
> If there is a select, filter between the union and the filesink, the select
> and the filter can be moved before the union, and the follow-up job can
> still be removed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
