[
https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721346#comment-13721346
]
Hive QA commented on HIVE-4827:
-------------------------------
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12594435/HIVE-4827.4.patch
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/199/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/199/console
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.CleanupPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests failed with: NonZeroExitCodeException: Command 'bash
/data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and
output '+ [[ -n '' ]]
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-Build-199/source-prep.txt
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'contrib/src/test/results/clientpositive/serde_typedbytes4.q.out'
Reverted 'contrib/src/test/results/clientpositive/dboutput.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join27.q.out'
Reverted 'ql/src/test/results/clientpositive/join14_hadoop20.q.out'
Reverted 'ql/src/test/results/clientpositive/udf9.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_clusterby.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join13.q.out'
Reverted
'ql/src/test/results/clientpositive/groupby_multi_single_reducer2.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_update.q.out'
Reverted 'ql/src/test/results/clientpositive/notable_alias1.q.out'
Reverted 'ql/src/test/results/clientpositive/load_dyn_part13.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_union.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_3.q.out'
Reverted 'ql/src/test/results/clientpositive/join38.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part7.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out'
Reverted 'ql/src/test/results/clientpositive/pcr.q.out'
Reverted 'ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/cluster.q.out'
Reverted 'ql/src/test/results/clientpositive/input12_hadoop20.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_file_format.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part2.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_transform.q.out'
Reverted 'ql/src/test/results/clientpositive/create_view.q.out'
Reverted 'ql/src/test/results/clientpositive/alter_partition_coltype.q.out'
Reverted 'ql/src/test/results/clientpositive/infer_const_type.q.out'
Reverted 'ql/src/test/results/clientpositive/correlationoptimizer8.q.out'
Reverted 'ql/src/test/results/clientpositive/join7.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join4.q.out'
Reverted 'ql/src/test/results/clientpositive/input2_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/input11_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_like.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_hour.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_1.q.out'
Reverted 'ql/src/test/results/clientpositive/ppr_pushdown3.q.out'
Reverted 'ql/src/test/results/clientpositive/input11.q.out'
Reverted 'ql/src/test/results/clientpositive/noalias_subq1.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join29.q.out'
Reverted
'ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out'
Reverted 'ql/src/test/results/clientpositive/regex_col.q.out'
Reverted 'ql/src/test/results/clientpositive/allcolref_in_udf.q.out'
Reverted 'ql/src/test/results/clientpositive/udf1.q.out'
Reverted 'ql/src/test/results/clientpositive/join12.q.out'
Reverted 'ql/src/test/results/clientpositive/join35.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part4.q.out'
Reverted 'ql/src/test/results/clientpositive/union20.q.out'
Reverted 'ql/src/test/results/clientpositive/column_access_stats.q.out'
Reverted 'ql/src/test/results/clientpositive/join21.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_multi_insert.q.out'
Reverted 'ql/src/test/results/clientpositive/index_stale_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join6.q.out'
Reverted 'ql/src/test/results/clientpositive/correlationoptimizer10.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_lower.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup.q.out'
Reverted 'ql/src/test/results/clientpositive/join4.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup4_multi_distinct.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap_compression.q.out'
Reverted 'ql/src/test/results/clientpositive/input18.q.out'
Reverted 'ql/src/test/results/clientpositive/multi_insert_gby.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_mult_tables.q.out'
Reverted 'ql/src/test/results/clientpositive/rand_partitionpruner3.q.out'
Reverted 'ql/src/test/results/clientpositive/input13.q.out'
Reverted 'ql/src/test/results/clientpositive/order2.q.out'
Reverted 'ql/src/test/results/clientpositive/multi_insert.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_udf_col.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join12.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd_outer_join1.q.out'
Reverted 'ql/src/test/results/clientpositive/ppd2.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_empty.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join21.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_2.q.out'
Reverted 'ql/src/test/results/clientpositive/union22.q.out'
Reverted 'ql/src/test/results/clientpositive/join23.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_second.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_multiple.q.out'
Reverted 'ql/src/test/results/clientpositive/union.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_10_trims.q.out'
Reverted
'ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part1.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join8.q.out'
Reverted 'ql/src/test/results/clientpositive/index_compression.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_parse_url.q.out'
Reverted 'ql/src/test/results/clientpositive/join41.q.out'
Reverted 'ql/src/test/results/clientpositive/multi_insert_lateral_view.q.out'
Reverted 'ql/src/test/results/clientpositive/no_hooks.q.out'
Reverted 'ql/src/test/results/clientpositive/join6.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap3.q.out'
Reverted 'ql/src/test/results/clientpositive/groupby_position.q.out'
Reverted 'ql/src/test/results/clientpositive/transform_ppr2.q.out'
Reverted 'ql/src/test/results/clientpositive/set_processor_namespaces.q.out'
Reverted 'ql/src/test/results/clientpositive/union_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join14_hadoop20.q.out'
Reverted
'ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out'
Reverted 'ql/src/test/results/clientpositive/nonmr_fetch.q.out'
Reverted 'ql/src/test/results/clientpositive/router_join_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join28.q.out'
Reverted
'ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out'
Reverted 'ql/src/test/results/clientpositive/udf_minute.q.out'
Reverted 'ql/src/test/results/clientpositive/input42.q.out'
Reverted 'ql/src/test/results/clientpositive/having.q.out'
Reverted 'ql/src/test/results/clientpositive/quote1.q.out'
Reverted 'ql/src/test/results/clientpositive/notable_alias2.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_unused.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join23.q.out'
Reverted 'ql/src/test/results/clientpositive/join16.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_4.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup2.q.out'
Reverted 'ql/src/test/results/clientpositive/join39.q.out'
Reverted 'ql/src/test/results/clientpositive/union24.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto.q.out'
Reverted 'ql/src/test/results/clientpositive/rcfile_null_value.q.out'
Reverted 'ql/src/test/results/clientpositive/union33.q.out'
Reverted 'ql/src/test/results/clientpositive/join11.q.out'
Reverted 'ql/src/test/results/clientpositive/join34.q.out'
Reverted 'ql/src/test/results/clientpositive/join20.q.out'
Reverted 'ql/src/test/results/clientpositive/join8.q.out'
Reverted 'ql/src/test/results/clientpositive/subq.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join5.q.out'
Reverted 'ql/src/test/results/clientpositive/input14_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/louter_join_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join0.q.out'
Reverted 'ql/src/test/results/clientpositive/skewjoin.q.out'
Reverted 'ql/src/test/results/clientpositive/index_auto_self_join.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join16.q.out'
Reverted 'ql/src/test/results/clientpositive/nullgroup4.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join11.q.out'
Reverted 'ql/src/test/results/clientpositive/subq2.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join20.q.out'
Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_1.q.out'
Reverted 'ql/src/test/results/clientpositive/join13.q.out'
Reverted 'ql/src/test/results/clientpositive/input_part5.q.out'
Reverted 'ql/src/test/results/clientpositive/cast1.q.out'
Reverted 'ql/src/test/results/clientpositive/index_stale.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join7.q.out'
Reverted 'ql/src/test/results/clientpositive/join40.q.out'
Reverted 'ql/src/test/results/clientpositive/join5.q.out'
Reverted 'ql/src/test/results/clientpositive/outer_join_ppr.q.out'
Reverted 'ql/src/test/results/clientpositive/input1_limit.q.out'
Reverted 'ql/src/test/results/clientpositive/index_bitmap_auto.q.out'
Reverted 'ql/src/test/results/clientpositive/transform_ppr1.q.out'
Reverted 'ql/src/test/results/clientpositive/join0.q.out'
Reverted 'ql/src/test/results/clientpositive/regexp_extract.q.out'
Reverted 'ql/src/test/results/clientpositive/ctas_colname.q.out'
Reverted 'ql/src/test/results/clientpositive/mergejoins.q.out'
Reverted 'ql/src/test/results/clientpositive/input14.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join27.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join13.q.out'
Reverted 'ql/src/test/results/beelinepositive/create_view.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join28.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join4.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join23.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join29.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join5.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join0.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join6.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join16.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join20.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join7.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join12.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join21.q.out'
Reverted 'ql/src/test/results/beelinepositive/auto_join8.q.out'
Reverted 'ql/src/test/results/compiler/plan/input2.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input3.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join4.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input4.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join5.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join6.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join7.q.xml'
Reverted 'ql/src/test/results/compiler/plan/join8.q.xml'
Reverted 'ql/src/test/results/compiler/plan/union.q.xml'
Reverted 'ql/src/test/results/compiler/plan/udf1.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input_part1.q.xml'
Reverted 'ql/src/test/results/compiler/plan/subq.q.xml'
Reverted 'ql/src/test/results/compiler/plan/cast1.q.xml'
Reverted 'ql/src/test/results/compiler/plan/input1.q.xml'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf build hcatalog/build hcatalog/core/build
hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build
hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build
hcatalog/hcatalog-pig-adapter/build common/src/gen
+ svn update
D testutils/ptest2/src/test/resources/TEST-SomeTest-failure.xml
D
testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin_union_remove_1.q-6fa31776-d2b0-4e13-9761-11f750627ad1-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D
testutils/ptest2/src/test/resources/test-outputs/TEST-union_remove_9.q-acb9de8f-1b9c-4874-924c-b2107ca7b07c-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D
testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin.q-ab8536a7-1b5c-45ed-ba29-14450f27db8b-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D
testutils/ptest2/src/test/resources/test-outputs/TEST-index_auth.q-bucketcontex-ba31fb54-1d7f-4c70-a89d-477b7d155191-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml
D
testutils/ptest2/src/test/resources/test-outputs/TEST-SomeTest-truncated.xml
Fetching external item into 'hcatalog/src/test/e2e/harness'
Updated external to revision 1507508.
Updated to revision 1507508.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0 to p2
+ exit 1
'
{noformat}
This message is automatically generated.
> Merge a Map-only job to its following MapReduce job with multiple inputs
> ------------------------------------------------------------------------
>
> Key: HIVE-4827
> URL: https://issues.apache.org/jira/browse/HIVE-4827
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 0.12.0
> Reporter: Yin Huai
> Assignee: Yin Huai
> Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch, HIVE-4827.3.patch,
> HIVE-4827.4.patch
>
>
> When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a
> Map-only job (MapJoin) to its following MapReduce job. But this merge only
> happens when the MapReduce job has a single input. With Correlation Optimizer
> (HIVE-2206), it is possible that the MapReduce job can have multiple inputs
> (for multiple operation paths). It is desired to improve CommonJoinResolver
> to merge a Map-only job to the corresponding Map task of the MapReduce job.
> Example:
> {code:sql}
> set hive.optimize.correlation=true;
> set hive.auto.convert.join=true;
> set hive.optimize.mapjoin.mapreduce=true;
> SELECT tmp1.key, count(*)
> FROM (SELECT x1.key1 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> GROUP BY x1.key1) tmp1
> JOIN (SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2)
> GROUP BY x2.key2) tmp2
> ON (tmp1.key = tmp2.key)
> GROUP BY tmp1.key;
> {\code}
> In this query, join operations inside tmp1 and tmp2 will be converted to two
> MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of
> tmp1 and tmp2, and the last aggregation will be executed in the same
> MapReduce job (Reduce side). Since this MapReduce job has two inputs, right
> now, CommonJoinResolver cannot attach two MapJoins to the Map side of a
> MapReduce job.
> Another example:
> {code:sql}
> SELECT tmp1.key
> FROM (SELECT x1.key2 AS key
> FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1)
> UNION ALL
> SELECT x2.key2 AS key
> FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1
> {\code}
> For this case, we will have three Map-only jobs (two for MapJoins and one for
> Union). It will be good to use a single Map-only job to execute this query.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
