[ https://issues.apache.org/jira/browse/HIVE-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13721346#comment-13721346 ]
Hive QA commented on HIVE-4827: ------------------------------- {color:red}Overall{color}: -1 no tests executed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12594435/HIVE-4827.4.patch Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/199/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/199/console Messages: {noformat} Executing org.apache.hive.ptest.execution.CleanupPhase Executing org.apache.hive.ptest.execution.PrepPhase Tests failed with: NonZeroExitCodeException: Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit status 1 and output '+ [[ -n '' ]] + export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost -Dhttp.proxyPort=3128' + cd /data/hive-ptest/working/ + tee /data/hive-ptest/logs/PreCommit-HIVE-Build-199/source-prep.txt + mkdir -p maven ivy + [[ svn = \s\v\n ]] + [[ -n '' ]] + [[ -d apache-svn-trunk-source ]] + [[ ! -d apache-svn-trunk-source/.svn ]] + [[ ! -d apache-svn-trunk-source ]] + cd apache-svn-trunk-source + svn revert -R . Reverted 'contrib/src/test/results/clientpositive/serde_typedbytes4.q.out' Reverted 'contrib/src/test/results/clientpositive/dboutput.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join27.q.out' Reverted 'ql/src/test/results/clientpositive/join14_hadoop20.q.out' Reverted 'ql/src/test/results/clientpositive/udf9.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_clusterby.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join13.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_multi_single_reducer2.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_update.q.out' Reverted 'ql/src/test/results/clientpositive/notable_alias1.q.out' Reverted 'ql/src/test/results/clientpositive/load_dyn_part13.q.out' Reverted 'ql/src/test/results/clientpositive/udf_union.q.out' Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_3.q.out' Reverted 'ql/src/test/results/clientpositive/join38.q.out' Reverted 'ql/src/test/results/clientpositive/input_part7.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_grouping_sets4.q.out' Reverted 'ql/src/test/results/clientpositive/pcr.q.out' Reverted 'ql/src/test/results/clientpositive/mapjoin_filter_on_outerjoin.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_partitioned.q.out' Reverted 'ql/src/test/results/clientpositive/cluster.q.out' Reverted 'ql/src/test/results/clientpositive/input12_hadoop20.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_file_format.q.out' Reverted 'ql/src/test/results/clientpositive/input_part2.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_transform.q.out' Reverted 'ql/src/test/results/clientpositive/create_view.q.out' Reverted 'ql/src/test/results/clientpositive/alter_partition_coltype.q.out' Reverted 'ql/src/test/results/clientpositive/infer_const_type.q.out' Reverted 'ql/src/test/results/clientpositive/correlationoptimizer8.q.out' Reverted 'ql/src/test/results/clientpositive/join7.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join4.q.out' Reverted 'ql/src/test/results/clientpositive/input2_limit.q.out' Reverted 'ql/src/test/results/clientpositive/input11_limit.q.out' Reverted 'ql/src/test/results/clientpositive/udf_like.q.out' Reverted 'ql/src/test/results/clientpositive/udf_hour.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_sort_1.q.out' Reverted 'ql/src/test/results/clientpositive/ppr_pushdown3.q.out' Reverted 'ql/src/test/results/clientpositive/input11.q.out' Reverted 'ql/src/test/results/clientpositive/noalias_subq1.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join29.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out' Reverted 'ql/src/test/results/clientpositive/regex_col.q.out' Reverted 'ql/src/test/results/clientpositive/allcolref_in_udf.q.out' Reverted 'ql/src/test/results/clientpositive/udf1.q.out' Reverted 'ql/src/test/results/clientpositive/join12.q.out' Reverted 'ql/src/test/results/clientpositive/join35.q.out' Reverted 'ql/src/test/results/clientpositive/input_part4.q.out' Reverted 'ql/src/test/results/clientpositive/union20.q.out' Reverted 'ql/src/test/results/clientpositive/column_access_stats.q.out' Reverted 'ql/src/test/results/clientpositive/join21.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_multi_insert.q.out' Reverted 'ql/src/test/results/clientpositive/index_stale_partitioned.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join6.q.out' Reverted 'ql/src/test/results/clientpositive/correlationoptimizer10.q.out' Reverted 'ql/src/test/results/clientpositive/udf_lower.q.out' Reverted 'ql/src/test/results/clientpositive/nullgroup.q.out' Reverted 'ql/src/test/results/clientpositive/join4.q.out' Reverted 'ql/src/test/results/clientpositive/nullgroup4_multi_distinct.q.out' Reverted 'ql/src/test/results/clientpositive/index_bitmap_compression.q.out' Reverted 'ql/src/test/results/clientpositive/input18.q.out' Reverted 'ql/src/test/results/clientpositive/multi_insert_gby.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_multi_single_reducer.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_mult_tables.q.out' Reverted 'ql/src/test/results/clientpositive/rand_partitionpruner3.q.out' Reverted 'ql/src/test/results/clientpositive/input13.q.out' Reverted 'ql/src/test/results/clientpositive/order2.q.out' Reverted 'ql/src/test/results/clientpositive/multi_insert.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_udf_col.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join12.q.out' Reverted 'ql/src/test/results/clientpositive/ppd_outer_join1.q.out' Reverted 'ql/src/test/results/clientpositive/ppd2.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_empty.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join21.q.out' Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_2.q.out' Reverted 'ql/src/test/results/clientpositive/union22.q.out' Reverted 'ql/src/test/results/clientpositive/join23.q.out' Reverted 'ql/src/test/results/clientpositive/udf_second.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_multiple.q.out' Reverted 'ql/src/test/results/clientpositive/union.q.out' Reverted 'ql/src/test/results/clientpositive/udf_10_trims.q.out' Reverted 'ql/src/test/results/clientpositive/multi_insert_move_tasks_share_dependencies.q.out' Reverted 'ql/src/test/results/clientpositive/input_part1.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_sort_skew_1.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join8.q.out' Reverted 'ql/src/test/results/clientpositive/index_compression.q.out' Reverted 'ql/src/test/results/clientpositive/udf_parse_url.q.out' Reverted 'ql/src/test/results/clientpositive/join41.q.out' Reverted 'ql/src/test/results/clientpositive/multi_insert_lateral_view.q.out' Reverted 'ql/src/test/results/clientpositive/no_hooks.q.out' Reverted 'ql/src/test/results/clientpositive/join6.q.out' Reverted 'ql/src/test/results/clientpositive/index_bitmap3.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_position.q.out' Reverted 'ql/src/test/results/clientpositive/transform_ppr2.q.out' Reverted 'ql/src/test/results/clientpositive/set_processor_namespaces.q.out' Reverted 'ql/src/test/results/clientpositive/union_ppr.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join14_hadoop20.q.out' Reverted 'ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out' Reverted 'ql/src/test/results/clientpositive/nonmr_fetch.q.out' Reverted 'ql/src/test/results/clientpositive/router_join_ppr.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join28.q.out' Reverted 'ql/src/test/results/clientpositive/groupby_multi_single_reducer3.q.out' Reverted 'ql/src/test/results/clientpositive/udf_minute.q.out' Reverted 'ql/src/test/results/clientpositive/input42.q.out' Reverted 'ql/src/test/results/clientpositive/having.q.out' Reverted 'ql/src/test/results/clientpositive/quote1.q.out' Reverted 'ql/src/test/results/clientpositive/notable_alias2.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_unused.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join23.q.out' Reverted 'ql/src/test/results/clientpositive/join16.q.out' Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_4.q.out' Reverted 'ql/src/test/results/clientpositive/nullgroup2.q.out' Reverted 'ql/src/test/results/clientpositive/join39.q.out' Reverted 'ql/src/test/results/clientpositive/union24.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto.q.out' Reverted 'ql/src/test/results/clientpositive/rcfile_null_value.q.out' Reverted 'ql/src/test/results/clientpositive/union33.q.out' Reverted 'ql/src/test/results/clientpositive/join11.q.out' Reverted 'ql/src/test/results/clientpositive/join34.q.out' Reverted 'ql/src/test/results/clientpositive/join20.q.out' Reverted 'ql/src/test/results/clientpositive/join8.q.out' Reverted 'ql/src/test/results/clientpositive/subq.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join5.q.out' Reverted 'ql/src/test/results/clientpositive/input14_limit.q.out' Reverted 'ql/src/test/results/clientpositive/louter_join_ppr.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join0.q.out' Reverted 'ql/src/test/results/clientpositive/skewjoin.q.out' Reverted 'ql/src/test/results/clientpositive/index_auto_self_join.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join16.q.out' Reverted 'ql/src/test/results/clientpositive/nullgroup4.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join11.q.out' Reverted 'ql/src/test/results/clientpositive/subq2.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join20.q.out' Reverted 'ql/src/test/results/clientpositive/sort_merge_join_desc_1.q.out' Reverted 'ql/src/test/results/clientpositive/join13.q.out' Reverted 'ql/src/test/results/clientpositive/input_part5.q.out' Reverted 'ql/src/test/results/clientpositive/cast1.q.out' Reverted 'ql/src/test/results/clientpositive/index_stale.q.out' Reverted 'ql/src/test/results/clientpositive/auto_join7.q.out' Reverted 'ql/src/test/results/clientpositive/join40.q.out' Reverted 'ql/src/test/results/clientpositive/join5.q.out' Reverted 'ql/src/test/results/clientpositive/outer_join_ppr.q.out' Reverted 'ql/src/test/results/clientpositive/input1_limit.q.out' Reverted 'ql/src/test/results/clientpositive/index_bitmap_auto.q.out' Reverted 'ql/src/test/results/clientpositive/transform_ppr1.q.out' Reverted 'ql/src/test/results/clientpositive/join0.q.out' Reverted 'ql/src/test/results/clientpositive/regexp_extract.q.out' Reverted 'ql/src/test/results/clientpositive/ctas_colname.q.out' Reverted 'ql/src/test/results/clientpositive/mergejoins.q.out' Reverted 'ql/src/test/results/clientpositive/input14.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join27.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join13.q.out' Reverted 'ql/src/test/results/beelinepositive/create_view.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join28.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join4.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join23.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join29.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join5.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join0.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join6.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join16.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join20.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join7.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join12.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join21.q.out' Reverted 'ql/src/test/results/beelinepositive/auto_join8.q.out' Reverted 'ql/src/test/results/compiler/plan/input2.q.xml' Reverted 'ql/src/test/results/compiler/plan/input3.q.xml' Reverted 'ql/src/test/results/compiler/plan/join4.q.xml' Reverted 'ql/src/test/results/compiler/plan/input4.q.xml' Reverted 'ql/src/test/results/compiler/plan/join5.q.xml' Reverted 'ql/src/test/results/compiler/plan/join6.q.xml' Reverted 'ql/src/test/results/compiler/plan/join7.q.xml' Reverted 'ql/src/test/results/compiler/plan/join8.q.xml' Reverted 'ql/src/test/results/compiler/plan/union.q.xml' Reverted 'ql/src/test/results/compiler/plan/udf1.q.xml' Reverted 'ql/src/test/results/compiler/plan/input_part1.q.xml' Reverted 'ql/src/test/results/compiler/plan/subq.q.xml' Reverted 'ql/src/test/results/compiler/plan/cast1.q.xml' Reverted 'ql/src/test/results/compiler/plan/input1.q.xml' Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java' ++ egrep -v '^X|^Performing status on external' ++ awk '{print $2}' ++ svn status --no-ignore + rm -rf build hcatalog/build hcatalog/core/build hcatalog/storage-handlers/hbase/build hcatalog/server-extensions/build hcatalog/webhcat/svr/build hcatalog/webhcat/java-client/build hcatalog/hcatalog-pig-adapter/build common/src/gen + svn update D testutils/ptest2/src/test/resources/TEST-SomeTest-failure.xml D testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin_union_remove_1.q-6fa31776-d2b0-4e13-9761-11f750627ad1-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml D testutils/ptest2/src/test/resources/test-outputs/TEST-union_remove_9.q-acb9de8f-1b9c-4874-924c-b2107ca7b07c-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml D testutils/ptest2/src/test/resources/test-outputs/TEST-skewjoin.q-ab8536a7-1b5c-45ed-ba29-14450f27db8b-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml D testutils/ptest2/src/test/resources/test-outputs/TEST-index_auth.q-bucketcontex-ba31fb54-1d7f-4c70-a89d-477b7d155191-TEST-org.apache.hadoop.hive.cli.TestCliDriver.xml D testutils/ptest2/src/test/resources/test-outputs/TEST-SomeTest-truncated.xml Fetching external item into 'hcatalog/src/test/e2e/harness' Updated external to revision 1507508. Updated to revision 1507508. + patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh + patchFilePath=/data/hive-ptest/working/scratch/build.patch + [[ -f /data/hive-ptest/working/scratch/build.patch ]] + chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh + /data/hive-ptest/working/scratch/smart-apply-patch.sh /data/hive-ptest/working/scratch/build.patch The patch does not appear to apply with p0 to p2 + exit 1 ' {noformat} This message is automatically generated. > Merge a Map-only job to its following MapReduce job with multiple inputs > ------------------------------------------------------------------------ > > Key: HIVE-4827 > URL: https://issues.apache.org/jira/browse/HIVE-4827 > Project: Hive > Issue Type: Improvement > Components: Query Processor > Affects Versions: 0.12.0 > Reporter: Yin Huai > Assignee: Yin Huai > Attachments: HIVE-4827.1.patch, HIVE-4827.2.patch, HIVE-4827.3.patch, > HIVE-4827.4.patch > > > When hive.optimize.mapjoin.mapreduce is on, CommonJoinResolver can attach a > Map-only job (MapJoin) to its following MapReduce job. But this merge only > happens when the MapReduce job has a single input. With Correlation Optimizer > (HIVE-2206), it is possible that the MapReduce job can have multiple inputs > (for multiple operation paths). It is desired to improve CommonJoinResolver > to merge a Map-only job to the corresponding Map task of the MapReduce job. > Example: > {code:sql} > set hive.optimize.correlation=true; > set hive.auto.convert.join=true; > set hive.optimize.mapjoin.mapreduce=true; > SELECT tmp1.key, count(*) > FROM (SELECT x1.key1 AS key > FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1) > GROUP BY x1.key1) tmp1 > JOIN (SELECT x2.key2 AS key > FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key2 = y2.key2) > GROUP BY x2.key2) tmp2 > ON (tmp1.key = tmp2.key) > GROUP BY tmp1.key; > {\code} > In this query, join operations inside tmp1 and tmp2 will be converted to two > MapJoins. With Correlation Optimizer, aggregations in tmp1, tmp2, and join of > tmp1 and tmp2, and the last aggregation will be executed in the same > MapReduce job (Reduce side). Since this MapReduce job has two inputs, right > now, CommonJoinResolver cannot attach two MapJoins to the Map side of a > MapReduce job. > Another example: > {code:sql} > SELECT tmp1.key > FROM (SELECT x1.key2 AS key > FROM bigTable1 x1 JOIN smallTable1 y1 ON (x1.key1 = y1.key1) > UNION ALL > SELECT x2.key2 AS key > FROM bigTable2 x2 JOIN smallTable2 y2 ON (x2.key1 = y2.key1)) tmp1 > {\code} > For this case, we will have three Map-only jobs (two for MapJoins and one for > Union). It will be good to use a single Map-only job to execute this query. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira