[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298763#comment-15298763 ] Ashutosh Chauhan commented on HIVE-13518: - [~vikram.dixit] Does this need more work? > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298769#comment-15298769 ] Vikram Dixit K commented on HIVE-13518: --- Yes. The test failures are related. I will be posting an update for addressing them. > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304522#comment-15304522 ] Gunther Hagleitner commented on HIVE-13518: --- +1 to .3. > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch, HIVE-13518.2.patch, > HIVE-13518.3.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305289#comment-15305289 ] Hive QA commented on HIVE-13518: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12806300/HIVE-13518.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 252 failed/errored test(s), 6197 tests executed *Failed tests:* {noformat} TestCliDriver-alter_file_format.q-bucket_map_join_tez1.q-fetch_aggregation.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-alter_rename_table.q-udf_aes_decrypt.q-udf_add.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-alter_table_not_sorted.q-cbo_udf_max.q-exim_11_managed_external.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-ambiguitycheck.q-udf_bitwise_and.q-insert_overwrite_directory.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_1_sql_std.q-disallow_incompatible_type_change_off.q-schema_evol_text_nonvec_mapwork_table.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_cli_nonsql.q-orc_createas1.q-skewjoinopt19.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_cli_stdconfigauth.q-str_to_map.q-touch.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_create_table_owner_privs.q-cte_7.q-groupby_sort_7.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_create_temp_table.q-update_all_partitioned.q-lock4.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_parts.q-authorization_view_1.q-parquet_map_of_maps.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-authorization_role_grant2.q-union_remove_7.q-alter_view_as_select.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_join18.q-dynamic_partition_pruning.q-date_comparison.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_join30.q-udf_bitmap_empty.q-input17.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_join_reordering_values.q-subquery_in.q-partition_timestamp2.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-auto_sortmerge_join_7.q-exim_04_evolved_parts.q-query_with_semi.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-autogen_colalias.q-skewjoinopt3.q-rcfile_merge1.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-avro_add_column.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-bucketcontext_4.q-orc_ends_with_nulls.q-correlationoptimizer9.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-bucketsortoptimize_insert_7.q-list_bucket_query_multiskew_1.q-skewjoin_noskew.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cast_to_int.q-literal_double.q-skewjoinopt20.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cbo_rp_gby_empty.q-load_nonpart_authsuccess.q-truncate_table.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cbo_rp_join1.q-union_top_level.q-join18.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cbo_rp_limit.q-load_exist_part_authsuccess.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cbo_rp_lineage2.q-authorization_update.q-udf_pmod.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cbo_rp_semijoin.q-custom_input_output_format.q-udf_reflect2.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cbo_rp_udf_udaf.q-date_udf.q-limit0.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cbo_rp_views.q-orc_split_elimination.q-authorization_view_disable_cbo_4.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-cp_mj_rc.q-show_columns.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-columnstats_partlvl_dp.q-input31.q-udaf_percentile.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-compute_stats_string.q-noalias_subq1.q-vector_auto_smb_mapjoin_14.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-create_func1.q-bucketmapjoin3.q-enforce_order.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-create_genericudf.q-dynamic_partition_insert.q-skewjoinopt16.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-create_or_replace_view.q-compute_stats_double.q-reduce_deduplicate_exclude_gby.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-cte_mat_1.q-groupby_sort_6.q-update_after_multiple_inserts_special_characters.q-and-12-more - did not produce a TEST-*.xml file TestCliDriver-decimal_join.q-ma
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307588#comment-15307588 ] Hive QA commented on HIVE-13518: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12806300/HIVE-13518.3.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10172 tests executed *Failed tests:* {noformat} TestHWISessionManager - did not produce a TEST-*.xml file TestJdbcWithMiniHA - did not produce a TEST-*.xml file TestJdbcWithMiniMr - did not produce a TEST-*.xml file TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more - did not produce a TEST-*.xml file TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLocksInSubquery {noformat} Test results: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/467/testReport Console output: http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/467/console Test logs: http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-467/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 14 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12806300 - PreCommit-HIVE-MASTER-Build > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch, HIVE-13518.2.patch, > HIVE-13518.3.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308394#comment-15308394 ] Jesus Camacho Rodriguez commented on HIVE-13518: Removing 2.1.0 target as issue is not tagged as Critical/Blocker and the RC will be created tomorrow. Please feel free to commit to branch-2.1 anyway and fix for 2.1.0 if this happens before the release. > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch, HIVE-13518.2.patch, > HIVE-13518.3.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241692#comment-15241692 ] Vikram Dixit K commented on HIVE-13518: --- Ping [~gopalv]/[~rajesh.balamohan] > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242738#comment-15242738 ] Jesus Camacho Rodriguez commented on HIVE-13518: LGTM, +1. Nitpick: {{getMapJoinConversionPos}} could be renamed, as it does not apply only to MapJoin anymore (getBigTablePos?) > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244532#comment-15244532 ] Hive QA commented on HIVE-13518: Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12798786/HIVE-13518.1.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9982 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join29 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_stats org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1 org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query39 {noformat} Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7624/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7624/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7624/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 10 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12798786 - PreCommit-HIVE-TRUNK-Build > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263064#comment-15263064 ] Gunther Hagleitner commented on HIVE-13518: --- [~vikram.dixit] is this ready? > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)