[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298763#comment-15298763 ] Ashutosh Chauhan commented on HIVE-13518: - [~vikram.dixit] Does this need more work? > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15298769#comment-15298769 ] Vikram Dixit K commented on HIVE-13518: --- Yes. The test failures are related. I will be posting an update for addressing them. > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15304522#comment-15304522 ] Gunther Hagleitner commented on HIVE-13518: --- +1 to .3. > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch, HIVE-13518.2.patch, > HIVE-13518.3.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[
https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305289#comment-15305289
]
Hive QA commented on HIVE-13518:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806300/HIVE-13518.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 252 failed/errored test(s), 6197 tests
executed
*Failed tests:*
{noformat}
TestCliDriver-alter_file_format.q-bucket_map_join_tez1.q-fetch_aggregation.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-alter_rename_table.q-udf_aes_decrypt.q-udf_add.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-alter_table_not_sorted.q-cbo_udf_max.q-exim_11_managed_external.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ambiguitycheck.q-udf_bitwise_and.q-insert_overwrite_directory.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_1_sql_std.q-disallow_incompatible_type_change_off.q-schema_evol_text_nonvec_mapwork_table.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_nonsql.q-orc_createas1.q-skewjoinopt19.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_cli_stdconfigauth.q-str_to_map.q-touch.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_create_table_owner_privs.q-cte_7.q-groupby_sort_7.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_create_temp_table.q-update_all_partitioned.q-lock4.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_parts.q-authorization_view_1.q-parquet_map_of_maps.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-authorization_role_grant2.q-union_remove_7.q-alter_view_as_select.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-auto_join18.q-dynamic_partition_pruning.q-date_comparison.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-auto_join30.q-udf_bitmap_empty.q-input17.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-auto_join_reordering_values.q-subquery_in.q-partition_timestamp2.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-auto_sortmerge_join_7.q-exim_04_evolved_parts.q-query_with_semi.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-autogen_colalias.q-skewjoinopt3.q-rcfile_merge1.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-avro_add_column.q-bucketmapjoin_negative.q-bucket_map_join_spark2.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-ba_table1.q-udf_notequal.q-input_part1.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-bucketcontext_4.q-orc_ends_with_nulls.q-correlationoptimizer9.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-bucketsortoptimize_insert_7.q-list_bucket_query_multiskew_1.q-skewjoin_noskew.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-cast_to_int.q-literal_double.q-skewjoinopt20.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-cbo_rp_gby_empty.q-load_nonpart_authsuccess.q-truncate_table.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_join1.q-union_top_level.q-join18.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-cbo_rp_limit.q-load_exist_part_authsuccess.q-schema_evol_text_nonvec_mapwork_part_all_primitive.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_lineage2.q-authorization_update.q-udf_pmod.q-and-12-more -
did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_semijoin.q-custom_input_output_format.q-udf_reflect2.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-cbo_rp_udf_udaf.q-date_udf.q-limit0.q-and-12-more - did not
produce a TEST-*.xml file
TestCliDriver-cbo_rp_views.q-orc_split_elimination.q-authorization_view_disable_cbo_4.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-columnStatsUpdateForStatsOptimizer_2.q-cp_mj_rc.q-show_columns.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-columnstats_partlvl_dp.q-input31.q-udaf_percentile.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-compute_stats_string.q-noalias_subq1.q-vector_auto_smb_mapjoin_14.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-create_func1.q-bucketmapjoin3.q-enforce_order.q-and-12-more - did
not produce a TEST-*.xml file
TestCliDriver-create_genericudf.q-dynamic_partition_insert.q-skewjoinopt16.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-create_or_replace_view.q-compute_stats_double.q-reduce_deduplicate_exclude_gby.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-cte_mat_1.q-groupby_sort_6.q-update_after_multiple_inserts_special_characters.q-and-12-more
- did not produce a TEST-*.xml file
TestCliDriver-decimal_join.q-ma
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[
https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15307588#comment-15307588
]
Hive QA commented on HIVE-13518:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12806300/HIVE-13518.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10172 tests
executed
*Failed tests:*
{noformat}
TestHWISessionManager - did not produce a TEST-*.xml file
TestJdbcWithMiniHA - did not produce a TEST-*.xml file
TestJdbcWithMiniMr - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-schema_evol_orc_nonvec_mapwork_part.q-union5.q-and-12-more
- did not produce a TEST-*.xml file
TestOperationLoggingAPIWithTez - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ivyDownload
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_constprog_partitioner
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_all
org.apache.hadoop.hive.llap.tez.TestConverters.testFragmentSpecToTaskSpec
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testLocksInSubquery
{noformat}
Test results:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/467/testReport
Console output:
http://ec2-54-177-240-2.us-west-1.compute.amazonaws.com/job/PreCommit-HIVE-MASTER-Build/467/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-467/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12806300 - PreCommit-HIVE-MASTER-Build
> Hive on Tez: Shuffle joins do not choose the right 'big' table.
> ---
>
> Key: HIVE-13518
> URL: https://issues.apache.org/jira/browse/HIVE-13518
> Project: Hive
> Issue Type: Bug
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13518.1.patch, HIVE-13518.2.patch,
> HIVE-13518.3.patch
>
>
> Currently the big table is always assumed to be at position 0 but this isn't
> efficient for some queries as the big table at position 1 could have a lot
> more keys/skew. We already have a mechanism of choosing the big table that
> can be leveraged to make the right choice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308394#comment-15308394 ] Jesus Camacho Rodriguez commented on HIVE-13518: Removing 2.1.0 target as issue is not tagged as Critical/Blocker and the RC will be created tomorrow. Please feel free to commit to branch-2.1 anyway and fix for 2.1.0 if this happens before the release. > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch, HIVE-13518.2.patch, > HIVE-13518.3.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15241692#comment-15241692 ] Vikram Dixit K commented on HIVE-13518: --- Ping [~gopalv]/[~rajesh.balamohan] > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[
https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15242738#comment-15242738
]
Jesus Camacho Rodriguez commented on HIVE-13518:
LGTM, +1.
Nitpick: {{getMapJoinConversionPos}} could be renamed, as it does not apply
only to MapJoin anymore (getBigTablePos?)
> Hive on Tez: Shuffle joins do not choose the right 'big' table.
> ---
>
> Key: HIVE-13518
> URL: https://issues.apache.org/jira/browse/HIVE-13518
> Project: Hive
> Issue Type: Bug
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13518.1.patch
>
>
> Currently the big table is always assumed to be at position 0 but this isn't
> efficient for some queries as the big table at position 1 could have a lot
> more keys/skew. We already have a mechanism of choosing the big table that
> can be leveraged to make the right choice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[
https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244532#comment-15244532
]
Hive QA commented on HIVE-13518:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12798786/HIVE-13518.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 9982 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_stats
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_metadataonly1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.cli.TestPerfCliDriver.testPerfCliDriver_query39
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7624/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/7624/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-7624/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12798786 - PreCommit-HIVE-TRUNK-Build
> Hive on Tez: Shuffle joins do not choose the right 'big' table.
> ---
>
> Key: HIVE-13518
> URL: https://issues.apache.org/jira/browse/HIVE-13518
> Project: Hive
> Issue Type: Bug
>Reporter: Vikram Dixit K
>Assignee: Vikram Dixit K
> Attachments: HIVE-13518.1.patch
>
>
> Currently the big table is always assumed to be at position 0 but this isn't
> efficient for some queries as the big table at position 1 could have a lot
> more keys/skew. We already have a mechanism of choosing the big table that
> can be leveraged to make the right choice.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13518) Hive on Tez: Shuffle joins do not choose the right 'big' table.
[ https://issues.apache.org/jira/browse/HIVE-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15263064#comment-15263064 ] Gunther Hagleitner commented on HIVE-13518: --- [~vikram.dixit] is this ready? > Hive on Tez: Shuffle joins do not choose the right 'big' table. > --- > > Key: HIVE-13518 > URL: https://issues.apache.org/jira/browse/HIVE-13518 > Project: Hive > Issue Type: Bug >Reporter: Vikram Dixit K >Assignee: Vikram Dixit K > Attachments: HIVE-13518.1.patch > > > Currently the big table is always assumed to be at position 0 but this isn't > efficient for some queries as the big table at position 1 could have a lot > more keys/skew. We already have a mechanism of choosing the big table that > can be leveraged to make the right choice. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
